Shannon's entropy | planetmath.org

Let $X$ be a discrete random variable on a finite set $\mathcal{X}=\{x_{1},\ldots,x_{n}\}$, with probability distribution function $p(x)=\Pr(X=x)$. The entropy $H(X)$ of $X$ is defined as

$$H(X)=-\sum_{{x\in\mathcal{X}}}p(x)\log_{b}p(x).$$

(1)

The convention $0\log 0=0$ is adopted in the definition. The logarithm is usually taken to the base 2, in which case the entropy is measured in "bits," or to the base e, in which case $H(X)$ is measured in "nats."

If $X$ and $Y$ are random variables on $\mathcal{X}$ and $\mathcal{Y}$ respectively, the joint entropy of $X$ and $Y$ is

$$H(X,Y)=-\sum_{{(x,y)\in\mathcal{X}\times\mathcal{Y}}}p(x,y)\log_{b}p(x,y),$$

where $p(x,y)$ denote the joint distribution of $X$ and $Y$.

Read full article from Shannon's entropy | planetmath.org