InformationTheory
The Shannon entropy of a r.v. taking countable values is a measure of the uncertainty of the r.v. By Jensen’s Inequality the entropy for a r.v. taking values is upper bounded by that of the uniform distribution In particular, the entropy of a Bernoulli random variable is