Kullback-Leibler Divergence

BayesianMachineLearningInformationTheoryMachineLearningStatistics The relative entropy or KL divergence between two distributions and is It has the same properties of a metric

• Non-negative:
• Definite: but it does not define a metric as it is
• Asymmetric: for
• Does not obey the triangle inequality
• Not a Bregman Divergence

The unnormalized relative entropy is a Bregman Divergence and is given by when and are probability distributions thus defined on the probability simplex, the KL divergence and unnormalized relative entropy coincide.