Generative Model

MachineLearningProbabilityTheoryStatistics A primary goal of Machine Learning is to represent high-dimensional signals in a way that best captures the relationships between parts of the signals in the most compressed way possible. In particular we wish to model the joint probability of features and targets: In discriminative models, one models , the probability of a target given some features, but a generative model requires that one learn and . The primary task of generative modeling is to determine how to both parameterize and simplify (via conditional independencies) the conditionals given by the chain rule: Given :

• : the data probability space
• : a parameterized family of probability measures
• : a distance between probability measures a generative model, , is an approximation of the data distribution, such that: The generative model consists of two key components:

1. Latent representation: The choice of joint distribution, encapsulated by
2. Learning: How to compare probability distributions, encapsulated by

One can control the output of a generative model by conditioning on “control signals” by modeling , which can be useful in using generative models for Inverse Problems or Imitation Learning.

We can use a generative model for

• Sampling
• Density estimation, such that is concentrated on likely ‘s
• Representation/unsupervised learning by capturing the common features of the ‘s

The primary challenge in generative modeling is the curse of dimensionality present in representing a high-dimensional distribution, i.e. the interactions/correlations between random variables. This can be reduced by utilizing local conditional distributions vs structured probabilistic graphical models.

Examples:

• Images A full generative model over an 8-bit image would require parameters to specify the full joint distribution over pixel-color space.
• Audio A full generative model over seconds of 44.1 kHz, 24-bit audio would require parameters to specify the full joint distribution over sample-audio level space.