In probability and statistics, an exponential family is a parametric set of probability distributions of a certain form, specified below. This special form is chosen for mathematical convenience, including the enabling of the user to calculate expectations, covariances using differentiation based on some useful algebraic properties, as well as for generality, as exponential families are in a sense very natural sets of distributions to consider. The term exponential class is sometimes used in place of "exponential family", or the older term Koopman–Darmois family. Sometimes loosely referred to as the exponential family, this class of distributions is distinct because they all possess a variety of desirable properties, most importantly the existence of a sufficient statistic.
The concept of exponential families is credited to E. J. G. Pitman, G. Darmois, and B. O. Koopman in 1935–1936. Exponential families of distributions provide a general framework for selecting a possible alternative parameterisation of a parametric family of distributions, in terms of natural parameters, and for defining useful sample statistics, called the natural sufficient statistics of the family.
Nomenclature difficulty The terms "distribution" and "family" are often used loosely: Specifically, an exponential family is a set of distributions, where the specific distribution varies with the parameter; however, a parametric family of distributions is often referred to as "a distribution" (like "the normal distribution", meaning "the family of normal distributions"), and the set of all exponential families is sometimes loosely referred to as "the" exponential family.
Definition Most of the commonly used distributions form an exponential family or subset of an exponential family, listed in the subsection below. The subsections following it are a sequence of increasingly more general mathematical definitions of an exponential family. A casual reader may wish to restrict attention to the first and simplest definition, which corresponds to a single-parameter family of discrete or continuous probability distributions.
Examples of exponential family distributions Exponential families include many of the most common distributions. Among many others, exponential families include the following:
normal exponential gamma chi-squared beta Dirichlet Bernoulli categorical Poisson Wishart inverse Wishart geometric
A number of common distributions are exponential families, but only when certain parameters are fixed and known. For example: binomial (with fixed number of trials) multinomial (with fixed number of trials) negative binomial (with fixed number of failures) Note that in each case, the parameters which must be fixed are those that set a limit on the range of values that can possibly be observed.
Examples of common distributions that are not exponential families are Student's t, most mixture distributions, and even the family of uniform distributions when the bounds are not fixed. See the section below on examples for more discussion.
Scalar parameter The value of is called the parameter of the family.
A single-parameter exponential family is a set of probability distributions whose probability density function (or probability mass function, for the case of a discrete distribution) can be expressed in the form
where , , , and are known functions. The function must be non-negative.
An alternative, equivalent form often given is
or equivalently
In terms of log probability,
Note that and .
Support must be independent of Importantly, the support of (all the possible values for which is greater than ) is required to not depend on This requirement can be used to exclude a parametric family distribution from being an exponential family.
For example: The Pareto distribution has a pdf which is defined for (the minimum value, being the scale parameter) and its support, therefore, has a lower limit of Since the support of is dependent on the value of the parameter, the family of Pareto distributions does not form an exponential family of distributions (at least when is unknown).
Another example: Bernoulli-type distributions – binomial, negative binomial, geometric distribution, and similar – can only be included in the exponential class if the number of Bernoulli trials, , is treated as a fixed constant – excluded from the free parameter(s) – since the allowed number of trials sets the limits for the number of "successes" or "failures" that can be observed in a set of trials.
Vector valued and Often is a vector of measurements, in which case may be a function from the space of possible values of to the real numbers.
More generally, and can each be vector-valued such that is real-valued. However, see the discussion below on vector parameters, regarding the exponential family.
Canonical formulation If then the exponential family is said to be in canonical form. By defining a transformed parameter it is always possible to convert an exponential family to canonical form. The canonical form is non-unique, since can be multiplied by any nonzero constant, provided that is multiplied by that constant's reciprocal, or a constant can be added to and multiplied by to offset it. In the special case that and , then the family is called a natural exponential family.
Even when is a scalar, and there is only a single parameter, the functions and can still be vectors, as described below.
The function or equivalently is automatically determined once the other functions have been chosen, since it must assume a form that causes the distribution to be normalized (sum or integrate to one over the entire domain). Furthermore, both of these functions can always be written as functions of even when is not a one-to-one function, i.e. two or more different values of map to the same value of and hence cannot be inverted. In such a case, all values of mapping to the same will also have the same value for and
Factorization of the variables involved What is important to note, and what characterizes all exponential family variants, is that the parameter(s) and the observation variable(s) must factorize (can be separated into products each of which involves only one type of variable), either directly or within either part (the base or exponent) of an exponentiation operation. Generally, this means that all of the factors constituting the density or mass function must be of one of the following forms:
where and are arbitrary functions of , the observed statistical variable; and are arbitrary functions of the fixed parameters defining the shape of the distribution; and is any arbitrary constant expression (i.e. a number or an expression that does not change with either or ).
There are further restrictions on how many such factors can occur. For example, the two expressions:
are the same, i.e. a product of two "allowed" factors. However, when rewritten into the factorized form,
it can be seen that it cannot be expressed in the required form. (However, a form of this sort is a member of a curved exponential family, which allows multiple factorized terms in the exponent.)
To see why an expression of the form
qualifies,
and hence factorizes inside of the exponent. Similarly,
and again factorizes inside of the exponent.
A factor consisting of a sum where both types of variables are involved (e.g. a factor of the form ) cannot be factorized in this fashion (except in some cases where occurring directly in an exponent); this is why, for example, the Cauchy distribution and Student's t distribution are not exponential families.
Vector parameter The definition in terms of one real-number parameter can be extended to one real-vector parameter
A family of distributions is said to belong to a vector exponential family if the probability density function (or probability mass function, for discrete distributions) can be written as
or in a more compact form,
This form writes the sum as a dot product of vector-valued functions and .
An alternative, equivalent form often seen is
As in the scalar valued case, the exponential family is said to be in canonical form if
A vector exponential family is said to be curved if the dimension of
is less than the dimension of the vector
That is, if the dimension, , of the parameter vector is less than the number of functions, , of the parameter vector in the above representation of the probability density function. Most common distributions in the exponential family are not curved, and many algorithms designed to work with any exponential family implicitly or explicitly assume that the distribution is not curved.
Just as in the case of a scalar-valued parameter, the function or equivalently is automatically determined by the normalization constraint, once the other functions have been chosen. Even if is not one-to-one, functions and can be defined by requiring that the distribution is normalized for each value of the natural parameter . This yields the canonical form
or equivalently
The above forms may sometimes be seen with in place of . These are exactly equivalent formulations, merely using different notation for the dot product.
Vector parameter, vector variable The vector-parameter form over a single scalar-valued random variable can be trivially expanded to cover a joint distribution over a vector of random variables. The resulting distribution is simply the same as the above distribution for a scalar-valued random variable with each occurrence of the scalar replaced by the vector
The dimensions of the random variable need not match the dimension of the parameter vector, nor (in the case of a curved exponential function) the dimension of the natural parameter and sufficient statistic .
The distribution in this case is written as
Or more compactly as
Or alternatively as
Measure-theoretic formulation We use cumulative distribution functions (CDF) in order to encompass both discrete and continuous distributions.
Suppose is a non-decreasing function of a real variable. Then Lebesgue–Stieltjes integrals with respect to are integrals with respect to the reference measure of the exponential family generated by .
Any member of that exponential family has cumulative distribution function
is a Lebesgue–Stieltjes integrator for the reference measure. When the reference measure is finite, it can be normalized and is actually the cumulative distribution function of a probability distribution. If is absolutely continuous with a density with respect to a reference measure (typically Lebesgue measure), one can write . In this case, is also absolutely continuous and can be written so the formulas reduce to that of the previous paragraphs. If is discrete, then is a step function (with steps on the support of ).
Alternatively, we can write the probability measure directly as
for some reference measure .
Interpretation In the definitions above, the functions , , and were arbitrary. However, these functions have important interpretations in the resulting probability distribution.
is a sufficient statistic of the distribution. For exponential families, the sufficient statistic is a function of the data that holds all information the data provides with regard to the unknown parameter values. This means that, for any data sets and , the likelihood ratio is the same, that is if . This is true even if and are not equal to each other. The dimension of equals the number of parameters of and encompasses all of the information regarding the data related to the parameter . The sufficient statistic of a set of independent identically distributed data observations is simply the sum of individual sufficient statistics, and encapsulates all the information needed to describe the posterior distribution of the parameters, given the data (and hence to derive any desired estimate of the parameters). (This important property is discussed further below.)