This post gives a simple example of an exponential family that has natural parameter space being a point and that its natural sufficient statistic is not minimal.
Let us define the concept of exponential family with natural parameters.
Definition 1 (Natural exponential family)
A family of probability densities (or probability mass function) with parameter (index) is said to be a natural exponential family if can be written as
Here for , we call
- : natural sufficient statistic
- : natural parameter
- : natural parameter space.
A lot of well-known distributions actually are exponential families, e.g., normal distribution, Binomial distribution, Poisson distribution, beta distribution, and gamma distribution. The exponential family is central to the modeling and analysis of data.
Next, we define the concept of sufficiency and minimal sufficiency.
Definition 2 (Sufficiency and minimal sufficiency)
For a family of probability density , , and a random variable , a statistic
is sufficient if the conditional distribution is independent of . A sufficient statistic is minimal if for any other sufficient statistic , there is a function such that .
Intuitively, a sufficient statistic captures all the information of the underlying parameter . Indeed, suppose someone hands you a sufficient statistic . Because is independent of , you know the distribution already. Now if you can generate the data according to , then the unconditional distribution of is simply ! So even though you don’t know the underlying distribution , you can generate so long as is available.
The minimality of a sufficient statistic means the data is reduced in an optimal way. As all other sufficient statistics actually contain more information than needed. One thing to note is that this definition of minimality has nothing to say about the dimension of a minimal sufficient statistic. Indeed, if is minimal, then is
It is easily verified that natural sufficient statistic is actually sufficient using the factorization theorem. A natural question occurs at this point, is the natural sufficient statistic always minimal? The following example reveals that we do need to put a few more condition on or .
Consider the density family
The natural parameter space is the place where the integeral
is finite (we use Fubini’s theorem in the middle step). Hence the parameter needs to satisfy that
for the integral to be finite and for the integral to be finite. This means actually and so the natural parameter space is a single point .
The natural sufficient statistic is indeed sufficient. But any constant estimator is also sufficient and minimal as any other sufficient statistic under a constant function is a constant. But is not a constant estimator so we see the natural sufficient statistic is not always necessarily minimal.
It can be shown that so long as the natural parameter space contains an open set then the natural sufficient is indeed minimal. See Theorem 4.5 b) of this note.