Draft:Random Group Formation Distribution

Many real-world samples seem to follow a power-law distribution and come from N individuals being placed into M groups. An example is the number of people in each county. Another is number of people working at each company. Another, in a different area, is all the words in a document grouped by the word. (E.g., the word "the" occurs in 10 places, "of" occurs in 5 places, etc.)

While many other fat-tailed distributions have been used to fit these data samples, the RGF distribution tries to fit them by defining the information in the grouping and choosing the minimum information cost distribution.

Baek, Bernhardsson, and Minnhagen define the information in the grouping as:^[1]

${\displaystyle I_{cost}=\sum _{k}P(k)\ln(N(k)\cdot k)}$

where ${\displaystyle k}$ ranges over the number of members in a group, ${\displaystyle P(k)}$ is the probability of an individual being in a group of size ${\displaystyle k}$, ${\displaystyle \ln()}$ is the natural log, and ${\displaystyle N(k)}$ is the number of groups with ${\displaystyle k}$ members.

The resulting distribution is:

${\displaystyle P(k)=A\cdot \exp(-Bk)/k}$

where ${\displaystyle A}$ and ${\displaystyle B}$ are constants gotten by solving a Lagrangian equation with particular ${\displaystyle N}$ and ${\displaystyle M}$.

This distribution does not fit all real world samples. Baek, Bernhardsson, and Minnhagen generalize the definition by allowing some ordering to the grouping. That is realized with function ${\displaystyle S(P(k))}$ that computes the discounted entropy as a function of the distribution. In practice, this doesn't need to be calculated. The size of the largest group is sufficient to fit the discounted entropy. That distribution is:

${\displaystyle P(k)=A\cdot \exp(-Bk)/k^{\gamma }}$

The RGF is a maximum entropy distribution. Other ones include the normal distribution (when the mean and variance is known), the exponential distribution and Laplace distribution.

Matt Visser created a similar distribution.^[2] It is a maximum entropy distribution that generates a power law distribution, with a simpler constraint: ${\displaystyle \sum _{n}p_{n}log(n)}$.

The RGF distribution is not flat in a log-log graph like a power-law distributions. Data presented by Baek, Bernhardsson, and Minnhagen show that the curved RGF distribution matchs certain real-world samples better than the flat power-law distributions.^[1]