Bayes' theorem

Medical diagnosis

Suppose that a doctor is testing a patient for the presence of a certain disease. The patient either has the disease or does not; the test returns either positive or negative. If the patient does not have the disease but the test returns a positive result, that is a false positive. If the patient has the disease and the test returns a positive result, that is a true positive. Bayes' theorem gives the means to calculate the probability that the patient has the disease given a positive test result, using quantities that specify how prevalent the disease is in the population and how well the test works. Let ${\displaystyle E}$ be the event that the patient has the disease. ${\displaystyle P(E)}$ is the probability that the patient has the disease. Let ${\displaystyle F}$ be the event that the patient tests positive. The probability that the patient has the disease given that they test positive is thus denoted ${\displaystyle P(E|F)}$. Bayes' theorem states:$${\displaystyle P(E|F)={\frac {P(F|E)P(E)}{P(F|E)P(E)+P(F|\neg E)P(\neg E)}}.}$$Here, ${\displaystyle P(E)}$ is the prevalence rate of the disease, and ${\displaystyle P(F|E)}$ is the true positive rate or test sensitivity.^{[20]: 44–45}

For example, if all patients with pancreatic cancer have certain symptoms, it does not follow that anyone who has those symptoms has pancreatic cancer. Assuming the incidence rate of pancreatic cancer is 1/100000, while 10/99999 healthy people have the symptoms, the probability that a person who has the symptoms has pancreatic cancer is 9.1%.

Based on incidence rate, the following table presents the corresponding numbers per 100,000 people.

These numbers can then be used to calculate the probability that a patient who has the symptoms has cancer:

${\displaystyle {\begin{aligned}P({\text{Cancer}}|{\text{Symptoms}})&={\frac {P({\text{Symptoms}}|{\text{Cancer}})P({\text{Cancer}})}{P({\text{Symptoms}})}}\\&={\frac {P({\text{Symptoms}}|{\text{Cancer}})P({\text{Cancer}})}{P({\text{Symptoms}}|{\text{Cancer}})P({\text{Cancer}})+P({\text{Symptoms}}|{\text{Non-Cancer}})P({\text{Non-Cancer}})}}\\[8pt]&={\frac {1\times 0.00001}{1\times 0.00001+(10/99999)\times 0.99999}}={\frac {1}{11}}\approx 9.1\%\end{aligned}}}$

Drug testing

Suppose a particular test for whether someone has been using a drug (e.g., Substance D) is 99% sensitive, meaning the true positive rate (TPR) = 0.99. The test then has 99% true positive results (correct identification of drug use) for drug users. The test is also 99% specific, meaning its true negative rate (TNR) = 0.99. Therefore, the test correctly identifies 99% of non-use for non-users, but also generates 1% false positives, or false positive rate (FPR) = 0.01, for non-users. Assuming 0.3% of people use the drug, Bayes' theorem gives the probability that a random person who tests positive is a drug user:$${\displaystyle {\begin{aligned}P({\text{User}}\vert {\text{Positive}})&={\frac {P({\text{Positive}}\vert {\text{User}})P({\text{User}})}{P({\text{Positive}})}}\\&={\frac {P({\text{Positive}}\vert {\text{User}})P({\text{User}})}{P({\text{Positive}}\vert {\text{User}})P({\text{User}})+P({\text{Positive}}\vert {\text{Non-user}})P({\text{Non-user}})}}\\[8pt]&={\frac {0.99\times 0.003}{0.99\times 0.003+0.01\times 0.997}}\approx 23\%.\end{aligned}}}$$Consequently, even though the drug test is "99% accurate", most of its positive results will be false.^[21]

Bent coins

An urn contains coins of three different types: A, B, and C. Coins of type A are fair and, when flipped, come up heads with probability 0.5. Coins of type B are biased and have probability 0.6 of turning up heads, and type-C coins come up heads with probability 0.9. The urn contains 2 type-A coins, 2 type-B coins and 1 type-C coin. A coin is drawn at random from the urn and flipped. Bayes' theorem gives the probability of a coin being of a given type given that it comes up heads:$${\displaystyle P(A|H)={\frac {P(H|A)P(A)}{P(H)}},}$$and likewise for ${\displaystyle P(B|H)}$ and ${\displaystyle P(C|H)}$. The denominator can be found via the law of total probability:$${\displaystyle P(H)=P(H|A)P(A)+P(H|B)P(B)+P(H|C)P(C).}$$Assuming the coins are drawn from the urn at random, ${\displaystyle P(A)=2/5}$, ${\displaystyle P(B)=2/5}$, and ${\displaystyle P(C)=1/5}$. It follows that ${\displaystyle P(H)=0.62}$, and^[22]$${\displaystyle P(A|H)=0.2/0.62\approx 32\%.}$$

Bayesian interpretations

In Bayesian (or epistemological) interpretations, probability measures a "degree of belief".^[23][24] Bayes' theorem links the degree of belief in a proposition before and after accounting for evidence. For example, suppose it is believed with 50% certainty that a coin is twice as likely to land heads than tails. If the coin is flipped a number of times and the outcomes observed, that degree of belief will probably rise or fall, but might remain the same, depending on the results. For proposition A and evidence B,

• P (A), the prior, is the initial degree of belief in A.
• P (A|B), the posterior, is the degree of belief after incorporating news that B is true.
• the quotient ⁠P(B|A)/P(B)⁠ represents the support B provides for A.

For more on the application of Bayes' theorem under Bayesian interpretations of probability, see Bayesian inference.

Frequentist interpretations

In the frequentist interpretations, probability measures a "proportion of outcomes".^[25] For example, suppose an experiment is performed many times. P(A) is the proportion of outcomes with property A (the prior) and P(B) is the proportion with property B. P(B|A) is the proportion of outcomes with property B out of outcomes with property A, and P(A|B) is the proportion of those with A out of those with B (the posterior).

The role of Bayes' theorem can be shown with tree diagrams. The two diagrams partition the same outcomes by A and B in opposite orders, to obtain the inverse probabilities. Bayes' theorem links the different partitionings.

Example

An entomologist spots what might, due to the pattern on its back, be a rare subspecies of beetle. A full 98% of the members of the rare subspecies have the pattern, so P(Pattern|Rare) = 98%. Only 5% of members of the common subspecies have the pattern. The rare subspecies is 0.1% of the total population. How likely is the beetle having the pattern to be rare: what is P(Rare|Pattern)?

Since any beetle is either rare or common, Bayes' theorem can be applied as follows: $${\displaystyle {\begin{aligned}P({\text{Rare}}\vert {\text{Pattern}})&={\frac {P({\text{Pattern}}\vert {\text{Rare}})\,P({\text{Rare}})}{P({\text{Pattern}})}}\\[8pt]&={\tfrac {P({\text{Pattern}}\vert {\text{Rare}})\,P({\text{Rare}})}{P({\text{Pattern}}\vert {\text{Rare}})\,P({\text{Rare}})+P({\text{Pattern}}\vert {\text{Common}})\,P({\text{Common}})}}\\[8pt]&={\frac {0.98\times 0.001}{0.98\times 0.001+0.05\times 0.999}}\\[8pt]&\approx 1.9\%\end{aligned}}}$$

Events

For two competing hypotheses

More information ⁠ ...

Contingency table

Background Proposition	B	⁠${\displaystyle \lnot B}$⁠ (not B)	Total
A	${\displaystyle P(B|A)\cdot P(A)}$ ${\displaystyle =P(A|B)\cdot P(B)}$	${\displaystyle P(\neg B|A)\cdot P(A)}$ ${\displaystyle =P(A|\neg B)\cdot P(\neg B)}$	⁠${\displaystyle P(A)}$⁠
⁠${\displaystyle \neg A}$⁠ (not A)	${\displaystyle P(B|\neg A)\cdot P(\neg A)}$ ${\displaystyle =P(\neg A|B)\cdot P(B)}$	${\displaystyle P(\neg B|\neg A)\cdot P(\neg A)}$ ${\displaystyle =P(\neg A|\neg B)\cdot P(\neg B)}$	${\displaystyle P(\neg A)}$= ${\displaystyle 1-P(A)}$
Total	⁠${\displaystyle P(B)}$⁠	${\displaystyle P(\neg B)=1-P(B)}$	1

Close

The events ${\displaystyle A}$ and not-${\displaystyle A}$, written ${\displaystyle \neg A}$, are a mutually exclusive and exhaustive pair. Therefore the previous formula can be applied, and the sum in the denominator will have only two terms:

${\displaystyle P(A|B)={\frac {P(B|A)P(A)}{P(B|A)P(A)+P(B|\neg A)P(\neg A)}}.}$

If A denotes a proposition and B the evidence or background B, then^[27]

• ${\displaystyle P(A)}$ is the prior probability, the initial degree of belief in A.
• ${\displaystyle P(\neg A)}$ is the corresponding initial degree of belief that A is false, which is ${\displaystyle P(\neg A)=1-P(A)}$
• ${\displaystyle P(B|A)}$ is the conditional probability or likelihood, the degree of belief in B given that A is true.
• ${\displaystyle P(B|\neg A)}$ is the conditional probability or likelihood, the degree of belief in B given that A is false.
• ${\displaystyle P(A|B)}$ is the posterior probability, the probability of A after taking into account B.

Random variables

For two continuous random variables X and Y, Bayes' theorem may be analogously derived from the definition of conditional density:

${\displaystyle f_{X\vert Y=y}(x)={\frac {f_{X,Y}(x,y)}{f_{Y}(y)}}}$

${\displaystyle f_{Y\vert X=x}(y)={\frac {f_{X,Y}(x,y)}{f_{X}(x)}}}$

Bayes' theorem states:^{[26]: 20–21}

${\displaystyle f_{X\vert Y=y}(x)={\frac {f_{Y\vert X=x}(y)f_{X}(x)}{f_{Y}(y)}}.}$

This holds for values ${\displaystyle x}$ and ${\displaystyle y}$ within the support of X and Y, ensuring ${\displaystyle f_{X}(x)>0}$ and ${\displaystyle f_{Y}(y)>0}$.

Bayes' rule in odds form

Probabilities are sometimes specified in terms of odds. For any proposition ${\displaystyle A}$, the ratio of the probability that ${\displaystyle A}$ is true to the probability that ${\displaystyle A}$ is false is called the odds on ${\displaystyle A}$.^[32]: 90 Bayes' theorem can be written using odds as follows. First, apply Bayes' theorem to the probability that ${\displaystyle A}$ is true given some other proposition ${\displaystyle B}$:$${\displaystyle P(A|B)=P(B){\frac {P(B|A)}{P(A)}}.}$$Likewise, Bayes' theorem holds for the probability that ${\displaystyle A}$ is false:$${\displaystyle P(\neg A|B)=P(B){\frac {P(B|\neg A)}{P(\neg A)}}.}$$Dividing these two equations, the probability ${\displaystyle P(B)}$ drops out:$${\displaystyle {\frac {P(A|B)}{P(\neg A|B)}}={\frac {P(A)}{P(\neg A)}}{\frac {P(B|A)}{P(B|\neg A)}}.}$$Consequently, the odds of ${\displaystyle A}$ given ${\displaystyle B}$ are the odds of ${\displaystyle A}$ multiplied by the ratio ${\displaystyle P(B|A)/P(B|\neg A)}$, a quantity called the Bayes factor or likelihood ratio. In symbols:$${\displaystyle O(A|B)=O(A){\frac {P(B|A)}{P(B|\neg A)}}.}$$This is often summarized by saying that the posterior odds are the prior odds times the likelihood.^{[32]: 90 [20]: 49}

For example, suppose that a disease has a prevalence of 1%, i.e., the probability that a randomly chosen person is infected is 0.01. And suppose that a test for the disease returns the correct result with 99% probability. The prior odds of being infected are$${\displaystyle O({\text{infected}})={\frac {P({\text{infected}})}{P({\text{not infected}})}}={\frac {0.01}{1-0.01}}={\frac {1}{99}}.}$$

The probability that the test is positive given an infection is 99%:$${\displaystyle P({\text{positive}}|{\text{infected}})=0.99.}$$The probability that the test is negative given absence of an infection is also 99%, and so$${\displaystyle P({\text{positive}}|{\text{not infected}})=1-P({\text{negative}}|{\text{not infected}})=1-0.99=0.01.}$$The Bayes factor is the ratio$${\displaystyle {\frac {P({\text{positive}}|{\text{infected}})}{P({\text{positive}}|{\text{not infected}})}}={\frac {0.99}{0.01}}={\frac {99}{1}}.}$$Therefore, the posterior odds of an infection given a positive test result are$${\displaystyle {\frac {1}{99}}\times {\frac {99}{1}}={\frac {1}{1}};}$$in other words, the posterior odds of infection are one to one, and the posterior probability of infection is 50%.^[33]

Bayes' theorem for 3 events

A version of Bayes' theorem for 3 events^[34] results from the addition of a third event ${\displaystyle C}$, with ${\displaystyle P(C)>0,}$ on which all probabilities are conditioned:

${\displaystyle P(A\vert B\cap C)={\frac {P(B\vert A\cap C)\,P(A\vert C)}{P(B\vert C)}}}$

This can be deduced as follows. Using the chain rule

${\displaystyle P(A\cap B\cap C)=P(A\vert B\cap C)\,P(B\vert C)\,P(C)}$

And, on the other hand

${\displaystyle P(A\cap B\cap C)=P(B\cap A\cap C)=P(B\vert A\cap C)\,P(A\vert C)\,P(C)}$

The desired result is obtained by identifying both expressions and solving for ${\displaystyle P(A\vert B\cap C)}$.

Inference rules

In subjective interpretations of probability theory, an event's probability is regarded as an agent's belief that the event will happen. Bayes' theorem is widely invoked to justify how an agent should update or modify their beliefs after receiving new information. If an agent assigns the probability ${\displaystyle P(A)}$ to the event ${\displaystyle A}$, ${\displaystyle P(B)}$ to the event ${\displaystyle B}$, and ${\displaystyle P(B|A)}$ to the event "${\displaystyle B}$ happens assuming ${\displaystyle A}$ has already happened", Bayes' theorem gives the value of ${\displaystyle P(A|B)}$. Suppose that, after the agent has made these probability assignments, event ${\displaystyle B}$ occurs. An agent who follows Bayesian updating, also known as updating by conditionalization, will change their probability for event ${\displaystyle A}$ from the old value ${\displaystyle P(A)}$ to the new value ${\displaystyle P'(A)=P(A|B)}$. Conditionalization is not the only updating rule that might be considered rational.^[35][36][37] The issue of which assumptions can be invoked to constrain updating rules remains somewhat controversial.^[38][39]

Quantum

In quantum mechanics, probability distributions are generalized to density matrices, arrays of complex numbers that describe the preparation of a quantum system. A quantum Bayes rule can be formulated that expresses how density matrices are updated as new experimental data about a system is obtained.^[40][41]

Parameter estimation

In Bayesian statistics, an object or system is represented by a mathematical model that contains one or more numerical parameters. The available information about the system is used to write a probability distribution over the possible values of those parameters, indicating which values are more plausible than others. Bayes' theorem is then used to update that probability distribution as new evidence is obtained.^[42]

Recreational mathematics

Bayes' rule and computing conditional probabilities provide a method to solve a number of popular puzzles, such as the Three Prisoners problem,^[43] the Monty Hall problem,^[44] the Boy or Girl paradox,^[45] and the two envelopes problem.^[46]

Cryptanalysis

Alan Turing and his collaborators at Bletchley Park pioneered the use of Bayes' theorem for breaking ciphers during World War II. In 1941, Turing wrote a set of pedagogical notes introducing Bayesian methods of cryptanalysis by applying them to problems like solving a Vigenère cipher, which were simpler than his team's primary focus, reading messages encrypted by the Enigma machine. Bayes' theorem was also applied to the cracking of the Japanese naval code JN 25.^[47][48][49]

Genetics

In genetics, Bayes' rule can be used to estimate the probability that someone has a specific genotype. Many people seek to assess their chances of being affected by a genetic disease or their likelihood of being a carrier for a recessive gene of interest. A Bayesian analysis can be done based on family history or genetic testing to predict whether someone will develop a disease or pass one on to their children.^[50] (Genetic testing and prediction is common among couples who plan to have children but are concerned that they may both be carriers for a disease, especially in communities with low genetic variance.^[51])

Evolutionary biology

Bayes' theorem is employed in evolutionary biology to deduce trees of relationships among species that best explain the available data. This approach was made practical by the use of Markov chain Monte Carlo methods.^[52][53]