8.2: Probability Rules

8.2.1: The Addition Rule

Addition Law

The addition law of probability (sometimes referred to as the addition rule or sum rule), states that the probability that $\text{A}$ or $\text{B}$ will occur is the sum of the probabilities that $\text{A}$ will happen and that $\text{B}$ will happen, minus the probability that both $\text{A}$ and $\text{B}$will happen. The addition rule is summarized by the formula:

$\text{P}(\text{A}\cup \text{B})=\text{P}\left(\text{A}\right)+\text{P}\left(\text{B}\right)-\text{P}(\text{A}\cap \text{B})$

Consider the following example. When drawing one card out of a deck of $52$playing cards, what is the probability of getting heart or a face card (king, queen, or jack)? Let $\text{H}$ denote drawing a heart and $\text{F}$ denote drawing a face card. Since there are $13$ hearts and a total of $12$ face cards ($3$ of each suit: spades, hearts, diamonds and clubs), but only $3$ face cards of hearts, we obtain:

$\text{P}\left(\text{H}\right)=\frac{\mathrm{13/}}{52}$

$\text{P}\left(\text{F}\right)=\frac{\mathrm{12/}}{52}$

$\text{P}(\text{F}\cap \text{H})=\frac{\mathrm{3/}}{52}$

Using the addition rule, we get:

$\begin{array}{cc}\text{P}(\text{H}\cup \text{F})& =\text{P}\left(\text{H}\right)+\text{P}\left(\text{F}\right)-\text{P}(\text{H}\cap \text{F})\end{array}$

$\begin{array}{cc}& =(\frac{\mathrm{13/}}{\mathrm{52)}}+(\frac{\mathrm{12/}}{\mathrm{52)}}-\frac{\mathrm{3/}}{52}\end{array}$

The reason for subtracting the last term is that otherwise we would be counting the middle section twice (since $\text{H}$and $\text{F}$ overlap).

Addition Rule for Disjoint Events

Suppose $\text{A}$ and $\text{B}$ are disjoint, their intersection is empty. Then the probability of their intersection is zero. In symbols: $\text{P}(\text{A}\cap \text{B})=0$. The addition law then simplifies to:

$\text{P}(\text{A}\cup \text{B})=\text{P}\left(\text{A}\right)+\text{P}\left(\text{B}\right)\text{when}\text{A}\cap \text{B}=\varnothing$

The symbol $\varnothing$represents the empty set, which indicates that in this case $\text{A}$ and $\text{B}$ do not have any elements in common (they do not overlap).

8.2.2: The Multiplication Rule

The Multiplication Rule

In probability theory, the Multiplication Rule states that the probability that $\text{A}$ and $\text{B}$ occur is equal to the probability that $\text{A}$ occurs times the conditional probability that $\text{B}$ occurs, given that we know $\text{A}$has already occurred. This rule can be written:

$\text{P}(\text{A}\cap \text{B})=\text{P}\left(\text{B}\right)\cdot \text{P}\left(\text{A}|\text{B}\right)$

Switching the role of $\text{A}$and $\text{B}$, we can also write the rule as:

$\text{P}(\text{A}\cap \text{B})=\text{P}\left(\text{A}\right)\cdot \text{P}\left(\text{B}|\text{A}\right)$

We obtain the general multiplication rule by multiplying both sides of the definition of conditional probability by the denominator. That is, in the equation , if we multiply both sides by $\text{P}\left(\text{B}\right)$, we obtain the Multiplication Rule.

The rule is useful when we know both $\text{P}\left(\text{B}\right)$ and $\text{P}\left(\text{A}|\text{B}\right)$, or both $\text{P}\left(\text{A}\right)$ and $\text{P}\left(\text{B}|\text{A}\right).$

Independent Event

Note that when $\text{A}$ and $\text{B}$ are independent, we have that $\text{P}\left(\text{B}|\text{A}\right)=\text{P}\left(\text{B}\right)$, so the formula becomes $\text{P}(\text{A}\cap \text{B})=\text{P}\left(\text{A}\right)\text{P}\left(\text{B}\right)$, which we encountered in a previous section. As an example, consider the experiment of rolling a die and flipping a coin. The probability that we get a $2$ on the die and a tails on the coin is ($\frac{\mathrm{1/}}{\mathrm{6)}}\cdot (\frac{\mathrm{1/}}{\mathrm{2)}}=\frac{\mathrm{1/}}{12}$, since the two events are independent.

8.2.3: Independence

Independent Events

In probability theory, to say that two events are independent means that the occurrence of one does not affect the probability that the other will occur. In other words, if events $\text{A}$ and $\text{B}$ are independent, then the chance of $\text{A}$ occurring does not affect the chance of $\text{B}$ occurring and vice versa. The concept of independence extends to dealing with collections of more than two events.

Two events are independent if any of the following are true:

1. $\text{P}\left(\text{A}|\text{B}\right)=\text{P}\left(\text{A}\right)$
2. $\text{P}\left(\text{B}|\text{A}\right)=\text{P}\left(\text{B}\right)$
3. $\text{P}\left(\text{A}\text{and}\text{B}\right)=\text{P}\left(\text{A}\right)\cdot \text{P}\left(\text{B}\right)$

To show that two events are independent, you must show only one of the conditions listed above. If any one of these conditions is true, then all of them are true.

Translating the symbols into words, the first two mathematical statements listed above say that the probability for the event with the condition is the same as the probability for the event without the condition. For independent events, the condition does not change the probability for the event. The third statement says that the probability of both independent events $\text{A}$ and $\text{B}$ occurring is the same as the probability of $\text{A}$ occurring, multiplied by the probability of $\text{B}$occurring.

As an example, imagine you select two cards consecutively from a complete deck of playing cards. The two selections are not independent. The result of the first selection changes the remaining deck and affects the probabilities for the second selection. This is referred to as selecting “without replacement” because the first card has not been replaced into the deck before the second card is selected.

However, suppose you were to select two cards “with replacement” by returning your first card to the deck and shuffling the deck before selecting the second card. Because the deck of cards is complete for both selections, the first selection does not affect the probability of the second selection. When selecting cards with replacement, the selections are independent.

Consider a fair die role, which provides another example of independent events. If a person roles two die, the outcome of the first roll does not change the probability for the outcome of the second roll.

Example

When flipping a coin, what is the probability of getting tails 5 times in a row?

Recall that each coin flip is independent, and the probability of getting tails is 1/2 for any flip. Also recall that the following statement holds true for any two independent events A and B:

$\text{P}\left(\text{A}\text{and}\text{B}\right)=\text{P}\left(\text{A}\right)\cdot \text{P}\left(\text{B}\right)$

Finally, the concept of independence extends to collections of more than 2 events.

Therefore, the probability of getting tails 4 times in a row is:

8.2.4: Counting Rules and Techniques

Combinatorics is a branch of mathematics concerning the study of finite or countable discrete structures. Combinatorial techniques are applicable to many areas of mathematics, and a knowledge of combinatorics is necessary to build a solid command of statistics. It involves the enumeration, combination, and permutation of sets of elements and the mathematical relations that characterize their properties.

Aspects of combinatorics include: counting the structures of a given kind and size, deciding when certain criteria can be met, and constructing and analyzing objects meeting the criteria. Aspects also include finding “largest,” “smallest,” or “optimal” objects, studying combinatorial structures arising in an algebraic context, or applying algebraic techniques to combinatorial problems.

Combinatorial Rules and Techniques

Several useful combinatorial rules or combinatorial principles are commonly recognized and used. Each of these principles is used for a specific purpose. The rule of sum (addition rule), rule of product (multiplication rule), and inclusion-exclusion principle are often used for enumerative purposes. Bijective proofs are utilized to demonstrate that two sets have the same number of elements. Double counting is a method of showing that two expressions are equal. The pigeonhole principle often ascertains the existence of something or is used to determine the minimum or maximum number of something in a discrete context. Generating functions and recurrence relations are powerful tools that can be used to manipulate sequences, and can describe if not resolve many combinatorial situations. Each of these techniques is described in greater detail below.

Rule of Sum

The rule of sum is an intuitive principle stating that if there are $\text{a}$ possible ways to do something, and $\text{b}$ possible ways to do another thing, and the two things can’t both be done, then there are $\text{a}+\text{b}$ total possible ways to do one of the things. More formally, the sum of the sizes of two disjoint sets is equal to the size of the union of these sets.

Rule of Product

The rule of product is another intuitive principle stating that if there are $\text{a}$ ways to do something and $\text{b}$ ways to do another thing, then there are $\text{a}\cdot \text{b}$ ways to do both things.

Inclusion-Exclusion Principle

The inclusion-exclusion principle is a counting technique that is used to obtain the number of elements in a union of multiple sets. This counting method ensures that elements that are present in more than one set in the union are not counted more than once. It considers the size of each set and the size of the intersections of the sets. The smallest example is when there are two sets: the number of elements in the union of $\text{A}$ and $\text{B}$ is equal to the sum of the number of elements in $\text{A}$ and $\text{B}$, minus the number of elements in their intersection. See the diagram below for an example with three sets.

Bijective Proof

A bijective proof is a proof technique that finds a bijective function $\text{f}:\text{A}\to \text{B}$ between two finite sets $\text{A}$ and $\text{B}$, which proves that they have the same number of elements, $|\text{A}|=|\text{B}|$. A bijective function is one in which there is a one-to-one correspondence between the elements of two sets. In other words, each element in set $\text{B}$ is paired with exactly one element in set $\text{A}$. This technique is useful if we wish to know the size of $\text{A}$, but can find no direct way of counting its elements. If $\text{B}$ is more easily countable, establishing a bijection from $\text{A}$ to $\text{B}$ solves the problem.

Double Counting

Double counting is a combinatorial proof technique for showing that two expressions are equal. This is done by demonstrating that the two expressions are two different ways of counting the size of one set. In this technique, a finite set $\text{X}$ is described from two perspectives, leading to two distinct expressions for the size of the set. Since both expressions equal the size of the same set, they equal each other.

Pigeonhole Principle

The pigeonhole principle states that if $\text{a}$ items are each put into one of $\text{b}$ boxes, where $\text{a}>\text{b}$, then at least one of the boxes contains more than one item. This principle allows one to demonstrate the existence of some element in a set with some specific properties. For example, consider a set of three gloves. In such a set, there must be either two left gloves or two right gloves (or three of left or right). This is an application of the pigeonhole principle that yields information about the properties of the gloves in the set.

Generating Function

Generating functions can be thought of as polynomials with infinitely many terms whose coefficients correspond to the terms of a sequence. The (ordinary) generating function of a sequence ${\text{a}}_{\text{n}}$is given by:

whose coefficients give the sequence $\left\{{\text{a}}_0,{\text{a}}_1,{\text{a}}_2,\dots \right\}$.

Recurrence Relation

A recurrence relation defines each term of a sequence in terms of the preceding terms. In other words, once one or more initial terms are given, each of the following terms of the sequence is a function of the preceding terms.

The Fibonacci sequence is one example of a recurrence relation. Each term of the Fibonacci sequence is given by ${\text{F}}_{\text{n}}={\text{F}}_{\text{n}-1}+{\text{F}}_{\text{n}-2}$, with initial values ${\text{F}}_0=0$ and ${\text{F}}_1=1$. Thus, the sequence of Fibonacci numbers begins:

$0,1,1,2,3,5,8,13,21,34,55,89,\dots$

8.2.5: Bayes’ Rule

In probability theory and statistics, Bayes’ theorem (or Bayes’ rule ) is a result that is of importance in the mathematical manipulation of conditional probabilities. It is a result that derives from the more basic axioms of probability. When applied, the probabilities involved in Bayes’ theorem may have any of a number of probability interpretations. In one of these interpretations, the theorem is used directly as part of a particular approach to statistical inference. In particular, with the Bayesian interpretation of probability, the theorem expresses how a subjective degree of belief should rationally change to account for evidence. This is known as Bayesian inference, which is fundamental to Bayesian statistics.

Bayes’ rule relates the odds of event ${\text{A}}_1$to event ${\text{A}}_2$, before (prior to) and after (posterior to) conditioning on another event $\text{B}$. The odds on ${\text{A}}_1$ to event ${\text{A}}_2$ is simply the ratio of the probabilities of the two events. The relationship is expressed in terms of the likelihood ratio, or Bayes’ factor. By definition, this is the ratio of the conditional probabilities of the event $\text{B}$ given that ${\text{A}}_1$ is the case or that ${\text{A}}_2$is the case, respectively. The rule simply states:

Posterior odds equals prior odds times Bayes’ factor.

More specifically, given events ${\text{A}}_1$, ${\text{A}}_2$ and $\text{B}$, Bayes’ rule states that the conditional odds of ${\text{A}}_1:{\text{A}}_2$ given $\text{B}$ are equal to the marginal odds ${\text{A}}_1:{\text{A}}_2$multiplied by the Bayes factor or likelihood ratio. This is shown in the following formulas:

$O({\text{A}}_1:{\text{A}}_2|\text{B})=\Lambda ({\text{A}}_1:{\text{A}}_2|\text{B})\cdot O({\text{A}}_1:{\text{A}}_2)$

Where the likelihood ratio $\Lambda$ is the ratio of the conditional probabilities of the event $\text{B}$ given that ${\text{A}}_1$ is the case or that ${\text{A}}_2$is the case, respectively:

$\Lambda ({\text{A}}_1:{\text{A}}_2|\text{B})=\frac{\text{P}\left(\text{B}|{\text{A}}_1\right)}{\text{P}\left(\text{B}|{\text{A}}_2\right)}$

Bayes’ rule is widely used in statistics, science and engineering, such as in: model selection, probabilistic expert systems based on Bayes’ networks, statistical proof in legal proceedings, email spam filters, etc. Bayes’ rule tells us how unconditional and conditional probabilities are related whether we work with a frequentist or a Bayesian interpretation of probability. Under the Bayesian interpretation it is frequently applied in the situation where ${\text{A}}_1$and ${\text{A}}_2$ are competing hypotheses, and $\text{B}$ is some observed evidence. The rule shows how one’s judgement on whether ${\text{A}}_1$ or ${\text{A}}_2$ is true should be updated on observing the evidence.

Bayesian Inference

Bayesian inference is a method of inference in which Bayes’ rule is used to update the probability estimate for a hypothesis as additional evidence is learned. Bayesian updating is an important technique throughout statistics, and especially in mathematical statistics. Bayesian updating is especially important in the dynamic analysis of a sequence of data. Bayesian inference has found application in a range of fields including science, engineering, philosophy, medicine, and law.

Informal Definition

Rationally, Bayes’ rule makes a great deal of sense. If the evidence does not match up with a hypothesis, one should reject the hypothesis. But if a hypothesis is extremely unlikely a priori, one should also reject it, even if the evidence does appear to match up.

For example, imagine that we have various hypotheses about the nature of a newborn baby of a friend, including:

• H₁: The baby is a brown-haired boy.
• H₂: The baby is a blond-haired girl.
• H₃: The baby is a dog.

Then, consider two scenarios:

1. We’re presented with evidence in the form of a picture of a blond-haired baby girl. We find this evidence supports ${\text{H}}_2$ and opposes ${\text{H}}_1$ and ${\text{H}}_3$.
2. We’re presented with evidence in the form of a picture of a baby dog. Although this evidence, treated in isolation, supports ${\text{H}}_3$, my prior belief in this hypothesis (that a human can give birth to a dog) is extremely small. Therefore, the posterior probability is nevertheless small.

The critical point about Bayesian inference, then, is that it provides a principled way of combining new evidence with prior beliefs, through the application of Bayes’ rule. Furthermore, Bayes’ rule can be applied iteratively. After observing some evidence, the resulting posterior probability can then be treated as a prior probability, and a new posterior probability computed from new evidence. This allows for Bayesian principles to be applied to various kinds of evidence, whether viewed all at once or over time. This procedure is termed Bayesian updating.

8.2.6: The Collins Case

The People of the State of California v. Collins was a 1968 jury trial in California. It made notorious forensic use of statistics and probability. Bystanders to a robbery in Los Angeles testified that the perpetrators had been a black male, with a beard and moustache, and a caucasian female with blonde hair tied in a ponytail. They had escaped in a yellow motor car.

The prosecutor called upon for testimony an instructor in mathematics from a local state college. The instructor explained the multiplication rule to the jury, but failed to give weight to independence, or the difference between conditional and unconditional probabilities. The prosecutor then suggested that the jury would be safe in estimating the following probabilities:

• Black man with beard: 1 in 10
• Man with moustache: 1 in 4
• White woman with pony tail: 1 in 10
• White woman with blonde hair: 1 in 3
• Yellow motor car: 1 in 10
• Interracial couple in car: 1 in 1000

These probabilities, when considered together, result in a 1 in 12,000,000 chance that any other couple with similar characteristics had committed the crime – according to the prosecutor, that is. The jury returned a verdict of guilty.

As seen in , upon appeal, the Supreme Court of California set aside the conviction, criticizing the statistical reasoning and disallowing the way the decision was put to the jury. In their judgment, the justices observed that mathematics:

… while assisting the trier of fact in the search of truth, must not cast a spell over him.

Prosecutor’s Fallacy

The Collins’ case is a prime example of a phenomenon known as the prosecutor’s fallacy—a fallacy of statistical reasoning when used as an argument in legal proceedings. At its heart, the fallacy involves assuming that the prior probability of a random match is equal to the probability that the defendant is innocent. For example, if a perpetrator is known to have the same blood type as a defendant (and 10% of the population share that blood type), to argue solely on that basis that the probability of the defendant being guilty is 90% makes the prosecutors’s fallacy (in a very simple form).

The basic fallacy results from misunderstanding conditional probability, and neglecting the prior odds of a defendant being guilty before that evidence was introduced. When a prosecutor has collected some evidence (for instance, a DNA match) and has an expert testify that the probability of finding this evidence if the accused were innocent is tiny, the fallacy occurs if it is concluded that the probability of the accused being innocent must be comparably tiny. If the DNA match is used to confirm guilt that is otherwise suspected, then it is indeed strong evidence. However, if the DNA evidence is the sole evidence against the accused, and the accused was picked out of a large database of DNA profiles, then the odds of the match being made at random may be reduced. Therefore, it is less damaging to the defendant. The odds in this scenario do not relate to the odds of being guilty; they relate to the odds of being picked at random.

Attributions

• The Addition Rule
  • “Boundless.”
    http://www.boundless.com/.
    Boundless Learning
    CC BY-SA 3.0.
  • “Probability axioms.”
    http://en.wikipedia.org/wiki/Probability_axioms.
    Wikipedia
    CC BY-SA 3.0.
  • “addition rule.”
    http://en.wikipedia.org/wiki/addition%20rule.
    Wikipedia
    CC BY-SA 3.0.
  • “probability.”
    http://en.wiktionary.org/wiki/probability.
    Wiktionary
    CC BY-SA 3.0.
  • “Probability.”
    https://en.wikipedia.org/wiki/Probability.
    Wikipedia
    CC BY-SA 3.0.
  • “Some rules of probability – Statistics.”
    http://statistics.wikidot.com/ch5.
    Wikidot
    CC BY-SA.
• The Multiplication Rule
  • “Boundless.”
    http://www.boundless.com/.
    Boundless Learning
    CC BY-SA 3.0.
  • “sample space.”
    http://en.wiktionary.org/wiki/sample_space.
    Wiktionary
    CC BY-SA 3.0.
  • “Some rules of probability – Statistics.”
    http://statistics.wikidot.com/ch5.
    Wikidot
    CC BY-SA.
• Independence
  • “Boundless.”
    http://www.boundless.com/.
    Boundless Learning
    CC BY-SA 3.0.
  • “Independence (probability theory).”
    http://en.wikipedia.org/wiki/Independence_(probability_theory).
    Wikipedia
    CC BY-SA 3.0.
  • “independence.”
    http://en.wikipedia.org/wiki/independence.
    Wikipedia
    CC BY-SA 3.0.
  • “probability theory.”
    http://en.wiktionary.org/wiki/probability_theory.
    Wiktionary
    CC BY-SA 3.0.
  • “Roberta Bloom, Probability Topics: Independent & Mutually Exclusive Events (modified R. Bloom). September 17, 2013.”
    http://cnx.org/content/m18868/latest/.
    OpenStax CNX
    CC BY 3.0.
  • “All sizes | Ace of Spades Card Deck Trick Magic Macro 10-19-09 2 | Flickr – Photo Sharing!.”
    http://www.flickr.com/photos/stevendepolo/4028160820/sizes/o/in/photostream/.
    Flickr
    CC BY.
• Counting Rules and Techniques
  • “Boundless.”
    http://www.boundless.com/.
    Boundless Learning
    CC BY-SA 3.0.
  • “Combinatorics.”
    http://en.wikipedia.org/wiki/Combinatorics.
    Wikipedia
    CC BY-SA 3.0.
  • “Pigeonhole principle.”
    https://en.wikipedia.org/wiki/Pigeonhole_principle.
    Wikipedi
    CC BY-SA 3.0.
  • “Bijective proof.”
    http://en.wikipedia.org/wiki/Bijective_proof.
    Wikipedia
    CC BY-SA 3.0.
  • “Double counting (proof technique).”
    http://en.wikipedia.org/wiki/Double_counting_(proof_technique).
    Wikipedia
    CC BY-SA 3.0.
  • “Bijection.”
    https://en.wikipedia.org/wiki/Bijection.
    Wikipedia
    CC BY-SA 3.0.
  • “Inclusion-exclusion principle.”
    https://en.wikipedia.org/wiki/Inclusion%E2%80%93exclusion_principle.
    Wikipedia
    CC BY-SA 3.0.
  • “polynomial.”
    http://en.wiktionary.org/wiki/polynomial.
    Wiktionary
    CC BY-SA 3.0.
  • “combinatorics.”
    http://en.wiktionary.org/wiki/combinatorics.
    Wiktionary
    CC BY-SA 3.0.
  • “Combinatorial principles.”
    http://en.wikipedia.org/wiki/Combinatorial_principles.
    Wikipedia
    CC BY-SA 3.0.
• Bayes’ Rule
  • “Boundless.”
    http://www.boundless.com/.
    Boundless Learning
    CC BY-SA 3.0.
  • “Bayes’ rule.”
    http://en.wikipedia.org/wiki/Bayes’_rule.
    Wikipedia
    CC BY-SA 3.0.
  • “Bayesian inference.”
    http://en.wikipedia.org/wiki/Bayesian_inference.
    Wikipedia
    CC BY-SA 3.0.
  • “Bayes’ theorem.”
    http://en.wikipedia.org/wiki/Bayes’_theorem.
    Wikipedia
    CC BY-SA 3.0.
  • “Bayes’ factor.”
    http://en.wikipedia.org/wiki/Bayes’%20factor.
    Wikipedia
    CC BY-SA 3.0.
  • “Bayes’ Theorem MMB 01.”
    http://commons.wikimedia.org/wiki/File:Bayes’_Theorem_MMB_01.jpg.
    Wikimedia
    CC BY-SA.
• The Collins Case
  • “Boundless.”
    http://www.boundless.com/.
    Boundless Learning
    CC BY-SA 3.0.
  • “Prosecutor’s fallacy.”
    http://en.wikipedia.org/wiki/Prosecutor’s_fallacy.
    Wikipedia
    CC BY-SA 3.0.
  • “People v. Collins.”
    http://en.wikipedia.org/wiki/People_v._Collins.
    Wikipedia
    CC BY-SA 3.0.
  • “prosecutor’s fallacy.”
    http://en.wikipedia.org/wiki/prosecutor’s%20fallacy.
    Wikipedia
    CC BY-SA 3.0.
  • “CA SC seal.”
    http://commons.wikimedia.org/wiki/File:CA_SC_seal.png.
    Wikimedia
    Public domain.