Probability and Statistics — Lecture 2

#probability-statistics #conditional-probability #bayes #independence #permutations #combinations #EE2M1

Conditional Probability and Bayes' Rule a

Summary

This lecture focuses on:

Conditional probability
Multiplication rule
Law of total probability
Bayes' rule
Independence of events

It also briefly reviews permutations and combinations. :contentReference[oaicite:0]

1. Permutations and Combinations

Permutations

For a permutation, the order matters.

If we arrange all $n$ distinct objects:

n! = n (n - 1) (n - 2) \dots 2 \cdot 1

Example

25 students are placed in a queue:

25!

If 3 of the 25 students sit on three different chairs:

25 \cdot 24 \cdot 23

because there are:

25 choices for the first chair
24 choices for the second chair
23 choices for the third chair

Combinations

For a combination, the order does not matter.

The number of ways to choose $k$ objects from $n$ is:

(\binom{n}{k}) = \frac{n!}{k! (n - k)!}

Example

3 out of 25 students are allowed to enter the museum shop:

(\binom{25}{3})

Tip

Permutation: who goes where → order matters
Combination: who is selected → order does not matter

2. Probability of an Intersection

For two events $A$ and $B$ :

P (A \cap B) = P (A) + P (B) - P (A \cup B)

This follows from the sum rule:

P (A \cup B) = P (A) + P (B) - P (A \cap B)

However,

P (A \cap B) = P (A) P (B)

is not always true.

It is only true when $A$ and $B$ are independent.

3. Conditional Probability

Definition

The probability of $A$ , given that $C$ has occurred, is:

P (A ∣ C) = \frac{P (A \cap C)}{P (C)}

provided that:

P (C) > 0

Interpretation

Knowing that $C$ occurred means we restrict the original sample space $Ω$ to $C$ .

The conditional probability is then the probability of $A$ in this reduced sample space.

Important

$P (A ∣ C)$ means:

"the probability of $A$ , given $C$ ."

Example: Throwing a Die

A die is thrown twice.

Without any extra information:

P (first throw = 5) = \frac{1}{6}

Now suppose we are told that:

X_{1} + X_{2} = 9

The possible outcomes are:

(3, 6), (4, 5), (5, 4), (6, 3)

Only one of these has first throw equal to 5.

Therefore:

P (X_{1} = 5 ∣ X_{1} + X_{2} = 9) = \frac{1}{4}

The extra information changes the relevant sample space.

4. The Difference Between $P (A ∣ B)$ and $P (B ∣ A)$

In general:

P (A ∣ B) \neq P (B ∣ A)

The lecture illustrates this using teachers and park rangers.

Even if:

P (likes nature ∣ park ranger) > P (likes nature ∣ teacher)

this does not imply:

P (park ranger ∣ likes nature) > P (teacher ∣ likes nature)

because there may be many more teachers than park rangers.

Warning

Never confuse:

P (A ∣ B)

with

P (B ∣ A)

5. Multiplication Rule

Starting from the definition:

P (A ∣ C) = \frac{P (A \cap C)}{P (C)}

we obtain:

P (A \cap C) = P (A ∣ C) P (C)

Equivalently:

P (A \cap C) = P (C ∣ A) P (A)

Example: Pebbles

A box contains:

10 pebbles
3 granite pebbles

Two pebbles are selected without replacement.

Let:

$C =$ first pebble is granite
$A =$ second pebble is granite

Then:

P (C) = \frac{3}{10}

After one granite pebble has been selected:

9 pebbles remain
2 granite pebbles remain

so:

P (A ∣ C) = \frac{2}{9}

Using the multiplication rule:

P (A \cap C) = P (A ∣ C) P (C)

= \frac{2}{9} \cdot \frac{3}{10}

= \frac{1}{15}

6. Summary of Probability Rules

Sample Space

P (Ω) = 1

Sum Rule

P (A \cup B) = P (A) + P (B) - P (A \cap B)

Complement Rule

P (A^{c}) = 1 - P (A)

Multiplication Rule

P (A \cap B) = P (A ∣ B) P (B)

De Morgan's Laws

(A \cap B)^{c} = A^{c} \cup B^{c}

(A \cup B)^{c} = A^{c} \cap B^{c}

Therefore:

P ((A \cap B)^{c}) = P (A^{c} \cup B^{c})

and:

P ((A \cup B)^{c}) = P (A^{c} \cap B^{c})

7. Law of Total Probability

Suppose the sample space is split into:

C and C^{c}

Then event $A$ can occur through either part:

A = (A \cap C) \cup (A \cap C^{c})

Therefore:

P (A) = P (A \cap C) + P (A \cap C^{c})

Using the multiplication rule:

P (A) = P (A ∣ C) P (C) + P (A ∣ C^{c}) P (C^{c})

General Form

Suppose:

C_{1}, C_{2}, \dots, C_{m}

are disjoint and fill the whole sample space:

C_{i} \cap C_{j} = \emptyset i \neq j

and:

C_{1} \cup C_{2} \cup \dots \cup C_{m} = Ω

Then:

P (A) = \sum_{i = 1}^{m} P (A ∣ C_{i}) P (C_{i})

Example: Production Machines

Three machines produce resistors:

Machine	Accuracy	Production share
$C_{1}$	80%	30%
$C_{2}$	90%	50%
$C_{3}$	60%	20%

Let:

A = {resistor is accurate}

Then:

P (A) = P (A ∣ C_{1}) P (C_{1}) + P (A ∣ C_{2}) P (C_{2}) + P (A ∣ C_{3}) P (C_{3})

= 0.8 (0.3) + 0.9 (0.5) + 0.6 (0.2)

= 0.24 + 0.45 + 0.12

P (A) = 0.81

So a randomly selected resistor has an 81% probability of being accurate.

8. Bayes' Rule

Bayes' rule is used when we want to reverse the direction of a conditional probability.

For example, we may know:

P (A ∣ C)

but want:

P (C ∣ A)

From:

P (C ∣ A) = \frac{P (A \cap C)}{P (A)}

and:

P (A \cap C) = P (A ∣ C) P (C)

we get:

P (C ∣ A) = \frac{P (A ∣ C) P (C)}{P (A)}

Using the law of total probability:

P (C ∣ A) = \frac{P (A ∣ C) P (C)}{P (A ∣ C) P (C) + P (A ∣ C^{c}) P (C^{c})}

9. Bayes' Rule for Multiple Causes

If:

C_{1}, \dots, C_{m}

form a partition of the sample space, then:

P (C_{i} ∣ A) = \frac{P (A ∣ C_{i}) P (C_{i})}{\sum_{j = 1}^{m} P (A ∣ C_{j}) P (C_{j})}

The numerator is:

P (A ∣ C_{i}) P (C_{i})

which represents the probability that:

cause $C_{i}$ occurs
and observation $A$ is observed

The denominator is the total probability:

P (A)

10. Example: Medical Test

Let:

I = {infected}

and:

T = {test positive}

The test satisfies:

P (T ∣ I) = 0.90

95% of non-infected people test negative:

P (T^{c} ∣ I^{c}) = 0.95

therefore:

P (T ∣ I^{c}) = 0.05

Also:

P (I) = 0.02

and:

P (I^{c}) = 0.98

We want:

P (I ∣ T)

Using Bayes' rule:

P (I ∣ T) = \frac{P (T ∣ I) P (I)}{P (T ∣ I) P (I) + P (T ∣ I^{c}) P (I^{c})}

Substituting:

= \frac{0.9 (0.02)}{0.9 (0.02) + 0.05 (0.98)}

= \frac{0.018}{0.067}

\approx 0.269

Therefore:

P (I ∣ T) \approx 26.9 %

Important

Even though 90% of infected people test positive, a positive test does not mean there is a 90% probability of being infected.

The base probability of infection, $P (I)$ , also matters.

11. Independence

Definition

An event $A$ is independent of event $B$ if:

P (A ∣ B) = P (A)

In other words, knowing that $B$ occurred does not change the probability of $A$ .

Equivalent Condition

From the multiplication rule:

P (A \cap B) = P (A ∣ B) P (B)

If $A$ and $B$ are independent:

P (A ∣ B) = P (A)

so:

P (A \cap B) = P (A) P (B)

This is often the easiest condition to use when checking independence.

Symmetry

If $A$ is independent of $B$ , then $B$ is also independent of $A$ .

A ⊥ B ⟺ B ⊥ A

12. Independence of More Than Two Events

For events:

A_{1}, A_{2}, \dots, A_{m}

independence must hold for every subset.

For three events, for example, we need:

P (A_{1} \cap A_{2}) = P (A_{1}) P (A_{2})

P (A_{1} \cap A_{3}) = P (A_{1}) P (A_{3})

P (A_{2} \cap A_{3}) = P (A_{2}) P (A_{3})

and also:

P (A_{1} \cap A_{2} \cap A_{3}) = P (A_{1}) P (A_{2}) P (A_{3})

Warning

Pairwise independence does not automatically imply full independence.

Three events may be independent in every pair while not being jointly independent.

This is explicitly highlighted as an exam-style pitfall in the lecture. :contentReference[oaicite:1]

13. Independent vs Disjoint

These concepts are very different.

Disjoint Events

Events are disjoint when:

A \cap B = \emptyset

Therefore:

P (A \cap B) = 0

They cannot happen at the same time.

Independent Events

Events are independent when:

P (A \cap B) = P (A) P (B)

If $A$ and $B$ both have positive probability and are disjoint:

P (A \cap B) = 0

while:

P (A) P (B) > 0

so they cannot be independent.

Danger

For events with positive probability:

disjoint $\Rightarrow$ dependent

If $A$ happens, then you know that $B$ cannot happen.

14. Which Rule Should I Use?

Situation	Rule
Probability of $A$ given $B$	Conditional probability
Probability of both $A$ and $B$	Multiplication rule
Several possible causes can produce $A$	Law of total probability
Observation is known and cause is unknown	Bayes' rule
Check whether events affect each other	Independence
Probability of $A$ or $B$	Sum rule
Probability of not $A$	Complement rule

15. Formula Overview

Conditional probability

$P (A ∣ B) = \frac{P (A \cap B)}{P (B)}$

Multiplication rule

$P (A \cap B) = P (A ∣ B) P (B)$

Law of total probability

$P (A) = \sum_{i} P (A ∣ C_{i}) P (C_{i})$

Bayes' rule

$P (C_{i} ∣ A) = \frac{P (A ∣ C_{i}) P (C_{i})}{\sum_{j} P (A ∣ C_{j}) P (C_{j})}$

Independence

$P (A ∣ B) = P (A)$

or equivalently:

P (A \cap B) = P (A) P (B)

16. Common Pitfalls

Warning

1. Reversing conditional probabilities

P (A ∣ B) \neq P (B ∣ A)

Warning

2. Ignoring the base probability

A highly accurate medical test can still give a relatively low

P (infected ∣ positive)

when the infection is rare.

Warning

3. Using $P (A \cap B) = P (A) P (B)$ automatically

This is only valid when the events are independent.

Warning

4. Confusing pairwise and full independence

For three or more events, the larger intersections must also be checked.

Warning

5. Confusing disjoint with independent

Disjoint events with positive probability are actually dependent.

17. What You Should Be Able to Do

According to the lecture wrap-up, you should be able to:

compute and interpret conditional probabilities
apply the different probability rules
compute conditional probabilities using Bayes' rule
check whether events are independent :contentReference[oaicite:2]

Core Idea

The most important starting point of the lecture is:

P (A ∣ B) = \frac{P (A \cap B)}{P (B)}

From this follows the multiplication rule:

P (A \cap B) = P (A ∣ B) P (B)

and together with the law of total probability, we obtain Bayes' rule:

P (B ∣ A) = \frac{P (A ∣ B) P (B)}{P (A)}

Conditional Probability and Bayes' Rule a

1. Permutations and Combinations

Permutations

Example

Combinations

Example

2. Probability of an Intersection

3. Conditional Probability

Definition

Interpretation

Example: Throwing a Die

4. The Difference Between P(A∣B) and P(B∣A)

5. Multiplication Rule

Example: Pebbles

6. Summary of Probability Rules

Sample Space

Sum Rule

Complement Rule

Multiplication Rule

De Morgan's Laws

7. Law of Total Probability

General Form

Example: Production Machines

8. Bayes' Rule

9. Bayes' Rule for Multiple Causes

10. Example: Medical Test

11. Independence

Definition

Equivalent Condition

Symmetry

12. Independence of More Than Two Events

13. Independent vs Disjoint

Disjoint Events

Independent Events

14. Which Rule Should I Use?

15. Formula Overview

16. Common Pitfalls

1. Reversing conditional probabilities

2. Ignoring the base probability

3. Using P(A∩B)=P(A)P(B) automatically

4. Confusing pairwise and full independence

5. Confusing disjoint with independent

17. What You Should Be Able to Do

Core Idea

4. The Difference Between $P (A ∣ B)$ and $P (B ∣ A)$

3. Using $P (A \cap B) = P (A) P (B)$ automatically