Bayes' theorem

免费电子图书馆

金融学习网：学习金融知识，观察市场动态

杨甫刚：在校大学生开网店月赚万元

Prospect theory

大 | 中 | 小

[

2008/04/28 22:40 | by mlzy ]

In probability theory, Bayes' theorem (often called Bayes' Law) relates the conditional and marginal probabilities of two random events. It is often used to compute posterior probabilities given observations. For example, a patient may be observed to have certain symptoms. Bayes' theorem can be used to compute the probability that a proposed diagnosis is correct, given that observation. (See example 2)

As a formal theorem, Bayes' theorem is valid in all interpretations of probability. However, it plays a central role in the debate around the foundations of statistics: frequentist and Bayesian interpretations disagree about the ways in which probabilities should be assigned in applications. Frequentists assign probabilities to random events according to their frequencies of occurrence or to subsets of populations as proportions of the whole, while Bayesians describe probabilities in terms of beliefs and degrees of uncertainty. The articles on Bayesian probability and frequentist probability discuss these debates at greater length.

Statement of Bayes' theorem

Bayes' theorem relates the conditional and marginal probabilities of events A and B, where B has a non-vanishing probability:

$P(A|B) = \frac{P(B | A)\, P(A)}{P(B)}.$

Each term in Bayes' theorem has a conventional name:

P(A) is the prior probability or marginal probability of A. It is "prior" in the sense that it does not take into account any information about B.
P(A|B) is the conditional probability of A, given B. It is also called the posterior probability because it is derived from or depends upon the specified value of B.
P(B|A) is the conditional probability of B given A.
P(B) is the prior or marginal probability of B, and acts as a normalizing constant.

Intuitively, Bayes' theorem in this form describes the way in which one's beliefs about observing 'A' are updated by having observed 'B'.

[edit] Bayes' theorem in terms of likelihood

Bayes' theorem can also be interpreted in terms of likelihood:

$P(A|B) \propto L(A | b)\, P(A).$

Here L(A|b) is the likelihood of A given fixed b. The rule is then an immediate consequence of the relationship $P(B | A) \propto L(A | B)$ .

With this terminology, the theorem may be paraphrased as

$\mbox{posterior} = \frac{\mbox{likelihood} \times \mbox{prior}} {\alpha}$

(where $α$ is a normalising constant).

In words: the posterior probability is proportional to the product of the prior probability and the likelihood.

[edit] Derivation from conditional probabilities

To derive the theorem, we start from the definition of conditional probability. The probability of event A given event B is

$P(A|B)=\frac{P(A \cap B)}{P(B)}.$

Equivalently, the probability of event B given event A is

$P(B|A) = \frac{P(A \cap B)}{P(A)}. \!$

Rearranging and combining these two equations, we find

$P(A|B)\, P(B) = P(A \cap B) = P(B|A)\, P(A). \!$

This lemma is sometimes called the product rule for probabilities. Dividing both sides by P(B), providing that it is non-zero, we obtain Bayes' theorem:

$P(A|B) = \frac{P(A \cap B)}{P(B)} = \frac{P(B|A)\,P(A)}{P(B)}. \!$

[edit] Alternative forms of Bayes' theorem

Bayes' theorem is often embellished by noting that

$P(B) = P(A\cap B) + P(A^C\cap B) = P(B|A) P(A) + P(B|A^C) P(A^C)\,$

where A^C is the complementary event of A (often called "not A"). So the theorem can be restated as

$P(A|B) = \frac{P(B | A)\, P(A)}{P(B|A) P(A) + P(B|A^C) P(A^C)}. \!$

More generally, where {A_i} forms a partition of the event space,

$P(A_i|B) = \frac{P(B | A_i)\, P(A_i)}{\sum_j P(B|A_j)\,P(A_j)} , \!$

for any A_i in the partition.

[edit] Bayes' theorem in terms of odds and likelihood ratio

Bayes' theorem can also be written neatly in terms of a likelihood ratio Λ and odds O as

$O(A|B)=O(A) \cdot \Lambda (A|B)$

where $O(A|B)=\frac{P(A|B)}{P(A^C|B)} \!$ are the odds of A given B,

and $O(A)=\frac{P(A)}{P(A^C)} \!$ are the odds of A by itself,

while $\Lambda (A|B) = \frac{L(A|B)}{L(A^C|B)} = \frac{P(B|A)}{P(B|A^C)} \!$ is the likelihood ratio.

[edit] Bayes' theorem for probability densities

There is also a version of Bayes' theorem for continuous distributions. It is somewhat harder to derive, since probability densities, strictly speaking, are not probabilities, so Bayes' theorem has to be established by a limit process; see Papoulis (citation below), Section 7.3 for an elementary derivation. Bayes's theorem for probability densities is formally similar to the theorem for probabilities:

$f_X(x|Y=y) = \frac{f_{X,Y}(x,y)}{f_Y(y)} = \frac{f_Y(y|X=x)\,f_X(x)}{f_Y(y)} = \frac{f_Y(y|X=x)\,f_X(x)}{\int_{-\infty}^{\infty} f_Y(y|X=\xi )\,f_X(\xi )\,d\xi }.\!$

There is an analogous statement of the law of total probability, which is used in the denomenator:

$f_Y(y) = \int_{-\infty}^{\infty} f_Y(y|X=x )\,f_X(x)\,dx .\!$

As in the discrete case, the terms have standard names.

$f_{X,Y}(x,y)\,$ is the joint distribution of X and Y,

$f_X(x|Y=y)\,$ is the posterior distribution of X given Y=y,

$f_Y(y|X=x) = L(x|y)\,$ is (as a function of x) the likelihood function of X given Y=y,

and

$f_X(x)\,$

and

$f_Y(y)\!$

are the marginal distributions of X and Y respectively, with $f_X(x)\,$ being the prior distribution of X.

[edit] Abstract Bayes' theorem

Given two absolutely continuous probability measures $P ˜ Q$ on the probability space $(\Omega, \mathcal{F})$ and a sigma-algebra $\mathcal{G} \subset \mathcal{F}$ , the abstract Bayes theorem for a $\mathcal{F}$ -measurable random variable $X$ becomes

$E_P[X|\mathcal{G}] = \frac{E_Q[\frac{dP}{dQ} X |\mathcal{G}]}{E_Q[\frac{dP}{dQ}|\mathcal{G}]}$ .

This formulation is used in Kalman filtering to find Zakai equations. It is also used in financial mathematics for change of numeraire techniques.

[edit] Extensions of Bayes' theorem

Theorems analogous to Bayes' theorem hold in problems with more than two variables. For example:

$P(A|B \cap C) = \frac{P(A) \, P(B|A) \, P(C|A \cap B)}{P(B) \, P(C|B)}\,.$

This can be derived in a few steps from Bayes' theorem and the definition of conditional probability:

$P(A|B \cap C) = \frac{P(A \cap B \cap C)}{P(B \cap C)} = \frac{P(C|A \cap B) \, P(A \cap B)}{P(B) \, P(C|B)} = \frac{P(A) \, P(B|A) \, P(C|A \cap B)}{P(B) \, P(C|B)}\,.$

Similarly, we have

$P(A|B \cap C) = \frac{P(B|A \cap C) \, P(A|C)}{P(B|C)}\,,$

which can be regarded as a conditional Bayes' Theorem, and can be derived by as follows:

$P(A|B \cap C) = \frac{P(A \cap B \cap C)}{P(B \cap C)} = \frac{P(B|A \cap C) \, P(A|C) \, P(C)}{P(C) \, P(B|C)} = \frac{P(B|A \cap C) \, P(A|C)}{P(B|C)}\,.$

A general strategy is to work with a decomposition of the joint probability, and to marginalize (integrate) over the variables that are not of interest. Depending on the form of the decomposition, it may be possible to prove that some integrals must be 1, and thus they fall out of the decomposition; exploiting this property can reduce the computations very substantially. A Bayesian network, for example, specifies a factorization of a joint distribution of several variables in which the conditional probability of any one variable given the remaining ones takes a particularly simple form (see Markov blanket).