Expectations

The expectation (or expected value) of a random variable g(X) is defined by

We use the notation Eg(X)

Example : we roll fair die until the first time we get a six. What is the expected number of rolls?
We saw that f(x) = 1/6*(5/6)^x-1 if x{1,2,..}. Here we just have g(x)=x, so

How do we compute this sum? Here is a "standard" trick:

and so we find

X is said to have a uniform [A,B] distribution if f(x)=1/(B-A) for A<x<B, 0 otherwise.
Find EX^k (this is called the k^th moment of X).

some special expectations are the mean of X defined by m=EX and the variance defined by s²=V(X)=E(X-m)². Related to the variance is the standard deviation s, the square root of the variance.

Here are some formulas for expectations:

the last one is a useful formula for finding the variance and/or the standard deviation.

Example : find the mean and the standard deviation of a uniform [A,B] r.v.

One way to "link" probabilities and expectations is via the indicator function I_A defined as

because with this we have for a continuous r.v. X with density f:

Covariance and Correlation

The covariance of two r.v. X and Y is defined by cov(X,Y)=E[(X-m_X)(Y-m_Y)]
The correlation of X and Y is defined by cor(X,Y)=cov(X,Y)/(s_Xs_Y)

Note cov(X,X) = V(X)

As with the variance we have a simpler formula for actual calculations: cov(X,Y) = E(XY) - (EX)(EY)

Example : take the Example of the sum and absolute value of the difference of two rolls of a die. What is the covariance of X and Y? So we have
m_X = EX = 2*1/36 + 3*2/36 + ... + 12*1/36 = 7.0
m_Y = EY = 0*6/36 + 1*12/36 + ... + 5*2/36 = 70/36
EXY = 0*2*1/36 + 1*2*0/36 + .2*2*0/36.. + 5*12*0/36 = 490/36
and so cov(X,Y) = EXY-EXEY = 490/36 - 7.0*70/36 = 0

It is easy to make a mistake doing a calculation like this, especially if the probabilities are not simple fractions. You can sometimes also do this using the computer!

In the matrix "sumdiff" we already have the pmf in R.
apply(sumdiff,1,sum) sums the matrix over the columns
sum(2:12*apply(sumdiff,1,sum)) finds the mean of X (7.0)
sum(0:5*apply(sumdiff,2,sum)) finds the mean of Y (1.9444444)

exy=0
for(i in 1:11) {
   for(j in 1:6) {
      exy=exy+c(2:12)[i]*c(0:5)[j]*sumdiff[i,j]
   }
}

finds EXY (13.6111111)

Note that we previously saw that X and Y are not independent, so we here have an Example that a covariance of 0 does not imply independence! It does work the other way around, though:

Theorem: If X and Y are independent, then cov(X,Y) = 0 ( = cor(X,Y))

proof (in the case of X and Y continuous):

and so cov(X,Y) = EXY-EXEY = EXEY - EXEY = 0

We saw above that E(X+Y) = EX + EY. How about V(X+Y)?

and if XY we have V(X+Y) = VX + VY

Conditional Expectation and Variance

Say X|Y=y is a conditional r.v. with pmf (pdf) f. Then the conditional expectation of X|Y=y is defined by

Let E[X|Y] denote the function of the random variable Y whose value at Y=y is given by E[X|Y=y]. Note then Z=E[X|Y] is itself a random variable.

Example: An urn contains 2 white and 3 black balls. We pick two balls from the urn. Let X be denote the number of white balls chosen. An additional ball is drawn from the remaining three. Let Y equal 1 if the ball is white and 0 otherwise.
For example f(0,0) = P(X=0,Y=0) = 3/5*2/4*1/3 = 1/10.
The complete pmf is given by:

Y\X 0 1 2

0 1/10 2/5 1/10

1 1/5 1/5 0

The marginals are given by

x 0 1 2

P(X=x) 3/10 3/5 1/10

and

y 0 1

P(Y=y) 3/5 2/5

The conditional distribution of X|Y=0 is

x 0 1 2

P(X=x|Y=0) 1/6 2/3 1/6

and so E[X|Y=0] = 0*1/6+1*2/3+2*1/6 = 1.0

The conditional distribution of X|Y=1 is

x 0 1 2

P(X=x|Y=1) 1/2 1/2 0

and so E[X|Y=1] = 0*1/2+1*1/2+2*0 = 1/2

Finally the conditional r.v. Z = E[X|Y] has pmf

z 1 1/2

P(Z=z) 3/5 2/5

with this we can find E[Z] = E[E[X|Y]] = 1*3/5+1/2*2/5 = 4/5

How about using simulation to do these calculations? - program urn1

There is a very useful formula for the expectation of conditional r.v.s: E[E[X|Y]] = E[X]
E[X] = 0*3/10 + 1*3/5 + 2*1/10 = 4/5

There is a simple explanation for this seemingly complicated formula!

Here is a corresponding formula for the variance:
V(X) = E[V(X|Y)] + V[E(X|Y)]

Example: let's say we have a continuous bivariate random vector with the joint pdf f(x,y) = c(x+2y) if 0<x<2 and 0<y<1, 0 otherwise.
Find C

Find the marginal distribution of X

Find the marginal distribution of Y

Find the conditional pdf of Y|X=x

Note: this is a proper pdf for any fixed value of x
Find E[Y|X=x]

Let Z=E[Y|X]. Find E[Z]