Power of a Hypothesis Test

A very important concept in hypothesis testing, and one where simulation plays a big role, is the idea of the power of a test. Basically it is the ability of a test to reject a false null hypothesis. Technically the power is defined as 1-b where b is the Type II error probability.

As an Example let's consider again the simple t-test. Here we have a sample X1, .. , Xn from a N(m,s2). Say we want to test H0: m = m0 vs. Ha: m > m0 at the a level. Then we reject the null hypothesis if P(T>ta,n-1)
Now let's assume that reality the population mean is m1. Then the power of the test is found as follows:
powerfig1.png - 3212 Bytes

Notice that p(m0) = a, which is the reason we use 1-b instead of b for the definition of the power.
The routine powerttest carries out the necessary calculations.

We could have found the power of this test also using simulation. For this we generate n observations from N(m1,s2), find T and see whether or not T>ta,n-1. Then we repeat this m times. The routine powersim1 does the work.

Simulation, though let's us go much farther. Say we want to use a test based on the median instead of the mean. Now we don't know the distribution of the sample median Med (this is still easy and we could look it up in a text book, but let's assume it is not known). Clearly we will reject the null hypothesis if Med > cv, but what is cv? We can use simulation as follows: generate m data sets of size n, compute the M medians and find the 100(1-a)% sample quantile. Use this as cv.
Once we have the rejection region Med>cv, we can proceed as in powersim1 to find the power of the test. The routine powermed has the necessary calculations.

Clearly this allows us to use just about any quantity as a teststatistics, without the need to be able to find the sampling distribution analytically.