A hypothesis test is a statistical method that answers a yes-no question

Example Is the average GPA of undergraduates at the Colegio less than 2.8?
Example Is the average income of men in Puerto Rico higher than the average income of women?
Example Is there a relationship between smoking and lung cancer?

A hypothesis test is usually phrased in the form of two statements rather than a question. These statements are called the null hypothesis (H₀) and the alternative or research hypothesis (H₁ or H_a)

Example
H₀: The average GPA of undergraduates at the Colegio is 2.8 (or maybe even higher).
H_a: The average GPA of undergraduates at the Colegio is less than 2.8.

Example
H₀: The average income of men and women in Puerto Rico is the same.
H_a: The average income of men in Puerto Rico is higher than the average income of women.

Example
H₀: There is no relationship between smoking and lung cancer.
H_a: There is a relationship between smoking and lung cancer.

Now instead of deciding whether we should answer the question with yes or no we are going to decide which statement we believe is true, but of course this is (almost) the same thing.

Often we will make our decision based on a parameter, and the value of the corresponding statistic. If so we can also express the hypotheses in terms of population parameters. If the hypothesis is written in terms of parameters this (almost always) means that the null hypothesis has an = sign.

Example Is the average GPA of undergraduates at the Colegio less than 2.8? Here we are looking at an "average". In Statistics we have several ways to compute an "average", such as the mean or the median. Which of these is better depends on many considerations. Let's say we use the mean. Now the standard symbol for a population mean is μ, and so we can write the hypotheses as follows:
H₀: μ = 2.8
H_a: μ < 2.8

Example Is the average income of men in Puerto Rico higher than the average income of women? Again we are interested in "averages", and let's say here we decide to use the median. The population median is sometimes denoted by l. But there are two medians: the median income of men and the median income of women. Let's denote them by l_M and l_W, respectively. Then the hypotheses are:
H₀: l_M = l_W
H_a: l_M > l_W

Example Is there a relationship between smoking and lung cancer? Now we are looking at the relationship between the two variables "smoking" and "lung cancer". If both are continuous variables (and some other conditions hold) we could use Pearson's correlation coefficient ρ as a measure of the strength of the relationship:
H₀: ρ = 0
H_a: ρ ≠ 0

Hypothesis Testing: Formalism and Notation

1) Parameter of interest
2) Method of analysis
3) Assumptions of Method
4) Type I error probability α
5) Null hypothesis H₀ (in terms of parameter and in plain language)
6) Alternative hypothesis H_a (in terms of parameter and in plain language)
7) Test statistic
8) Rejection region and decision on test
9) Conclusion (in plain language)

Warning In any homework or exam if any of these parts are missing you will loose points!

Example: Over the last five years the average score in the final exam of a course was 73 points. This semester a class with 27 students used a new textbook, and the mean score in the final was 78.1 points with a standard deviation of 7.1.
Question: did the class using the new text book do (statistically significantly) better?

For this specific example the complete hypothesis test looks as follows:

1) Parameter: mean μ
2) Method: 1-sample t test
3) Assumptions: data comes from normal distribution, or n large. Checked boxplot
4) α = 0.05
5) H₀: μ = 73 (mean score is still 73)
6) H_a: μ > 73 (mean score is higher than 73)
7)

8) reject H₀ if T>t_{26, 0.05} = 1.706, T = 3.81 > 1.706, so we reject the null hypothesis
9) The mean score in the final is statistically significantly higher than before.

Example for carrying out a hypothesis test using the p-value approach. We will do the same example as above for the new textbook:

1) Parameter: mean μ
2) Method: 1-sample t test
3) Assumptions: data comes from normal distribution, or n large. Checked boxplot
4) α = 0.05
5) H₀: μ = 73 (mean score is still 73)
6) H_a: μ > 73 (mean score is higher than 73)
7) p =0.000101 (Stat > Basic Statistics > 1-sample t > Summarized data: Sample size: 27, Mean: 78.1, Standard Deviation: 7.1, Test mean: 73, Options > Alternative: greater than )
8) p = 0.000101 < α = 0.05, so we reject the null hypothesis
9) The mean score in the final is statistically significantly higher than before.

What is the p value? It is the probability to observe our data or something even more extreme if the null hypothesis is true. So in the example above it is P( ≥ 78.1 | μ = 73)

The advantage of the p-value approach is that in addition to the decision on whether or not to reject the null hypothesis it also gives us some idea on how close a decision it was. Here with p=0.0001 we would have rejected the null hypothesis even if we had chosen α = 0.01, so it was not a close thing at all.

Let's do a little simulation to understand the p-value

Example We will do the following:

• generate 50 observations from a normal distribution with mean μ and standard deviation s=1.0
• find the p-value of the test H₀:μ=10.0 vs H_a: μ≠10.0
• repeat 1000 times
• draw the histogram of the 1000 p-values and find the percentage of p-values<0.05

This is done in the MACRO pvalue, run it with

CTRL-L, %k:\3101\pvalue 10.0

Now if mu=10.0, the null hypothesis is true. As we can see in that case any number between 0 and 1 is equally likely to be our p-value, and so 5% are < 0.05.
As mu gets father away from the hypothesized value of 10.0 (the null hypothesis is "more" wrong), the p-values start to "bunch up" around 0, so we correctly rejecting the null hypothsis more and more often.

Example Let's return for a moment to the 1970's draft. When we ran the simulation we were really carrying out the following hypothesis test:

1) Parameter: Pearson's correlation coefficient ρ
2) Method: Test for ρ
3) Assumptions: Data comes from normal distribution. Checked.
4) α = 0.05
5) H₀: ρ=0 (no relationship between Day of Year and Draft Number)
6) H_a: ρ≠0 (some relationship between Day of Year and Draft Number)
7) p<1/1000 (from simulation)
8) p<α = 0.05, so we reject the null hypothesis,
9) There is a statistically significant relationship between Day of Year and Draft Number.

H₀: μ_T=μ_C (the new drug does not work better)
H_a: μ_T<μ_C (the new drug does work better)

At first it seems a little strange to students that we would choose "new drug is not better than the old one" as H₀, but there are good reasons for this approach as we will see later. In practise it is very easy for us: H₀ always has the = sign!.

Warning: In the 6 parts of a hypothesis test, the first 3 (at least in theory) should be done before looking at the data. The following is not allowed: say we did a study of students at the Colegio. We asked them many questions. Afterwards we computed correlation coefficients for all the pairs of variables and found a high correlation between "Income" and "GPA". Then we carried out a hypothesis test H₀:ρ = 0 vs. H_a:ρ ¹ 0
The problem here is that this hypothesis test was suggested to us by the data, but hypothesis tests only work as advertised if the hypotheses are formulated without consideration of the data.

Related to this problem is the following issue: as we said we will always use H₀ with = (for example μ = 0). On the other hand there are three commonly used alternative hypotheses:
H_a: μ > 0
H_a: μ < 0
H_a: μ ¹ 0

Go back to our example of the new textbook. Here we have the following
Correct: we pick H_a: μ > 73 because we want to proof that the new textbook works better than the old one.
Wrong: we pick H_a: μ > 73 because the sample mean score was 78.1, so if anything the new scores are higher than the old ones.

Warning:
A null hypothesis looks something like this

H₀: μ=14.5

not

H₀=14.5 (What is the parameter?)
or
μ=14.5 (Is this H₀ or H_a?)

Warning: getting the correct null hypothesis is very important - if you do everything else right but picked the wrong statement as your null hypothesis you will always get the wrong answer!

So there are two possible mistakes (type I and type II), and the probabilities for making them (α and β). These two mistakes, though, are treated completely differently in statistics: when we do a hypothesis test we decide ahead of time what we are willing to accept as a type I error probability α, and then accept whatever the type II error probability β is. Well, not quite, but wait and see.

We have already talked about confidence intervals. At first a confidence interval and a hypothesis test seem to bee very different but they are actually closely related. So finding a 90% confidence interval is related to carrying out a hypothesis test with α=0.1 because 100·(1-α)%=90% leads to α=0.1.
As we saw before if you find a 95% CI instead of a 90% CI you make the interval wider. Similarly if you make α smaller, thereby reducing the probability of falsely rejecting the null hypothesis you (almost always) make β larger, that is you increase the probability of falsely accepting a wrong null hypothesis. The only way to make both α and β smaller is by increasing the sample size n.

How do you choose α? This in practice is a very difficult question. What you need to consider is the consequences of the type I and the type II errors.

Example In our example above with the new textbook, what does it mean to "commit the type I error"? If we do, what are the consequences? What does it mean to "commit the type II error" and what are its consequences?

Type I error: Reject H₀ although H₀ is true

H₀: μ=73
Scores are the same with new textbook as they were with old textbook
new textbook is not better

This is the truth, but we don't know this, based on our experiment and the hypothesis test we reject H₀, that is now we think the textbook is better

Consequences?
• We will change textbooks for everybody
• New students will not be able to buy used books, previous students will not be able to sell their books
• Professors have to rewrite their material, prontuarios etc.
• Professors will not consider other new textbooks that might really be better
• but scores will not go up, all of this is for nothing

Type II error: Fail to reject H₀ although H₀ is false

H₀: μ=73 (new textbook is not better)

This is false, but we don't know this, based on our experiment and the hypothesis test we fail to reject H₀, that is now we think the textbook is not better

Consequences?
• We will not change to the new textbook
• scores will not go up, but they would have if we had changed
• more students would have passed the course, got an A etc., but now they won't
• Professors will consider other new textbooks, but those might really be worse than the one we just rejected.

Note that in this example we will probably find out that we committed the type I error because we will observe that over the next few years the scores are not going up. On the other hand if we comit the type II error we are not likely to ever find out!

Example Let's illustrate the issue with a little simulation. For this we will generate some data and carry out a hypothesis test as follows:

Calc > Random Data > Normal, Generate 20 rows of data , Store in c1, Mean: 10.0, Standard Deviation: 3.0

Now we carry out the following hypothesis test:

1) Parameter: mean μ
2) Method: 1-sample t test
3) Assumptions: data comes from normal distribution (true because this is how data is generated)
4) α = 0.05
5) H₀: μ=10.0
6) H_a: μ≠10.0

But we generated the data, so we know that μ=10.0. Therefore we know that the null hypothesis is true, and so if we commit an error it will be the type I error.

Now if we keep doing the above many times, what will happen? According to the theory we should fail to reject H₀ (which is the correct conclusion) 95% of the time, and we should reject H₀ (commit the type I error) 5% of the time.

This is done in the MINITAB macro test. Run it with this command:

CTRL-l, %k:\3101\test 20 10.0 3 0.05

Now let's change things a bit. Instead of generating data from a normal with mean 10.0, generate it from a normal with mean 12.0, but still carry out the test above for μ=10.0. So now we know that H₀ is false, and we should reject it. If we don't we will commit the type II error. How often do we do this? Again run the macro:

CTRL-l, %k:\3101\test 20 12.0 3 0.05

As we see the method correctly rejects H₀ about 81% of the time and so commits the type II error 19% of the time. So we find for this exact setup β=0.19

Here are some interesting cases:

Effect of the true mean

Effect of the standard deviation

Effect of α

Effect of Sample Size n

Above we found the probability of the type I error β. In real live one usually calculates the power of a test, which is simply 1-β = P(correctly reject H₀|H₀ is wrong). MINITAB can calculate these numbers exactly using the Stat > Power and Sample Size command.

Example If we run CTRL-l, %k:\3101\test 400 10.5 3 0.05 we find the type I error probability is 0.09. Now Stat > Power and Sample Size . 1-sample t, Samplesize: 400, Difference 0.5 (10.5-10), standard deviation: 3, yields 0.913926, and 1-0.913926 = 0.09

Example: There is a famous (infamous?) case of three psychiatrists who studied a sample of schizophrenic persons and a sample of nonschizophrenic persons. They measured 77 variables for each subject - religion, family background, childhood experiences etc. Their goal was to discover what distinguishes persons who later become schizophrenic. Using their data they ran 77 hypothesis tests of the significance of the differences between the two groups of subjects, and found 2 significant at the 2% level.They immediately published their findings.

What's wrong here? Remember, if you run a hypothesis test at the 2% level you expect to make reject the null hypothesis of no relationship 2% of the time, but 2% of 77 is about 1 or 2, so just by random fluctuations they could (should?) have rejected that many null hypotheses! This is not to say that the variables they found to be different between the two groups were not really different, only that their method did not proof that.

Example let's do a little simulation to illustrate the problem: First this we generate 50 observations from a normal distribution with mean 10.0 and standard deviation 1.5:
Calc > Random Data > Normal, generate 50 observations, store in x, mean 10.0, standard deviation 1.5
Now we test H₀:μ=10.0 vs H_a:μ≠10.0:
Stat > Basic Statistics > 1-sample t, samples in column: x, Test mean: 10.0

Now we generated the data, so we know for sure that H₀ is true, and we should fail to reject it. But if we keep doing this sooner or later we will in fact reject H₀, that is we will commit the type I error.

I wrote a little MINITAB Macro to repeat the above simulation 10 times. Run it with
CTRL-L, %K\3101\multinf

For more on hypothesis testing see chapter 8 of the textbook.