Return to Topics page

The problem, and danger, of this is that the tests were run individually at the

The bottom line here is that you cannot just do a "shotgun" investigation of lots of null hypotheses. If you find yourself in a position where it looks like you have to do this, then one suggestion, the

Another issue with sample size is that you can effectively reject almost any null hypothesis if you make your sample large enough. Let me explain. First, if the null hypothesis is true, as in absolutely true, then increasing the sample size will not let me effectively reject the null hypothesis all the time. However, look at the case where the null hypothesis is false, but not by much. Consider the case where the null hypothesis is that the mean value of some measure on a population is 34.37. Now, if the true mean value for that measure is 34.362, and assuming we are running the test with the alternative hypothesis that the true mean is not equal to 34.37, do we really want any sample to reject our null hypothesis? That is, do we really care if the true mean is 34.37 or 34.362? Pick any standard deviation you want for the sample. I can find a sample size that is large enough to make the standard error, the standard deviation of the sample mean, so small that 34.37 is more than 5 standard deviations away from 34.362. In that case it is almost a certainly that any sample will yield results that justify our rejecting the null hypothesis.

I understand that the previous discussion is quite complex and a bit convoluted. The point is that the real world is not statistics class. There are consequences to decisions. If you are going to make a decision based on statistic you need to understand how your choice of statistical tests and sample size will affect the statistical decision.

Not to be too snotty, but in my career I have been asked, at various time, to find some statistical value. My first response is to ask if there is a decision to be made based on the result of that analysis. If the answer is "No" then I just make up an answer. Thus, if the head of Admissions at WCC asks for the average age of admitted students, my question is "Will you do anything different depending upon the answer?" When the answer to that comes back as "No" then I pull a number out of the air, perhaps 23.416 (adding the decimal places to make it look good) and life goes on at no cost to me or to the school.

However, if the answer is, "Yes, if the mean age is under 23 then we will completely change our advertising strategy?" then I start asking, "Are you sure? If the average comes in at 22.9 will you still make the changes?" Until we can get to the true "cut point" there is no fair way to move forward with the process.

©Roger M. Palay Saline, MI 48176 November, 2021