Power and sample size calculations

One of the other issues Jos Twisk commented upon in his inaugural address (see previous posting) was the doubtful obligation of researchers to provide research proposals with a sample size/power calculation. To demonstrate the alledged futility of this requirement, he announced that he had written a small computer program that has the following input dialogue:

====================================================
Q: Give the required alpha
A: 0.05

Q: Give the required beta
A: .80

Q: Give the required standard deviation
A: ....

Q: Give the required effect size
A: ...

The program would now always produce the following output:

Sample size: As many units as possible

====================================================

That power calculations can be doubtful and unreliable has been stated by many others (See, for instance, Kuik and Tobi, 1998). The problems with it are twofold:

1. Almost always, there are practical constraints to the number of patients/respondents that can be possibly be included in a study. This makes it tempting for the researcher (and the methodologist who helps him or her) to choose standard deviation and effect size in such a way that an acceptable number comes out. Usually there is some freedom to do so.
2. Although there is not much room to tamper with the effect size, the variability estimate can be choosen more freely: in most cases it is hard to tell whether a realistic value has been chosen.

On the other hand, it is not realistic in practice (as Jos did) to militate against power calculations altogether. From the point of view of the subsidizer, the wish to get an impression of the expected credibility of the results of a study can hardly be called exaggerated. To underpower a study is not only wasting money, but may also be unethical towards the included patients, while overpowering may place an undue burden on the patients.

So, what can we do? Since the bottleneck in the power calculations seems to be the variance estimate, a possible solution is to estimate that statistic from a small number of observations, collected, for instance, during a pilot study. See section 25.4 in Efron and Tibshirani (1993), on how to approach this. They give a more general treatment of the use of bootstrapping in power calculations.

Herman Adèr

PS. An overview of available free statistical software including software for power and sample size calculations can be found at: http://statpages.org/javasta2.html

References

Kuik, D. J. and H. Tobi (1998). On the Uselessness of Power Computations. In: Proceedings in Computational Statistics. 1998. Short Communications and Posters. Eds. R. Payne and P. Laine. pp 67-68.

Efron, B. and R. J. Tibshirani (1993), An Introduction to the Bootstrap. New York London: Chapman & Hall.