Small sample SEM

An issue that came up during a session on applications of structural equation modelling (SEM) after a lecture by David Kaplan at the Colloquium `Advising on research methods' in Amsterdam (29-30 March) was the fact that SEM is less often applied in medical/epidemiological research than it could be. Several bottlenecks can be (and were) identified:

1. The concept of `latent variable' (and, relatedly, the concept of (substantive) theory and corresponding model) seems to be less easy to conceptualize in Medicine than in the social/behavioral sciences.
2. Jules Ellis mentioned that SEM software is not easily optainable and costly and that specifying a SEM model in standard software is often awkward and difficult to achieve by the medical researcher/analyst.
3. Most SEM models require sample sizes that are too large for most small- to moderately- sized medical/epidemiological research projects.

Comments:

Ad 1. (The concept of `latent variable' seems to be less easy to conceptualize in Medicine than in the social/behavioral sciences).

Of course, this is an interesting but more or less philosophical point, if phrased in this way. In real-life medical/epidemiological research there often seem to be no compelling reasons to take a theoretical stand to start from. I won't go into the reasons for this.

However, there are several situations in which a theoretical conceptualization could help to specify a SEM model that is more appropriate than the regression-like models that are frequently used. For instance,

• In what is called `fundamental' or `basic' medical research, where complicated dynamic mechanisms are studied (think of genome studies or fysiological studies of illness progression), dynamic SEM models could be used to specify the dynamic process.
• In questionnaire design, confirmatory factor analysis can be used to analyze the factor structure, just as it is done in social science research (see de Vet et al. (2005)).

Ad 2 (SEM software is costly and specifying a SEM model in standard software is too difficult for the medical researcher/analyst).

In an e-mail afterwards, David Kaplan mentioned that a package to perform SEM modelling exists in R+ (a statistical package related to Splus, freely obtainable via the Internet, see, f.i.: http://cran.nedmirror.nl/). It turned out to be written by John Fox. Sources and binaries for SEM can be obtained at: http://finzi.psych.upenn.edu/R/library/sem/html/00Index.html

As to the other point, that the syntax of model specifation would be too complicated for the researcher-in-the-field: after all, many of them have also been able to learn how to use multilevel analysis. So, where there is a will, there is a way, even in learning how to use SEM, and in particular if there would be an clear need.

Ad 3 (Most SEM models require sample sizes that are too large).

To me, this seems the most important bottleneck for standard application in Medicine/Epidemiology. If SEM requires to have sample sizes of at least 500, application in most medical studies are out of the question.

In his comments, Jelte Wicherts mentioned that small sample size techniques are being studied. Afterwards, in some e-mail exchanges, I suggested that resampling techniques like bootstrapping could be applied in small sample situations. As it turns out, Fox's package, mentioned above, also contains a bootstrapping possibility (but see Kaplan's book chapter 5, where arguments are given why sample sizes would increase when non-normality has to be assumed).

Comment by David Kaplan:

I'm not sure what you mean by:

(but see Kaplan's book chapter 5, where arguments are given why sample sizes would increase when non-normality has to be assumed).

I think you mean that when estimating models with non-normal observed variables, larger sample sizes are typically needed for estimators to behave properly. That is partly true, and was true in the good old days. But now, there are estimators that don't require huge sample sizes. Also, I believe there are bootstrapping approaches to get standard errors when sample sizes are a bit smaller.

References

de Vet, H. C., Ader, H. J., Terwee, C. B., & Pouwer, F. (2005). Are factor analytical techniques used appropriately in the validation of health status questionnaires? A systematic review on the quality of factor analys of the SF-36. Quality of life research, 14(5), 1203–1218.

Kaplan, D. (2000). Structural Equation Modeling. Foundations and Extensions. Thousand Oaks London New Delhi: Sage Publications.

Colloquium `Advising on research methods'

This colloquium was organised by Don Mellenbergh and myself and held on March 29-30 in Amsterdam, the Netherlands.

Speakers were :

• Janice Derr (Advising in a multi-disciplinary setting)
• Steven Piantadosi (Research designs in Medicine)
• Don Mellenbergh (Advising on test construction)
• Gerald van Belle (Statistics and everyday life)
• Jules Ellis (Advising to policy makers in Health Care)
• Bo Lu (Bias correction using propensity scores)
• Paul de Boeck (Consulting in behaviour research)
• Willem Heiser (Survival skills in publishing)
• Robert Pool (Combining qualitative and quantitative methods)
• David Kaplan (Research problem and structural equation model)
• Denny Borsboom (Advising on test validity)
• Herman Adèr (Time and strategy)

The colloquium was preceded by a masterclass on Wednesday the 28st.

For more details, see: http://www.knaw.nl/colloquia/advising/
In the next few posts, I will describe some of the topics that were discussed.

Herman

Which methodological mistake is less harmful?

I hope you can agree to the following statement:

`Methodological considerations on an empirical study are relevant only in relation to the results'.

This idea is in line with the concepts of content quality of a study, defined as:

Content quality. The Content quality of a study indicates its bearing on the solution of the initial research problem.

Now assume that a study is methodologically flawed in some way. In other words, the content quality is low. The question now arises what is less harmful:

1. Something that could have been uncovered by using the right methodology, but that was not found due to these flaws.
2. Due to methodological flaws, erroneous conclusions were drawn (and published).

Most researchers would indicate the first option as the least harmful, reasoning that it happens all the time: different researchers draw different conclusions looking at the same data, and historically, new perspectives are opened time and again. However, one may ask what is meant by `the right methodology' in option 1: if this indicates that something could have been found, but wasn't, since the less subtle analysis technique was applied, the error could have been easily avoided. In particular, if the data set was published, as is required in some fields of research, your name as a serious researcher would be endangered if somebody else, repeating the analyses the right way, would come up with something completely different! (maybe that's why some researchers are hesitant to make their data publicly available).

In the second case, if these wrong conclusions are simply negative ones (we did not find nor report an effect although there is one present in the data), this is of course a pity, but there is no serious harm done. Even in case somebody showed that a positive conclusion could have been drawn, nobody will look askance at you, since your conclusions were at the safe side. And the one who has done the replication should explain very well why his results are deviant. If he or she has applied some outlandish technique that is not familiar to most, (s)he may even be considered a niggler who is out to harm your good name as a researcher.

(note that in the above, we talk about published results, although this may not have been obvious at the beginning).

Power and sample size calculations

One of the other issues Jos Twisk commented upon in his inaugural address (see previous posting) was the doubtful obligation of researchers to provide research proposals with a sample size/power calculation. To demonstrate the alledged futility of this requirement, he announced that he had written a small computer program that has the following input dialogue:

====================================================
Q: Give the required alpha
A: 0.05

Q: Give the required beta
A: .80

Q: Give the required standard deviation
A: ....

Q: Give the required effect size
A: ...

The program would now always produce the following output:

Sample size: As many units as possible

====================================================

That power calculations can be doubtful and unreliable has been stated by many others (See, for instance, Kuik and Tobi, 1998). The problems with it are twofold:

1. Almost always, there are practical constraints to the number of patients/respondents that can be possibly be included in a study. This makes it tempting for the researcher (and the methodologist who helps him or her) to choose standard deviation and effect size in such a way that an acceptable number comes out. Usually there is some freedom to do so.
2. Although there is not much room to tamper with the effect size, the variability estimate can be choosen more freely: in most cases it is hard to tell whether a realistic value has been chosen.

On the other hand, it is not realistic in practice (as Jos did) to militate against power calculations altogether. From the point of view of the subsidizer, the wish to get an impression of the expected credibility of the results of a study can hardly be called exaggerated. To underpower a study is not only wasting money, but may also be unethical towards the included patients, while overpowering may place an undue burden on the patients.

So, what can we do? Since the bottleneck in the power calculations seems to be the variance estimate, a possible solution is to estimate that statistic from a small number of observations, collected, for instance, during a pilot study. See section 25.4 in Efron and Tibshirani (1993), on how to approach this. They give a more general treatment of the use of bootstrapping in power calculations.

Herman Adèr

PS. An overview of available free statistical software including software for power and sample size calculations can be found at: http://statpages.org/javasta2.html

References

Kuik, D. J. and H. Tobi (1998). On the Uselessness of Power Computations. In: Proceedings in Computational Statistics. 1998. Short Communications and Posters. Eds. R. Payne and P. Laine. pp 67-68.

Efron, B. and R. J. Tibshirani (1993), An Introduction to the Bootstrap. New York London: Chapman & Hall.

Comments on the inaugural address of Prof dr Jos Twisk I: multiple imputation

On Friday, February 2nd, Jos Twisk held his inaugural address at the occasion of his appointment as a full professor applied biostatistics of longitudinal research at VU University,
Amsterdam, entitled (my translation):

Applied biostatistics: be swayed by the issues of the day or sustain ancient traditions?

It was a very amusing lecture, in which he brought up several methodological issues.

I will bring up these issues here once more as subsequent entries to this weblog and give my personal comments.

One of today's issues is `multiple imputation', introduced by Rubin in the nineties of the previous century (see for instance: Rubin, 1987/2004). Jos showed how results of multiple
imputed data sets have to be combined to get a composite result. His conclusion was:
don't impute if it can be avoided and certainly don't do it several times.

In general, I agree with the first part of that advice, but this is not realistic in real life. Techniques like repeated measurement analysis of (co-)variance require complete
data sets and one missing observation of a patient will cause the whole case to be left out from the analysis.

One can, of course, use multilevel analysis instead, but this is difficult if a complex factorial design with several covariates has to be analyzed. It takes away the possibility to use analysis of variance in a quick-and-dirty way to get rid of uninfluential covariates/factors (confounders).

Instead, I prefer to multiply (three times, as Rubin suggests, not five as Jos did in his deterrent) imputate anyway but use only one of the imputed data sets for the whole data analysis process.
Only when the analysis process is finished, the essential results are replicated using the other two data sets.

Four extra remarks about this suggestion:

1. Apart from a comparison with the two other imputed data sets, a complete case analysis should be done and the results compared.
2. Rubin's original proposal was aimed at parameter estimation. By pooling the results in a clever way combined parameters are optained in which the variance due to imputation is taken into account. However, we are usually not interested in estimation. In the above procedure, we only check whether the imputation procedure influences the effects of interest.
3. The above procedure does not serve as a check on the assumption of ignorability of nonresponse (`missing at randon'), since nothing much can be derived from the data set concerning the distribution of nonresponse.
4. Note that there is a publication problem. One can not publish results that are based on heavily imputed data sets derived from a raw data set with a large percentage of missings, unless the results are more or less the same as those obtained from the complete case analysis. In that case, reporting the results from the raw data set is preferable.

Herman

Daniel B. Rubin (1987). Multiple Imputation for Nonresponse in Surveys. New York: Wiley. (Reprinted in 2004).