Bootstrapping factorial ANOVA in SPSS

The following question appeared in Research Gate:

Elisabeth Fontaine
University of Adelaide · School of Psychology

Bootstrapping factorial ANOVA in SPSS v21?

Our institution provides SPSS v21 with the Bootstrap module loaded. I want to run factorial ANOVA (2x2) with a continuous DV that is skewed in each condition. I understand bootstrapping is a robust tool to apply to skewed data. Andy Field gives an excellent Youtube explanation of how to run the bootstrap option in SPSS and explains it is good for skewed data but the actual 'beer goggles' data he uses is not skewed. Perhaps I'm over-thinking it, but my question is: given the Bootstrap option is available for multi-way ANOVAs in SPSS does it therefore matter that the raw data is skewed when I run the ANOVA?

Suggested answer (Herman Adèr): 

The problem with bootstrapping a factorial design Anova is not the characteristics of the dependent variable (Skewness in this case) but that Anova is sensitive to the unbalancedness of the factorial design itself (unequal cell counts). Different bootstrap samples can produce very unbalanced design tables which will produce biased F-tests.
Instead, consider using a (bootstrapped, least squares) regression analysis of the model:
y = const + factor1 + factor2 + factor1 * factor2 + epsilon
in which factor1 and factor2 are dummy variables. Regression analysis is not sensitive to the variance-covariance matrix within the cells and therefore less sensitive to cell unbalancedness.
Maybe do both analyses and compare the results.

Another comment/answer (Noel Artiles-Leon, University of Puerto Rico at Mayagüez)
Before jumping into the bootstrapping van wagon, I would try to apply a transformation to the dependent variable to make the residuals normally distributed. You may try first with Sqrt[Y]  and Log[Y] ... if these transformations fail, use a Box-Cox transformation.

Comment to this:

If you distrust bootstrapping an Anova, use regression analysis (with or) without bootstrapping.

Noel's transformation suggestion has the disadvantage of making the interpretation of the results less straightforward, unless the transformation has a proper substantive interpretation like it has in the case of reaction times. If you have count data you should use Poisson regression anyway. 

Personally, I prefer the regression solution since it avoids getting biased results due to the strict assumptions of Anova.

Herman Adèr

Advising on research methods: Selected topics 2013

In 2014, A third collection of selected topics:

Advising on research methods: Selected topics 2013
Herman J. Adèr and Gideon J. Mellenbergh (Eds.)

was published by Johannes van Kessel Publishing.

Like the previous booklets, this one also has its own website:
www.jvank.nl/ARMSelected2013

All books on methodological advising can be found at:
www.jvank.nl/publishing/scientific

The 2011 and 2012 selected topics booklets are also available as e-books.The 2013 edition will be converted soon.

Herman

CONTENTS

Random or non-random assignment:

 What difference does it make? 

by Daan R. van Renswoude

Parametric IRT models and item analysis in R

by Joost Kruis

Comparing item imputation methods 

in questionnaire research

by Paul Lodder

Bootstrap basics

by Abe Huijbers

Data mining: Characteristics and application

 to the Math Garden data

 by Lisa Wijsen

Interpreting economic games

by Simon Columbus

Advising on research methods: Selected topics 2011

A similar publication as the one mentioned in the previous post, did appear the year before, following the same format. The title is:

Advising on research methods: Selected topics 2011

Edited by Herman J. Adèr and Gideon J. Mellenbergh

also published by Johannes van Kessel.
The paperback edition is sold out. But a eBook/iBook version is still available. This booklet is also indexed for Google Books. For more information, see it's website.

Table of contents:

From research question to statistical model

by Anja Sommavilla and Corinne Brenner

Pitfalls and payoffs in Internet sampling

by Corinne Brenner and Charlotte M. W. Gaasterland

Introduction to Computerized Adaptive Testing

by César-Reyer Vroom and Daniel A. Bannan

A critique of stepwise model selection methods

by Daniel A. Bannan and César-Reyer Vroom

A short introduction into survival analysis

by Charlotte M. W. Gaasterland and Mattis van den Bergh

On the relevance of mixed methods

by Mattis van den Bergh and Anja Sommavilla

Advising on research methods: Selected topics 2012

During last year's course `Advising on research methods' given at the Department of psychological methods by Don Mellenbergh and myself, participants (master's students) wrote papers on methodological topics which were afterwards collected in a booklet entitled:

Advising on research methods: Selected topics 2012

Edited by Herman J. Adèr and Gideon J. Mellenbergh

The booklet is published by Johannes van Kessel (ISBN 97-890-79418-21-3).
Since the papers were carefully reviewed, the contributions are of high quality.
The table of contents is:

Measurement Invariance

by Jonas Dalege and Loes Kreemers

A Comparison of Classical Test Theory

and Item Response Theory
by Marie K. Deserno

Unit nonresponse in Surveys

by Milou K. M. Lünnemann

Effect size: A meta-analytic perspective

by Adam Sasiadek and David Scholz

Outliers and Extreme Observations:

What are they and how to handle them?
by Suzan Q. Blommestijn and Esther A. Lietaert Peerbolte

Questionable research practices and scientific fraud

by Jochem Bout

The booklet is also available as an eBook/iBook.
For more information see the website of the booklet.

Herman



 

Confidence Intervals for Animal Resource Selection

Confidence Intervals for Animal Resource Selection

(Question posed and answer given on Research Gate)

What is the best method for determining the Confidence Interval of Animal Resource Selection datasets?

I am not sure what your data set looks like.
However, a very general method to get confidence intervals for any statistic of which the probability distribution is not available, is by bootstrapping.
This can be done using the freely available, high quality R statistical package.
Regards.
Herman

And on Mohammad's further question after bootstrapping using the statistical package R:

Dear Mohammad,

There is a package called boot which you have to load.

Note that using R is not straightforward, but it has excellent online documentation.

So if you are not familiar with it, you have to take some time to familiarize.

As to the boot package, as a result you get several confidence intervals.

The preferred one is BCa (bias corrected and accelerated), but it not always

converges. You have to take a look and pick another one.

The package is based on Efron and Tibshirani's book:

Introduction to the bootstrap (1993).

Furthermore, if you have SPSS available: it has provisions

for bootstrapping also nowadays.

They may be less general than provisions offered

in R which allows to program a function that calculates

the statistic you want to bootstrap.

There is also the book by myself, Don Mellenbergh and David Hand:

`Advising on research methods: A consultant's companion' (2008)

which gives a concise but clear discussion of the bootstrap.

It has also been indexed for Google Books, so that you

can consult it online. See: www.jvank.nl/ARMHome

Best,

Herman

Calculate interobserver agreement

Which one is the best way to calculate interobserver agreement related with behavioral observations?
I became a member of Research Gate (http://www.researchgate.net/home.Home.html). Below one of the questions and the answer formulated by me.

Which one is the best way to calculate interobserver agreement related with behavioral observations? (lizard stress study)
A colleague and I performed a study with lizards, where we subjected them to 4 different types of stress (cold, heat, low frequency noise and high frequency noise). We have videos of the behaviors they expressed during the experiment (flicking, head turns and so on). Now to begin the analysis of the videos, we need to make sure that our observations are more less the same, so we can exclude differences due observers bias.

We have agreed on the behaviors that we are recording and some of them are frequencies of events meanwhile others deal with duration of events. So far, we have the data of a section of our recordings that we analyzed separately and right now, we need to statistically probe that the data each one produced has no meaningful differences. Is there any statistical method you recommend?

Karen Mancera

There is a difference between assessment of association and assessment of agreement between observers.

If the variable for which you wish to calculate the agreement between observers:

a) is continuous (or ordinal with more than 5 values) things are easiest: you can use variance component analysis and calculate the interclass correlation coefficent (ICC), possibly corrected for any background factors (See: Snijder & Bosker, below). Use procedure VARCOMP in SPSS (or a similar procedure in R)

b) is dichotomous or categorical, you can use Cohen's kappa. Kappa can be calculated in SPSS using the RELIABILITY program.

c) is ordinal (with less than five values): use weighted kappas (weights concern off-diagonal distance in a observerXobserver crosstable for the item to be assessed). This can also been done in SPSS. But even if the number of options is less than 5, you can also apply variance component analysis as in a). Actually, the quadratically weighted kappa is equivalent to the ICC.

For b) and c) there is also a commercially available program called AGREE developed by Popping.

a) can also been done using multilevel analysis (cf. MLwiN), but that requires some extra skills.

All the above can be found in section 17.8 (page 453) of Adèr, Mellenbergh & Hand (2008), which also gives the appropriate references. This book is indexed for Google Books. Use the book's website to consult GB on this topic: www.jvank.nl/ARMHome

Herman Adèr

Adèr, H.j., Mellenbergh, G. J. & Hand, D. J. (2008). Advising on research methods: A consultant's companion. Huizen, The Netherlands: van Kessel.

Snijders, T.A.B. & Bosker, R. J. (1999). Multilevel analysis: An introduction to basic and advanced multilevel modeling. London: Sage.

A log book or diary to track all the steps taken for a project

A log book or diary to track all the steps taken for a project,

I became a member of Research Gate (http://www.researchgate.net/home.Home.html).
In the next posts I give some of the questions and answers formulated by me.

The first one is posted by Julia Law:

Diary of research steps as preface to writing up methodology
I would appreciate advice on methods used to write up a log book or diary to track all the steps I am taking for a few projects I am working on. I would like to develop a template to use for future projects also, so any suggestions as to what works and what doesn't would be appreciated. Thanks!

Your question is an interesting one. I have done some work on this problem.

But the way you formulate your question is a bit to general to answer in a satisfactory way. In particular, it is unclear in what field of research your projects are. Different disciplines have different methods to plan and specify research. For instance, in Medicine it is usual to formulate a protocol that is first thoroughly discussed by a local research committy before it is submitted to a medical ethical committy for assessment. There is also a diagramming method developed to specify a clinical trial (CONSORT statement), although I am doubtful about its usefulnes. However, most medical journals require such a statement for articles that describe intended research.

I have stressed the importance of properly documenting the steps taken in data analysis in our book (See Section 15.1 in Adèr, Mellenbergh and Hand, 2008). I also proposed a special diagramming method to represent methodological knowledge (See Appendix B of the same book). But in practice projects are quite varied and it is difficult to think up a general method to formally specify research procedures, let alone to develop a template that could be generally used. But maybe your own projects are quite similar and then the above may be useful.

Finally, for my own projects I always use a program called `Advanced Diary'. It is commercially available for a few dollars and makes it possibly to log each of your projects separately.

I hope this helps.

Herman Adèr

Adèr, H. J., Mellenbergh, G. J. and Hand, D. J. (2008). Advising on research methods: A consultant's companion. Johannes van Kessel: Huizen, The Netherlands.

The book has its own website: www.jvank.nl/ARMHome that links to Google Books so that you can inspect it online.

Advising on research methods. Selected topics 2011

A charming booklet containing papers on various methodological topics will be published next February, 15. It was written by participants of the yearly course `Methodological Advice' at the University of Amsterdam, this Fall. The idea was for the students to get acquainted with the whole process of drafting, submitting, reviewing, adapting and correcting for which so often support is asked during methodological consultancy.
The booklet has its own website:
Please take a look.
Herman

 

Addition to Sampling with and without replacement

The difference between sampling with replacement and sampling without replacement is twofold:

1. In the former case a temporal order is essentially assumed during the sampling process. Elements {a_1, a_2, .....,a_n} are drawn one after another, and at any time during the process, the same set of elements to draw from (S, say) is available.
2. Sampling without replacement can also be described as `grabbing', taking all the elements {a_1, a_2, ...,a_n} of the sample at the same time. If done sequentially, that set has to be adapted since the element that has been drawn should be removed. If  the initial set of elements to draw from is indicated by S_0, at each draw the set  from has to be adapted in the following way: S_i = S_(i-1) \ a_i.   

Funnily enough, sampling without replacement as occurs in real life research is essentially sequential and the above formulation in 2 seems counterintuitive since there is no distinct background set S apart from the sampling frame (Hand, 2008) and the population P (Think of a clinical trial in which patients are included one after another).  

In short, the two sampling mechanisms are very different and are applied in different situations.
Sampling without replacement is the sampling procedure that is commonly used in all emperical research.
Sampling with replacement is used in resampling strategies like bootstrapping. In such cases the notion of population from which the sample is drawn, is remote.

However, sampling with replacement also occurs in real life sampling, for instance in Capture-Recapture sampling aimed at determining population size of animals in a habitat, see section 12.3.6 of Agresti's book.

References
Alan Agresti (2002). Categorical data analysis. Second edition. Hoboken, NJ: Wiley.
David J. Hand (2008). Sampling. In H. J. Adèr, G. J. Mellenbergh, & D. J. Hand, Advising on research methods: A consultant's companion, Chapter 5. Huizen, The Netherlands: Van Kessel.

Sampling with and without replacement

In the bootstrapping literature (Efron & Tibshirani, 1993), the idea of ‘sampling with replacement’ is essential. It indicates that different bootstrap samples of size n are drawn from the same data set of size n, without producing the original data set over and over again. For instance, bootstrap samples from the data set of the numbers {1,9,11,12} may be: {1,1,11,12}; {1,9,11,9}; {1,9,11,12} and so on.
However, the procedure underlying sampling with replacement differs completely from sampling without replacement, in which the same unit can never appear twice.
In fact, sampling with replacement supposes that units are drawn one after another, and that a copy of the unit that was drawn is put back into the data (the actual replacement). In contrast, sampling without replacement allows to ‘grab’ n units at the same time.
The same effect as the sequential drawing described above for sampling with replacement would be obtained if n units were grabbed from a data set consisting of n copies of the original data. The data set would then be of size n².

References

Efron, B., & Tibshirani, R. J. (1993). An introduction to the bootstrap. New York: Chapman & Hall.

Multiple imputed missings in a dependent variable: How to analyze them?

During the analysis of the International PIRLS project, a comparative study on reading comprehension of students of Grade 4 and 5, the following problem was encountered during the analysis phase:
Some of the dependent variables (the plausible scores) are obtained by doing multiple impution on several scores (5 times). Usually, properly analyzing the data sets resulting from multiple imputation makes it necessary to do the analysis of each data set separately and then afterwards, combining the results in some way (as far as I know, the only standard statistical package that does this for you automatically, is Mplus).
However, in case the dependent variable is imputed, one can use a repeated measurement design. In the case described above, an extra level was added to a multilevel analysis with the 5 plausible scores as a repeated measurement.
Drawback of this approach in multilevel analysis (MLwiN) is that the data sets which are already enormous in this case, are quintupled.

Herman

Distribution outside Belgium and the Netherlands

It turned out to be very difficult to get distribution of the ARM book (Advising on research methods: A consultant's companion by Adèr, Mellenbergh and Hand) via Barnes and Noble in the US up and running.
Until these problems are solved, readers outside Belgium and the Netherlands can order directly from the publisher, Johannes van Kessel Publishing at: Publishing@jvank.nl. They charge only local mailing costs (euro 6.20). Please contact them by email. To order via bol.com, van Stockum or Selexyz see: www.jvank.nl/ARMHome
The proceedings of the 2007 KNAW symposium `Advising on research methods' can also be ordered by contacting the publisher.

Herman

Proceedings of the KNAW colloquium on Advising now accessible via Google Books

We submitted the text of the Proceedings of the 2007 Colloquium `Advising on research methods' to Google Books. The book is now accessible. In Google Books, search for `Advising on research methods' or for the ISBN number 9789079418039.
Herman

Two announcements

I have two announcements to make:

1. November 11, 2008, the proceedings of the 2007 KNAW Colloquium `Advising on research methods' were published. The Colloquium was organized by Adèr and Mellenbergh and was funded by the Royal Netherlands Academy of Arts and Sciences (KNAW). The full reference is: Adèr & Mellenbergh (2008). Advising on research methods. Proceedings of the 2007 KNAW Colloquium. Huizen, the Netherlands: Johannes van Kessel (ISBN: 978-90-79418-03-9). The table of contents can be found at: http://www.knaw.nl/colloquia/advising/index.cfm . The commercial edition is sold at 40 euros. It can be ordered via http://www.bol.com/ , http://www.vanstockum.nl/ and http://www.selexyz.nl/ .
2. We (Don Mellenbergh and myself) are organizing a public course on Advising, using the ARM-book (http://www.jvank.nl/ARMHome) as material. It will be held May 13-20, 2009 nearby the Amsterdam central station. Pre-registration is already possible. For more information, see: www.jvank.nl/ARMCourse

Herman Adèr

Several Announcements

Finally, finally a new entry in this Weblog.

First, a few announcements:

1. Last February (2008), we (Don Mellenbergh, David Hand and myself) have published a (hand-/text-) book on methodological advising. The full reference is: Adèr, H. J., G. J. Mellenbergh and D. J. Hand (2008). Advising on research methods: A consultant's companion. ISBN 978907941801-5. Huizen: Johannes van Kessel. The book has its own webpage: www.jvank.nl/ARMHome . It is also accessible via Google books (http://www.books.google.com/ ; type: `Advising on research methods' or something of the kind).
2. During a colloquium, also called `Advising on research methods', held in March 2007 in Amsterdam and sponsored by the Royal Netherlands Academy of Arts and Sciences (KNAW), a masterclass was held during which participants of the colloquium functioned both as advisors and as clients. This masterclass was recorded and is now available on DVD (free of charge). It represents unique material of excellent quality. To access the colloquium website, click: http://www.knaw.nl/colloquia/advising/index.cfm . To order the DVD click `DVD of the consultation interviews'.
3. We are working on the proceedings of the above colloquium. If it becomes available, it will be announced here.

Herman Adèr.

An alternative modelling procedure based on a strong Theory

Suppose we have a strong theory T that translates into several (for instance, three) alternative models M1, M2 and M3. Furthermore, suppose we are able to construct a research design R, based on theory T which allows us to verify our theory. Suppose we are able to implement an experiment (for instance, a clinical trial ) based on R and that we have collected data D during this experiment.

The usual procedure would be to test whether the data D are consistent with models M1, M2 and/or M3.

But we could also go about as follows:

Generate data according to the models M1, M2 and M3, resulting in three data sets D1, D2 and D3 and test whether these data sets could have resulted from the same population as D.

Remarks:

• The above is only possible if we have a strong theory T on which we can base our models beforehand.
• An methodological advantage is that the researcher is forced to formulate his/her theoretical concepts and translate them into models before (s)he starts his or her experiment.
• A second advantage is that deviations between D and Di (i= 1, 2, 3) give information both on the relationships between variables and on the influence of the underlying (possibly multivariate) distributions (this is assuming that our models are based on known theoretical distributions like the normal distribution, which is common practice).
• A third advantage seems to be that we can directly test the alternative hypothesis.
• This procedure can not be combined with crossvalidation (randomly splitting the data in two parts, one part to find models consistent with the data, another part to test those models), because in the first part, models are formulated that are consistent with (possibly multivariate) distribution violations in the data: the same violations are present in the second part of the data, too.

Questions:

1. Does a weak theory simply translates into a larger set of models?
2. Simulating data based on models M1, M2 and M3 may not be trivial. Can we use similar procedures as are used in MCMC (Markov chain Monte Carlo) ?
3. Can we use a Bayesian perspective, for instance by assuming that D1, D2 and D3 are based on prior distributions for the data D?
4. Is the above approach known and described in the `simulation community'?

Paul de Boeck: Always do a PCA

Another of Paul's rules used during statistical consultation (rule 5) was: `Always do a PCA, it tells you about sources of differences in the data and about the interaction between the two modes of the data set', was met with scepticism, notably by David Kaplan, who said that he recommended clients never to use PCA.

Comments:

• PCA is an abbreviation of `Principal component analysis'. It is essentially a data reduction technique requiring no assumptions about the distribution of the variables. In a nutshell, the technique results in a representation of the data relative an orthogonal coordinate system. Data reduction is obtained by considering only a few axes of the coordinate system.
• Methodologically, the drawback of the technique lays in the orthogonality, which in most cases is not realistic in view of the substantive meaning of the data. To mend this, a promax rotation can be used which allows to obtain non-orthogonal axes.
• As an alternative to PCA, confirmatory factor analysis (CFA) can be used in an exploratory way, in particular, if some assumtions can be made about the relation between factors and items (note that CFA does have several assumptions on the distribution of the observed variables, notably multinormality.)
• Although Paul had to endure heavy critique on his fifthst rule, in my opinion he had a point. In fact, it is common practice among data analysts to use PCA as a quick and dirty technique to explore the data, even if they know how to apply CFA. If promax rotation is used instead of varimax rotation, some of the objections against the orthogonality assumption are mitigated, although not completely met: a CFA on the other half of the data using the factor structure found with PCA may result in completely different estimated angles between the factors.
• For those who heard Paul's talk, the recommendation to use PCA was not supprising: he did put heavy emphasis on data exploration as an antidote to the often theory-centered approach that prevails in social science and behavioral research, and PCA can very well used in an exploratory way. As holds for all exploratoration, the truth is never ascertained. The analysis has to be confirmed either on a another part of the data or by doing a new, carefully designed experiment, that allows for unequivocal confirmation.
• As a last remark, I want to stress the fact that the use of PCA is not so straightforward as it seems (in particular, if one wants to have some confidence in the results). For a recent article on PCA see Costello and Osborne (2005).

References

Costella, A. B. and J. W. Osborne (2005). Best Practices in Exploratory Factor Analysis: Four Recommendations for Getting the Most From YourAnalysis. Practical Assessment Research & Evaluation, 10 (7).

Paul de Boeck: rules during consultation

During the March colloquium on Advising on research methods (see previous logs), Paul de Boeck (http://www.kuleuven.be/cv/u0002630.htm) gave seven rules that are important in consultancy when clients involved in social science research in which theoretical concepts are investigated using emperical research methods, come for advice. His presentation and the abstract of the talk can be found at:
http://www.knaw.nl/colloquia/advising/index.cfm#proceedings

I give the rules below and will comment on some of them in this and the next few blogs:

1. Not everything is worth being measured or can be measured, often the data are more interesting than the concept.
2. Always reflect on which type of covariation is meant when the relationship between concepts is considered. All too often, automatically the covariation over persons is used as the basis, without good reasons.
3. Measurement, reliability and validity testing, and hypothesis testing don’t need to be sequential steps, they can all be done simultaneously.
4. So-called psychometric criteria are not theory-independent, and sometimes the theoretical implications of the psychometric criteria are wrong.
5. Always do a PCA, it tells you about sources of differences in the data and about the interaction between the two modes of the data set.
6. One does not necessarily have to care about the scale level of the data.
7. Don’t construct indices of concepts, unless for descriptive summaries.

Ad rule 1 (Not everything is worth being measured or can be measured, often the data are more interesting than the concept). It should be stressed that this rule is thought to be most relevant during a consultation session. Let's take it apart: the first part (`Not everything is worth being measured or can be measured') is difficult to `sell' during consultation, because it means that during the study data were collected that were not worth collecting: this is particularly painful when it has to be said about the primary variables of an investigation. It is difficult to see what the second part (`often the data are more interesting than the concept') has to do with the first part: one can hardly say: `your study design started from a wrong idea and thus the data collection is worthless, but let's look at the data'. However (and this was clearly demonstrated during the presentation), a case can be made for a much looser connection between the data and the concepts to which they refer, because much can go wrong during implementation.

In particular (my addition): if enough data are available, a crossvalidation strategy can be useful, in which the data are randomly split in two parts. The first part is used for exploration and the emphasis is on the data (and their relation to study design and implementation), the second part is to investigate all worthwhile findings/hypotheses that came out of the exploration phase. Of course, in many cases not enough data were collected to allow this strategy. In this case two other strategies are available: (1) One may use the expected crossvalidation index (ECVI) given in Kaplan (page 117 e.v) which gives an impression of the crossvalidation adequacy of a model. (2) One may split the sample in unequal parts, using the first part for exploration as before and the (smaller) second part to test the findings of the exploration as before but now using small sample techniques like bootstrapping, if needed.

Kaplan, D. (2000). Structural Equation Modeling. Foundations and Extensions. Thousand Oaks London New Delhi: Sage Publications.

Small sample SEM

An issue that came up during a session on applications of structural equation modelling (SEM) after a lecture by David Kaplan at the Colloquium `Advising on research methods' in Amsterdam (29-30 March) was the fact that SEM is less often applied in medical/epidemiological research than it could be. Several bottlenecks can be (and were) identified:

1. The concept of `latent variable' (and, relatedly, the concept of (substantive) theory and corresponding model) seems to be less easy to conceptualize in Medicine than in the social/behavioral sciences.
2. Jules Ellis mentioned that SEM software is not easily optainable and costly and that specifying a SEM model in standard software is often awkward and difficult to achieve by the medical researcher/analyst.
3. Most SEM models require sample sizes that are too large for most small- to moderately- sized medical/epidemiological research projects.

Comments:

Ad 1. (The concept of `latent variable' seems to be less easy to conceptualize in Medicine than in the social/behavioral sciences).

Of course, this is an interesting but more or less philosophical point, if phrased in this way. In real-life medical/epidemiological research there often seem to be no compelling reasons to take a theoretical stand to start from. I won't go into the reasons for this.

However, there are several situations in which a theoretical conceptualization could help to specify a SEM model that is more appropriate than the regression-like models that are frequently used. For instance,

• In what is called `fundamental' or `basic' medical research, where complicated dynamic mechanisms are studied (think of genome studies or fysiological studies of illness progression), dynamic SEM models could be used to specify the dynamic process.
• In questionnaire design, confirmatory factor analysis can be used to analyze the factor structure, just as it is done in social science research (see de Vet et al. (2005)).

Ad 2 (SEM software is costly and specifying a SEM model in standard software is too difficult for the medical researcher/analyst).

In an e-mail afterwards, David Kaplan mentioned that a package to perform SEM modelling exists in R+ (a statistical package related to Splus, freely obtainable via the Internet, see, f.i.: http://cran.nedmirror.nl/). It turned out to be written by John Fox. Sources and binaries for SEM can be obtained at: http://finzi.psych.upenn.edu/R/library/sem/html/00Index.html

As to the other point, that the syntax of model specifation would be too complicated for the researcher-in-the-field: after all, many of them have also been able to learn how to use multilevel analysis. So, where there is a will, there is a way, even in learning how to use SEM, and in particular if there would be an clear need.

Ad 3 (Most SEM models require sample sizes that are too large).

To me, this seems the most important bottleneck for standard application in Medicine/Epidemiology. If SEM requires to have sample sizes of at least 500, application in most medical studies are out of the question.

In his comments, Jelte Wicherts mentioned that small sample size techniques are being studied. Afterwards, in some e-mail exchanges, I suggested that resampling techniques like bootstrapping could be applied in small sample situations. As it turns out, Fox's package, mentioned above, also contains a bootstrapping possibility (but see Kaplan's book chapter 5, where arguments are given why sample sizes would increase when non-normality has to be assumed).

Comment by David Kaplan:

I'm not sure what you mean by:

(but see Kaplan's book chapter 5, where arguments are given why sample sizes would increase when non-normality has to be assumed).

I think you mean that when estimating models with non-normal observed variables, larger sample sizes are typically needed for estimators to behave properly. That is partly true, and was true in the good old days. But now, there are estimators that don't require huge sample sizes. Also, I believe there are bootstrapping approaches to get standard errors when sample sizes are a bit smaller.

References

de Vet, H. C., Ader, H. J., Terwee, C. B., & Pouwer, F. (2005). Are factor analytical techniques used appropriately in the validation of health status questionnaires? A systematic review on the quality of factor analys of the SF-36. Quality of life research, 14(5), 1203–1218.

Kaplan, D. (2000). Structural Equation Modeling. Foundations and Extensions. Thousand Oaks London New Delhi: Sage Publications.

Colloquium `Advising on research methods'

This colloquium was organised by Don Mellenbergh and myself and held on March 29-30 in Amsterdam, the Netherlands.

Speakers were :

• Janice Derr (Advising in a multi-disciplinary setting)
• Steven Piantadosi (Research designs in Medicine)
• Don Mellenbergh (Advising on test construction)
• Gerald van Belle (Statistics and everyday life)
• Jules Ellis (Advising to policy makers in Health Care)
• Bo Lu (Bias correction using propensity scores)
• Paul de Boeck (Consulting in behaviour research)
• Willem Heiser (Survival skills in publishing)
• Robert Pool (Combining qualitative and quantitative methods)
• David Kaplan (Research problem and structural equation model)
• Denny Borsboom (Advising on test validity)
• Herman Adèr (Time and strategy)

The colloquium was preceded by a masterclass on Wednesday the 28st.

For more details, see: http://www.knaw.nl/colloquia/advising/
In the next few posts, I will describe some of the topics that were discussed.

Herman