Paul de Boeck: rules during consultation

During the March colloquium on Advising on research methods (see previous logs), Paul de Boeck (http://www.kuleuven.be/cv/u0002630.htm) gave seven rules that are important in consultancy when clients involved in social science research in which theoretical concepts are investigated using emperical research methods, come for advice. His presentation and the abstract of the talk can be found at:
http://www.knaw.nl/colloquia/advising/index.cfm#proceedings

I give the rules below and will comment on some of them in this and the next few blogs:

1. Not everything is worth being measured or can be measured, often the data are more interesting than the concept.
2. Always reflect on which type of covariation is meant when the relationship between concepts is considered. All too often, automatically the covariation over persons is used as the basis, without good reasons.
3. Measurement, reliability and validity testing, and hypothesis testing don’t need to be sequential steps, they can all be done simultaneously.
4. So-called psychometric criteria are not theory-independent, and sometimes the theoretical implications of the psychometric criteria are wrong.
5. Always do a PCA, it tells you about sources of differences in the data and about the interaction between the two modes of the data set.
6. One does not necessarily have to care about the scale level of the data.
7. Don’t construct indices of concepts, unless for descriptive summaries.

Ad rule 1 (Not everything is worth being measured or can be measured, often the data are more interesting than the concept). It should be stressed that this rule is thought to be most relevant during a consultation session. Let's take it apart: the first part (`Not everything is worth being measured or can be measured') is difficult to `sell' during consultation, because it means that during the study data were collected that were not worth collecting: this is particularly painful when it has to be said about the primary variables of an investigation. It is difficult to see what the second part (`often the data are more interesting than the concept') has to do with the first part: one can hardly say: `your study design started from a wrong idea and thus the data collection is worthless, but let's look at the data'. However (and this was clearly demonstrated during the presentation), a case can be made for a much looser connection between the data and the concepts to which they refer, because much can go wrong during implementation.

In particular (my addition): if enough data are available, a crossvalidation strategy can be useful, in which the data are randomly split in two parts. The first part is used for exploration and the emphasis is on the data (and their relation to study design and implementation), the second part is to investigate all worthwhile findings/hypotheses that came out of the exploration phase. Of course, in many cases not enough data were collected to allow this strategy. In this case two other strategies are available: (1) One may use the expected crossvalidation index (ECVI) given in Kaplan (page 117 e.v) which gives an impression of the crossvalidation adequacy of a model. (2) One may split the sample in unequal parts, using the first part for exploration as before and the (smaller) second part to test the findings of the exploration as before but now using small sample techniques like bootstrapping, if needed.

Kaplan, D. (2000). Structural Equation Modeling. Foundations and Extensions. Thousand Oaks London New Delhi: Sage Publications.