Advising on research methods: Selected topics 2011

A similar publication as the one mentioned in the previous post, did appear the year before, following the same format. The title is:

Advising on research methods: Selected topics 2011

Edited by Herman J. Adèr and Gideon J. Mellenbergh

also published by Johannes van Kessel.
The paperback edition is sold out. But a eBook/iBook version is still available. This booklet is also indexed for Google Books. For more information, see it's website.

Table of contents:

From research question to statistical model

by Anja Sommavilla and Corinne Brenner

Pitfalls and payoffs in Internet sampling

by Corinne Brenner and Charlotte M. W. Gaasterland

Introduction to Computerized Adaptive Testing

by César-Reyer Vroom and Daniel A. Bannan

A critique of stepwise model selection methods

by Daniel A. Bannan and César-Reyer Vroom

A short introduction into survival analysis

by Charlotte M. W. Gaasterland and Mattis van den Bergh

On the relevance of mixed methods

by Mattis van den Bergh and Anja Sommavilla

Advising on research methods: Selected topics 2012

During last year's course `Advising on research methods' given at the Department of psychological methods by Don Mellenbergh and myself, participants (master's students) wrote papers on methodological topics which were afterwards collected in a booklet entitled:

Advising on research methods: Selected topics 2012

Edited by Herman J. Adèr and Gideon J. Mellenbergh

The booklet is published by Johannes van Kessel (ISBN 97-890-79418-21-3).
Since the papers were carefully reviewed, the contributions are of high quality.
The table of contents is:

Measurement Invariance

by Jonas Dalege and Loes Kreemers

A Comparison of Classical Test Theory

and Item Response Theory
by Marie K. Deserno

Unit nonresponse in Surveys

by Milou K. M. Lünnemann

Effect size: A meta-analytic perspective

by Adam Sasiadek and David Scholz

Outliers and Extreme Observations:

What are they and how to handle them?
by Suzan Q. Blommestijn and Esther A. Lietaert Peerbolte

Questionable research practices and scientific fraud

by Jochem Bout

The booklet is also available as an eBook/iBook.
For more information see the website of the booklet.

Herman



 

Confidence Intervals for Animal Resource Selection

Confidence Intervals for Animal Resource Selection

(Question posed and answer given on Research Gate)

What is the best method for determining the Confidence Interval of Animal Resource Selection datasets?

I am not sure what your data set looks like.
However, a very general method to get confidence intervals for any statistic of which the probability distribution is not available, is by bootstrapping.
This can be done using the freely available, high quality R statistical package.
Regards.
Herman

And on Mohammad's further question after bootstrapping using the statistical package R:

Dear Mohammad,

There is a package called boot which you have to load.

Note that using R is not straightforward, but it has excellent online documentation.

So if you are not familiar with it, you have to take some time to familiarize.

As to the boot package, as a result you get several confidence intervals.

The preferred one is BCa (bias corrected and accelerated), but it not always

converges. You have to take a look and pick another one.

The package is based on Efron and Tibshirani's book:

Introduction to the bootstrap (1993).

Furthermore, if you have SPSS available: it has provisions

for bootstrapping also nowadays.

They may be less general than provisions offered

in R which allows to program a function that calculates

the statistic you want to bootstrap.

There is also the book by myself, Don Mellenbergh and David Hand:

`Advising on research methods: A consultant's companion' (2008)

which gives a concise but clear discussion of the bootstrap.

It has also been indexed for Google Books, so that you

can consult it online. See: www.jvank.nl/ARMHome

Best,

Herman

Calculate interobserver agreement

Which one is the best way to calculate interobserver agreement related with behavioral observations?
I became a member of Research Gate (http://www.researchgate.net/home.Home.html). Below one of the questions and the answer formulated by me.

Which one is the best way to calculate interobserver agreement related with behavioral observations? (lizard stress study)
A colleague and I performed a study with lizards, where we subjected them to 4 different types of stress (cold, heat, low frequency noise and high frequency noise). We have videos of the behaviors they expressed during the experiment (flicking, head turns and so on). Now to begin the analysis of the videos, we need to make sure that our observations are more less the same, so we can exclude differences due observers bias.

We have agreed on the behaviors that we are recording and some of them are frequencies of events meanwhile others deal with duration of events. So far, we have the data of a section of our recordings that we analyzed separately and right now, we need to statistically probe that the data each one produced has no meaningful differences. Is there any statistical method you recommend?

Karen Mancera

There is a difference between assessment of association and assessment of agreement between observers.

If the variable for which you wish to calculate the agreement between observers:

a) is continuous (or ordinal with more than 5 values) things are easiest: you can use variance component analysis and calculate the interclass correlation coefficent (ICC), possibly corrected for any background factors (See: Snijder & Bosker, below). Use procedure VARCOMP in SPSS (or a similar procedure in R)

b) is dichotomous or categorical, you can use Cohen's kappa. Kappa can be calculated in SPSS using the RELIABILITY program.

c) is ordinal (with less than five values): use weighted kappas (weights concern off-diagonal distance in a observerXobserver crosstable for the item to be assessed). This can also been done in SPSS. But even if the number of options is less than 5, you can also apply variance component analysis as in a). Actually, the quadratically weighted kappa is equivalent to the ICC.

For b) and c) there is also a commercially available program called AGREE developed by Popping.

a) can also been done using multilevel analysis (cf. MLwiN), but that requires some extra skills.

All the above can be found in section 17.8 (page 453) of Adèr, Mellenbergh & Hand (2008), which also gives the appropriate references. This book is indexed for Google Books. Use the book's website to consult GB on this topic: www.jvank.nl/ARMHome

Herman Adèr

Adèr, H.j., Mellenbergh, G. J. & Hand, D. J. (2008). Advising on research methods: A consultant's companion. Huizen, The Netherlands: van Kessel.

Snijders, T.A.B. & Bosker, R. J. (1999). Multilevel analysis: An introduction to basic and advanced multilevel modeling. London: Sage.

A log book or diary to track all the steps taken for a project

A log book or diary to track all the steps taken for a project,

I became a member of Research Gate (http://www.researchgate.net/home.Home.html).
In the next posts I give some of the questions and answers formulated by me.

The first one is posted by Julia Law:

Diary of research steps as preface to writing up methodology
I would appreciate advice on methods used to write up a log book or diary to track all the steps I am taking for a few projects I am working on. I would like to develop a template to use for future projects also, so any suggestions as to what works and what doesn't would be appreciated. Thanks!

Your question is an interesting one. I have done some work on this problem.

But the way you formulate your question is a bit to general to answer in a satisfactory way. In particular, it is unclear in what field of research your projects are. Different disciplines have different methods to plan and specify research. For instance, in Medicine it is usual to formulate a protocol that is first thoroughly discussed by a local research committy before it is submitted to a medical ethical committy for assessment. There is also a diagramming method developed to specify a clinical trial (CONSORT statement), although I am doubtful about its usefulnes. However, most medical journals require such a statement for articles that describe intended research.

I have stressed the importance of properly documenting the steps taken in data analysis in our book (See Section 15.1 in Adèr, Mellenbergh and Hand, 2008). I also proposed a special diagramming method to represent methodological knowledge (See Appendix B of the same book). But in practice projects are quite varied and it is difficult to think up a general method to formally specify research procedures, let alone to develop a template that could be generally used. But maybe your own projects are quite similar and then the above may be useful.

Finally, for my own projects I always use a program called `Advanced Diary'. It is commercially available for a few dollars and makes it possibly to log each of your projects separately.

I hope this helps.

Herman Adèr

Adèr, H. J., Mellenbergh, G. J. and Hand, D. J. (2008). Advising on research methods: A consultant's companion. Johannes van Kessel: Huizen, The Netherlands.

The book has its own website: www.jvank.nl/ARMHome that links to Google Books so that you can inspect it online.

Advising on research methods. Selected topics 2011

A charming booklet containing papers on various methodological topics will be published next February, 15. It was written by participants of the yearly course `Methodological Advice' at the University of Amsterdam, this Fall. The idea was for the students to get acquainted with the whole process of drafting, submitting, reviewing, adapting and correcting for which so often support is asked during methodological consultancy.
The booklet has its own website:
Please take a look.
Herman

 

Addition to Sampling with and without replacement

The difference between sampling with replacement and sampling without replacement is twofold:

1. In the former case a temporal order is essentially assumed during the sampling process. Elements {a_1, a_2, .....,a_n} are drawn one after another, and at any time during the process, the same set of elements to draw from (S, say) is available.
2. Sampling without replacement can also be described as `grabbing', taking all the elements {a_1, a_2, ...,a_n} of the sample at the same time. If done sequentially, that set has to be adapted since the element that has been drawn should be removed. If  the initial set of elements to draw from is indicated by S_0, at each draw the set  from has to be adapted in the following way: S_i = S_(i-1) \ a_i.   

Funnily enough, sampling without replacement as occurs in real life research is essentially sequential and the above formulation in 2 seems counterintuitive since there is no distinct background set S apart from the sampling frame (Hand, 2008) and the population P (Think of a clinical trial in which patients are included one after another).  

In short, the two sampling mechanisms are very different and are applied in different situations.
Sampling without replacement is the sampling procedure that is commonly used in all emperical research.
Sampling with replacement is used in resampling strategies like bootstrapping. In such cases the notion of population from which the sample is drawn, is remote.

However, sampling with replacement also occurs in real life sampling, for instance in Capture-Recapture sampling aimed at determining population size of animals in a habitat, see section 12.3.6 of Agresti's book.

References
Alan Agresti (2002). Categorical data analysis. Second edition. Hoboken, NJ: Wiley.
David J. Hand (2008). Sampling. In H. J. Adèr, G. J. Mellenbergh, & D. J. Hand, Advising on research methods: A consultant's companion, Chapter 5. Huizen, The Netherlands: Van Kessel.