`Methodological considerations on an empirical study are relevant only in relation to the results'.
This idea is in line with the concepts of content quality of a study, defined as:
Content quality. The Content quality of a study indicates its bearing on the solution of the initial research problem.
Now assume that a study is methodologically flawed in some way. In other words, the content quality is low. The question now arises what is less harmful:
- Something that could have been uncovered by using the right methodology, but that was not found due to these flaws.
- Due to methodological flaws, erroneous conclusions were drawn (and published).
Most researchers would indicate the first option as the least harmful, reasoning that it happens all the time: different researchers draw different conclusions looking at the same data, and historically, new perspectives are opened time and again. However, one may ask what is meant by `the right methodology' in option 1: if this indicates that something could have been found, but wasn't, since the less subtle analysis technique was applied, the error could have been easily avoided. In particular, if the data set was published, as is required in some fields of research, your name as a serious researcher would be endangered if somebody else, repeating the analyses the right way, would come up with something completely different! (maybe that's why some researchers are hesitant to make their data publicly available).
In the second case, if these wrong conclusions are simply negative ones (we did not find nor report an effect although there is one present in the data), this is of course a pity, but there is no serious harm done. Even in case somebody showed that a positive conclusion could have been drawn, nobody will look askance at you, since your conclusions were at the safe side. And the one who has done the replication should explain very well why his results are deviant. If he or she has applied some outlandish technique that is not familiar to most, (s)he may even be considered a niggler who is out to harm your good name as a researcher.
(note that in the above, we talk about published results, although this may not have been obvious at the beginning).
