The BMJ has some neat features, such as paper-specific “instant responses” and published peer-review correspondence.
The latter feature allowed me to discover that in their initial “revise and resubmit” comments on a recent meta-analysis on sugar-sweetened beverages and type 2 diabetes, the BMJ manuscript committee requested the study’s authors to provide fewer data:
There is a very large number of supplementary files, tables and diagrams. It would be helpful if these could be reduced to the most important and essential supplementary items.
What? Why? Are they going to run out of server space? Give me ALL THE DATA! Finding the data I want in a huge 30mb supplementary data file is still much easier than asking the corresponding author for it 3 years later.
Is this kind of reviewer feedback common? I thought the top journals were generally on board with the open science trend.
Uh, what? That’s insane!
Romeo Stevens says
It might be the case that review panels are doing fuzzy pattern matching in much of their feedback rather than having explicit reasons. They do review a lot of bad papers (considering the crap that actually makes it through, think about what gets rejected and shudder).
Jess Riedel says
I agree that the reviewers did not handle this correctly. (Regardless, it was the editors job to step in and tell the reviewers they were making a mistake; I’m not sure what actually ended up happening here.) You’re right that there’s always a better solution than throwing out data.
However, the reviewers probably were responding to a huge data dump by the author’s which, without good organization, effectively drowns out the most important supplementary info. In practice, huge dumps like this actually make it harder on the typical reader to find the info their looking for, since most readers are looking for only a small number of the most important things. So, given a choice between a huge dump and a small selection of the most important info, it would actually be better to go with the latter. But a third, better solution, of course, is to organize the supplementary info correctly, so that the important info is at the front and easy to find, but the less important data is still archived somewhere.