Despite public pledges, leading scientific journals still allow statistical misconduct and refuse to correct it

A leading form of statistical malpractice in scientific studies is to retroactively comb through the data for "interesting" patterns; while such patterns may provide useful leads for future investigations, simply cherry-picking data that looks significant out of a study that has otherwise failed to prove out the researcher's initial hypothesis can generate false -- but plausible-seeming -- conclusions.

To combat this practice, many journals have signed up to the Consolidated Standards of Reporting Trials (CONSORT), a set of principles requiring researchers to publicly register, in advance, what data they will collect, and how they will analyze it, and to prominently note when they change methodologies after the fact, so that readers can treat any conclusions from these changes with additional care. CONSORT-compliant journals also promise to accept, review, and expeditiously publish correction letters when the papers they publish are found to have breached CONSORT standards.

Evidence-based medicine ninja Ben Goldacre (previously) and colleagues reviewed every single paper published in five leading CONSORT-signed-up journals (New England Journal of Medicine, The Lancet, Journal of the American Medical Association, British Medical Journal, and Annals of Internal Medicine) for six weeks, and, when they detected statistical practices that violated CONSORT principles, they informed the journals in writing and recorded and published their replies, and tabulated the findings, producing a league table of the journals that do the most to live up to their commitments to good statistical practice. They also analyzed the reasons that journals (and researchers) gave for not publishing corrections, and point out the wide gaps in the journal editors' and researchers' understanding of good statistical practice. Read the rest