May 3, 2010

"Peer review" – It's not what you think

By "Peer review" – It's not what you think

Author: Reed Hopper

There is a fundamental difference between research science and regulatory science–-time and money. While research scientists are generally free to pursue a line of inquiry to wherever it leads however long it takes, regulatory scientists do not have that luxury. To the contrary, to meet administrative deadlines and budgets, the latter are limited to a more cursory examination of the scientific information. This fact is evident in the peer review process.

Peer review in a research setting is usually quite different from peer review in a regulatory setting. In the research setting, a peer reviewer is not only allowed but encouraged to take a hard look at the data underlying the study under review and will often seek to replicate the author’s experiments to confirm the validity of the author’s conclusions. In a regulatory setting, however, a peer reviewer is often neither allowed nor encouraged to take a hard look at the data underling the study under review and virtually never seeks to replicate the author’s experiments to confirm the validity of the author’s conclusions. This has nothing to do with the integrity or ability of the regulatory peer reviewer. It has everything to do with time and money. As a practical matter, peer review of the science supporting an administrative rule is limited to whether the author’s conclusions follow logically from the author’s cited data or assumptions. As Ruhl and Salzman have noted: "[P]eer review would grind itself … to a screeching halt were it to require peer reviewers to engage in independent testing and data analysis." And therein lies the rub. The regulatory peer reviewer may never see or even have access to the raw data on which the study was based.

The unhappy result of this approach is that the regulatory peer review does not and often cannot reveal the author’s bias or error. But the peer review gives the author’s conclusions the patina of legitimacy because, after all, the study was "peer reviewed" by independent experts. If the study is flawed, the peer review perpetuates the fraud or mistake and may even propel it to the level of undisputed scientific fact. In that case, we all lose.

Because the peer review process in the regulatory setting is often not rigorous enough to uncover bias or error, it undermines public confidence in regulatory decision making instead of building public trust. When we read, as we have in recent months, that expert agencies around the world have adopted culture-changing rules and policies relying on the conclusions of climate scientists without ever seeing the underlying data on which those conclusions were based, we have to ask if the regulatory peer review process is giving us what we want. It certainly is not giving us what we need. Perhaps it's time we consider something more.

What to read next