Air pollution is bad for us, says a mountain of research in very convincing ways.
In fact, the research is so convincing that air pollution is a clear and present danger to our health (including, perhaps even especially, our cognition) that New York Times journalist Binyamin Applebaum recently argued for using the health effects of air pollution to get people concerned about climate change, since the sources of the two are largely the same and air pollution doesn't suffer from the "too far in the future to care about" problem climate change has.
But wait, argues statistician Andrew Gelman. We should make a habit of being skeptical of large impact claims in science — even if those findings are consistent across a literature — because each of the individual studies might have the same structural bias. Indeed, Gelman thinks at least some of the papers that attribute large negative health impacts to air pollution fall into the genre of noisy studies that find significance when it isn’t there. As he puts it in this 2017 paper:
Null hypothesis significance testing (NHST) only works when you have enough accuracy that you can confidently reject the null hypothesis. You get this accuracy from a large sample of measurements with low bias and low variance. But you also need a large effect size. Or, at least, a large effect size, compared to the accuracy of your experiment…But we’ve grabbed all the low-hanging fruit. In medicine, public health, social science, and policy analysis we are studying smaller and smaller effects. These effects can still be important in aggregate, but each individual effect is small.
Gelman reinforces this point witha lot of funny blog posts taking apart various studies on air pollution.
OK, but what should society do about air pollution? Something? Nothing? Or wait and “do preregistered replication studies of the cognitive effects of air pollution” — as Gelman suggests — “because we think the topic is important and we want to understand it better”?
I’m reading Adam Grant’s latest book, “Think Again,” which argues that one of the most important traits people should develop is the ability to rethink their beliefs — and that thinking like a scientist (you know, hypothesizing, experimenting, testing, reaching tentative conclusions but always open to new evidence and explanations) is the best way to develop the power to “think again.”
Except that scientists (and researchers) are people, too. Are part of society. At some point, if they hope to have social impact — as opposed to just improving their individual life skills — they have to say what they think. They call it.
Calling it is part of being a scientist, too, as well as the essence of being a public expert. If the literature dictates, we must say: This is what we think, and this is what we think should be done.
Gelman is a public expert, too — about how sloppy and ambitious researchers, sloppy and ambitious science journals, credulous and innumerate science journalists and the perverse incentives of getting funded for science today have combined to form a scientific system that too often churns out findings that aren’t sound. Gelman calls it about this system on his blog nearly every day.
The clash there between science on air pollution and his skepticism about the strength of that science is a reflection of that call.
A strain of science culture — caricatured by Grant — fetishizes not calling it. That's professionally comfortable but socially sterile.
What should we do? Attack air pollution and attack bad science.
Arrange replication — and act anyway.
We need your call. So call it. Then call it again if you have to.