Your Paper is a Brick in a Wall

Wharton’s Ethan Mollick is one of my favorite Twitter follows, and I’m hardly alone — Mollick summarizes tons of interesting papers that I would have otherwise missed. Like these two:

For Mollick, the upshot of these papers is clear: We hate hard thinking/being alone with our selves so much that we would rather act sadistically or hurt ourselves than do it

and that is a bad thing, obviously leading to idiocracy.

Except

and

and

Also

and best of all (quoting from one of the studies)

So a study (or two) might say something, or it might say something simply to you because of the priors you bring to it.

Because arguing on Twitter doesn’t work (well), another solution to this conundrum, economist Alex Tabarrok recommends, is a body of papers — a literature. A large literature that consistently tests a single hypothesis from different angles, times, places and methods, with multiple sources and types of evidence if possible. For instance, it is hard to dismiss the linkage between higher levels of air pollution and reduced performance — a linkage which has been made by the rich scientific literature on this hypothesis in all sorts of ways, in all sorts of times and places, involving all sorts of people.

Sometimes a single piece of research matters — like yesterday’s announced findings on booster shots and the Omicron variant. More often, a paper is a brick in a wall that will never be built. As Tabarrok says:

• Bigger samples are better.

• Small effects are to be distrusted.

• Multiple sources and types of evidence are desirable.

• Evaluate literatures not individual papers.

• Trust empirical papers which test other people’s theories more than empirical papers which test the author’s theory.
  

And don’t assume everyone will see what you see in your own research.