Let the data talk: embrace exploratory research

Attending a psychology conference this month, I was struck by a disturbing trend in social-science research. The abstract book revealed that nearly all of the 1,500 posters and talks were related to some kind of confirmatory work — using statistical analysis to test existing hypotheses. Hardly anyone reported the open-ended exploratory research needed to come up with those hypotheses.

In my view, it is important for the social sciences to adopt a research culture that strikes a balance between hypothesis testing and hypothesis generation, embracing confirmatory as well as exploratory studies.

To understand the difference between exploratory and confirmatory studies, consider a research group interested in the relationship between children watching violent movies and aggressive behavior.

In an exploratory study, the group might ask parents about children\’s aggressive behavior and collect information about household habits, personal characteristics, and media content that might influence children.

The authors will inspect the data in a number of ways, looking for any patterns. The patterns that emerge, along with theoretical background knowledge, can be used to make specific predictions – for example, explanations given by parents about violent content may reduce media-induced violence in adolescents, but not in younger children.

Confirmatory studies can attempt to support or question these hypotheses using new data sets. To properly conduct confirmatory studies, best practice dictates that researchers outline their analysis plan before they begin their work, often through pre-registration – by recording it in a public repository. Pre-registration helps prevent researchers from forming their hypotheses and analysis after looking at their data.

For example, in the above example, if the analyses for one group do not confirm the general hypothesis, the researchers may be tempted to change their type of analysis until it shows a statistically significant effect for early adolescent boys, after controlling for family income.

This widely reviled practice opens the door to false-positive findings. Like many in the social sciences, I was trained to think of hypothesis testing as the core work of the researcher, with exploration rarely worthy of publication.

But I changed my mind because I lost confidence in my ability to properly preregister my studies. I wholeheartedly believe that preregistration is vital to transparent science. But often, I would find that I had not done enough exploration to properly plan my hypothesis-testing analyses.

How could a researcher dealing with the question of aggression, for example, know without exploration which of the many variables – age, gender, socioeconomic background and more – might be associated with behavioral change, and how? Similarly, in my own work, I asked myself: Should I have formulated my statistical model differently? Would a different analysis lead to different conclusions? Did I miss something else interesting in my data? I felt I couldn’t let my data speak for themselves. I should have done more research before committing to my confirmatory analysis.