In the big data era, prioritize statistical significance in study design

But a study published this week in Nature should make all researchers – both readers and authors of papers – consider putting a little more thought into the methods part of the scientific process.

The study, led by biostatistician Simon Wendekker of Vanderbilt University Medical Center in Nashville, Tennessee, looks at how to make brain-wide association studies (BWAS) more robust (K. Kang et al. Nature https://doi.org/10.1038/s41586-024-08260-9; 2024). The main idea of ​​BWAS is to study a collection of brain images using statistical tools and machine-learning algorithms.

The aim is to predict which specific brain features or patterns of activity are associated with traits or behaviours, for example the ability to reason abstractly or the tendency to experience particular negative emotions.

But BWAS have a perennial and well-known problem of low replication: two studies on the same subject can reach different conclusions. A big part of the problem is that some BWAS studies require large numbers of samples to accurately characterise effects.

Small samples can exaggerate the association of a particular brain feature with a behaviour or trait. In the similar field of genome-wide association studies – which attempt to link differences in DNA with traits in health or disease – the problem of unreliability is being overcome by collecting data sets with thousands of samples from participants.

In the case of the brain, however, this is much more difficult, especially for researchers outside Europe and the United States. An hour of scanning in a molecular resonance imaging (MRI) machine costs about US$1,000. The US National Institutes of Health distributes about $2 billion for neuroimaging research each year, but few other countries have this level of resource.

Vandekar and his colleagues suggest that focusing on quality rather than quantity may be one answer. They analyzed more than 100,000 MRI scans of healthy adults and healthy children, as well as scans of children with mental-health conditions.

Their aim was to find out how factors such as age, sex, cognitive function and mental health are associated with brain structure and function in different study designs. For example, one study explored how brain volume changes with age. Vandekar and his co-authors found that, compared with one-time scans of many people – cross-sectional studies – repeated MRI scans of the same people over time produced more robust results (see R. J. Chauvin and N. U. F. Dosenbach Nature https://doi.org/10.1038/d41586-024-03650-5; 2024).

Such longitudinal studies have long proven their value in areas of science such as identifying biomarkers for chronic or degenerative diseases (Y. Guo et al. Nature Aging 4, 247–260; 2024). Although they do not work for certain types of questions that require cross-sectional studies, longitudinal studies are good at ruling out irrelevant factors that appear to be involved during smaller cross-sectional studies.

There are some caveats, however: for example, researchers conducting longitudinal studies must take care to leave sufficient intervals between measurements in any individual if they are to capture meaningful and statistically significant differences over time. Vandekar and his colleagues also emphasize that researchers must take into account both changes within individuals over time and differences between individuals.

All research needs to be planned. For BWAS, selecting participants in a way that yields robust results and using the right statistical models can improve the reliability of findings without the automatic need for massive sample sizes. The benefits of statistical rigour, in turn, highlight the need for greater collaboration between statisticians and neuroscientists, as they use more sophisticated data-handling methods in their research. These findings will be valuable to the neuroscience community, and deserve wider attention.

Many areas of science are plunging into data-driven discovery, increasingly aided by the pattern-seeking capabilities of artificial-intelligence algorithms. As they do so, questions of correlation and causation, and ensuring that findings are statistically significant and reproducible, are becoming even more relevant. This means that researchers should not lose sight of experimental design, whether they are studying or writing a paper.

Paying more attention to research methods, and to how a study obtains a signal of its effect, is the way to ensure that the results stand the test of time.