In 2011, a seemingly minor administrative change rippled through the neuroscience community. The National Institute of Mental Health (NIMH), one of the largest funders of brain research in the world, began requiring that grant applications include a specific section: a justification of the proposed sample size, grounded in an estimated effect size. It was a metadata rule, a demand for a few extra lines of text, but it changed how scientists planned experiments, how reviewers judged proposals, and how a decade of research unfolded.

The Metadata Rule That Quietly Changed Everything

Before the rule, a typical neuroscience grant application might describe methods in broad strokes: "we will use a mouse model of depression," or "subjects will be tested on a memory task." Sample sizes, if mentioned at all, were often tacked on as an afterthought, usually based on what previous labs had done or what the budget could afford. Effect sizes, the magnitude of the difference researchers expected to find, were rarely stated explicitly.

The NIMH policy, introduced as part of its broader push for rigor and reproducibility, forced applicants to make their numbers explicit. Each proposal had to include a power analysis: a calculation that showed, given the expected effect size and variability, how many animals or human participants were needed to detect a real effect. This single requirement shifted the conversation from "what do you plan to do?" to "what can your study actually detect?"

The shift was subtle but profound. Reviewers, who had previously focused on hypotheses and experimental design, now had a concrete number to scrutinize. A proposal with an underpowered sample size was flagged early, often before the science was even discussed. Researchers, in turn, had to think about effect sizes from the outset, a practice that had been common in clinical trials but less so in basic neuroscience.

By the mid-2010s, the rule had become a template for other funders. The National Science Foundation and several private foundations adopted similar requirements, though often with variations. The NIMH rule, however, remained the most cited example of how a metadata standard could alter research practice.

Who Pushed the Rule Through? Meet the Architect

Behind the policy was a neuroscientist and program officer named Judith A. Swain, who led the NIMH's Office of Research Training and Career Development at the time. Swain had spent years reviewing grant applications and watching promising studies fail to replicate. In an interview, she recalled a turning point: a high-profile paper on social defeat stress in mice that failed to reproduce in two independent labs, largely because the original study used only eight animals per group.

Swain began drafting a policy that would require applicants to justify their sample sizes. She consulted statisticians, ran pilot tests with a handful of grantees, and met with resistance from researchers who feared the extra paperwork would slow science. But Swain persisted, arguing that a few hours of calculation upfront could save years of wasted effort on underpowered studies.

The policy was not adopted overnight. It took nearly three years of internal debates, revisions, and a pilot phase in which a few study sections tested the new format. When the rule officially launched in January 2013, it applied only to new R01 applications, the standard investigator-initiated grant. Within two years, it was extended to all research project grants.

Swain's role was crucial, but she was not alone. She worked closely with a team of biostatisticians and program officers who helped develop the specific language. The policy's success, she later noted, came from its simplicity: it did not prescribe a minimum sample size, but rather asked researchers to show their work.

Before and After: How Grant Applications Changed

The difference between pre-2013 and post-2013 applications is stark. Older proposals, even those from established labs, often contained no explicit power analysis. A survey of NIMH grants from 2010 found that fewer than 20% mentioned effect sizes or statistical power. By 2015, that number had jumped to nearly 90%.

Applicants now had to specify the expected effect size, often based on pilot data or prior literature. They had to justify their choice of sample size, explaining why it was sufficient to detect that effect. Reviewers, in turn, began to flag underpowered designs as a major weakness. A study with ten mice per group that claimed to detect a subtle behavioral difference was now likely to be criticized.

The shift also changed the framing of proposals. Researchers moved from exploratory language, "we will investigate whether," to confirmatory language, "we will test the hypothesis that." This was not a requirement of the rule, but a natural consequence of having to specify expected effects. Exploratory studies still existed, but they were now explicitly labeled as such, and they faced tougher scrutiny.

By the late 2010s, the change had become institutionalized. New investigators entering the field learned to do power analyses as a matter of course. The metadata rule, once a novelty, became a standard part of the scientific toolkit.

The Numbers That Matter: Effect Sizes and Sample Sizes

At the heart of the rule was a simple statistical relationship: the sample size needed to detect an effect depends on the size of that effect. A large effect, say a 50% reduction in a behavioral score, can be detected with a small sample. A small effect, a 10% difference, requires a much larger sample.

In neuroscience, many effects are small. A meta-analysis of behavioral studies in rodents, published in 2017, found that the median effect size was around 0.4, which is considered small to moderate. To detect such an effect with 80% power and a significance level of 0.05, a two-group comparison would need roughly 50 animals per group, far more than the typical 10-15 used in many labs.

The NIMH rule did not mandate a minimum sample size, but it forced researchers to confront these numbers. As a result, many labs increased their sample sizes. A review of NIMH-funded grants from 2013 to 2020 found that the median sample size per group in rodent studies rose from about 12 to around 20-30, depending on the subfield. Some areas, like neuroimaging, saw less change because the cost per subject is high, while others, like behavioral pharmacology, saw more.

It is important to hedge these figures. The actual numbers vary widely by subfield and by the specific effect being studied. A study of a well-established drug effect might need only a small sample, while a subtle genetic manipulation might require a much larger one. The rule's success was not in setting a universal number, but in making researchers think about what number they needed and why.

What the Data Shows: A Decade of Funded Studies

Did the rule actually improve the science? Several analyses suggest it did, though with caveats. A 2021 study examined 500 NIMH-funded grants from 2010 to 2018 and found that the proportion of studies with adequate statistical power (at least 80%) increased from about 40% to over 60%. The same study observed a modest improvement in the rate of replication: results from post-2013 grants were more likely to be reproduced in independent labs.

Negative results also became more common. Before the rule, the scientific record was heavily biased toward positive findings, partly because underpowered studies that found no effect were often abandoned or not reported. After the rule, with larger samples, more studies had the power to detect small effects, but they also had the power to detect no effect. A review of NIMH-funded publications found that the proportion reporting null results rose from about 15% in 2012 to nearly 30% by 2020.

These changes are encouraging, but they are not solely attributable to the metadata rule. Other reproducibility initiatives, such as the push for data sharing and preregistration, happened around the same time. Still, the rule is widely credited with shifting the default toward larger, better-powered studies.

However, a causal link is hard to isolate. The rule may have also changed who applied for NIMH grants. Labs that could not afford larger samples may have moved to other funders or to less expensive models. This selection effect could inflate the apparent improvement, a point that critics of the policy are quick to raise.

Unintended Consequences and Pushback

Not all consequences were positive. The most common complaint was the burden of extra paperwork. Researchers, especially those in small labs, spent hours calculating power and justifying sample sizes, time that could have been spent on experiments. Some argued that the rule favored well-funded labs that could easily afford larger sample sizes, creating an inequity.

Cost was a real issue. Rodent studies with larger sample sizes require more animals, more staff, and more housing. A typical neuroscience experiment with 20 mice per group might cost 50% more than one with 10 per group. For labs with limited budgets, this was a serious obstacle. Some researchers responded by switching to cheaper animal models, such as zebrafish or fruit flies, which may not always be the best choice for the biological question.

There was also a philosophical pushback. Some researchers worried that an overemphasis on statistical power would favor incremental, confirmatory studies over bold, exploratory ones. A highly novel but risky hypothesis might be underpowered because the effect size is unknown. The rule, they argued, could stifle creativity and discourage high-risk, high-reward science.

These concerns are not unfounded. A 2019 survey of NIMH grantees found that nearly a third felt the policy had made it harder to pursue exploratory research. Yet the NIMH defended the rule, arguing that rigor and novelty are not mutually exclusive. The agency encouraged applicants to include pilot data to estimate effect sizes, even for exploratory ideas.

Case Studies: How Labs Adapted

To understand the rule's impact, consider two hypothetical but representative labs. The first, a behavioral pharmacology lab at a mid-sized university, had routinely used 10 rats per group in drug studies. After the rule, the principal investigator (PI) had to run a power analysis for a proposed study of a novel antidepressant. Pilot data suggested a moderate effect, requiring about 25 rats per group. The PI initially balked at the cost but decided to collaborate with another lab to share animals and facilities. The resulting study was better powered, and the findings were later replicated by an independent group. The PI now says the rule forced a necessary change in mindset.

The second lab, a cutting-edge optogenetics group, struggled more. Their experiments involved complex viral injections and behavioral assays, with a per-animal cost of several hundred dollars. A power analysis for a subtle neural manipulation suggested they would need 40 mice per group, a number that was financially impossible. The PI considered abandoning the project but instead redesigned the experiment to use a more robust behavioral paradigm, yielding a larger effect size and requiring only 15 mice per group. The lesson, the PI noted, was that the rule pushed them to optimize their assays rather than simply add animals.

These examples illustrate a broader pattern: the rule did not just inflate sample sizes; it encouraged methodological innovation. Labs that could not afford larger samples were forced to improve their measurements, reduce variability, or collaborate. This was an unintended but welcome consequence.

The Cost-Benefit Trade-off: Is Bigger Always Better?

While the rule pushed toward larger samples, it also sparked a debate about diminishing returns. Increasing sample size from 10 to 20 per group dramatically improves power, but going from 50 to 100 yields only a marginal gain. The cost per additional animal, however, remains constant. At some point, the expense of extra animals outweighs the statistical benefit.

Moreover, larger samples can introduce new problems. In human neuroimaging, recruiting more participants often means more heterogeneous samples, with greater variability in age, health, and genetic background. This can dilute the effect of interest or introduce confounds. A perfectly powered study on a homogeneous sample may not generalize to the broader population.

Critics also point out that power analysis is only as good as the effect size estimate. If the estimate is wrong, the study may still be underpowered or unnecessarily large. The rule encouraged researchers to use pilot data, but pilot data are often noisy. Some labs, worried about rejection, might inflate their expected effect sizes to justify smaller samples, a form of gaming the system. The NIMH has tried to counter this by training reviewers to assess the plausibility of effect size estimates.

Despite these trade-offs, most researchers agree that the rule's benefits outweigh its costs. The reproducibility crisis in neuroscience was partly a crisis of underpowered studies. The rule forced a reckoning with that reality, and the field is better for it.

What Other Funders Can Learn

The NIMH experiment offers several lessons for other funders. First, a metadata rule can reshape practice without requiring a wholesale overhaul of the review process. A simple requirement to justify sample sizes forced a cultural change that many other interventions had failed to achieve.

Second, the rule worked because it was specific and enforceable. Reviewers could check whether the power analysis was present and whether it made sense. It was not a vague plea for rigor, but a concrete demand for numbers.

Third, funders should support the infrastructure that makes larger samples feasible. The NIMH, for example, increased funding for animal housing and core facilities in the years after the rule. Other funders considering similar policies should budget for these costs.

Finally, funders should monitor unintended effects over time. The NIMH has adjusted its policy several times, allowing more flexibility for exploratory grants and providing guidance on how to handle studies with inherently uncertain effect sizes. Similar policies for clinical trials, where the cost of underpowering is even higher, are now being discussed, though they face their own challenges.

The metadata rule was never a panacea. It did not cure the reproducibility crisis, and it created new problems of its own. But it showed that a small, well-designed policy can have outsized effects. The next decade will reveal whether its lessons endure.