In early 2025, a preprint appeared on bioRxiv claiming a machine learning model could predict ICU mortality with remarkable accuracy. The authors, a team of computational biologists at the University of Copenhagen, had trained their model on the MIMIC-IV database, a large, publicly available critical care dataset. They reported an area under the curve (AUC) of 0.94, a figure that would have ranked among the best in the field. The paper was quickly picked up by news outlets and shared widely on social media. But a careful reader who dug into the supplementary files would have found two negative controls that told a different story. Both failed to beat a random baseline. The authors had acknowledged these failures in a footnote buried in the appendix, but the footnote was absent from the main text. This article traces how those controls slipped through peer review, what it took to uncover them, and what the episode reveals about the incentives that shape scientific publishing.

A Model That Looked Too Good on Arrival

The preprint, titled "Predicting ICU Mortality with Deep Learning on Electronic Health Records," reported that a neural network could identify patients at risk of dying in the intensive care unit with an accuracy that outperformed existing clinical scores. The authors used MIMIC-IV, which contains de-identified health data for over 50,000 ICU admissions. Their methodology appeared sound at first glance: they split the data into training and test sets, used cross-validation, and reported performance metrics that were consistently high across multiple patient subgroups. The main text was clean, well-written, and included several robustness checks, such as varying the model's hyperparameters and testing on a held-out cohort. Reviewers who saw the preprint praised its thoroughness.

But the supplementary files were a different matter. Tucked away in a section labeled "Additional Analyses," the authors had included two negative controls. The first was a shuffled-label test: they randomly permuted the outcome labels and retrained the model on the scrambled data. The second was a null feature permutation: they randomly shuffled the input features while keeping the labels intact. In both cases, the model's performance dropped to chance levels, which is exactly what one would expect if the model had learned real patterns. The problem was that the authors reported these results in a single footnote, with no accompanying figure or discussion in the main text. A reader who never opened the supplementary PDF would never know they existed.

Negative controls are a standard tool in machine learning to detect data leakage, overfitting, or hidden confounds. If a model performs well on shuffled data, it suggests the model is picking up on spurious correlations rather than genuine signal. In this case, the controls passed, which was good news for the model's validity. But the way they were buried raised a question: why would authors who had done the right thing hide the evidence? The answer, as later became clear, had more to do with the incentives of the publishing system than with any intentional deception.

Peer reviewers, it turns out, rarely inspect supplementary materials. A 2019 survey of reviewers found that fewer than a third said they regularly opened supplementary files, and even fewer examined code or raw data. Reviewers are typically unpaid, overburdened, and focused on the main claims. When a paper reports a striking result, the natural instinct is to check the methods and the primary analysis, not to hunt for buried caveats. The preprint's authors, knowing this, may have reasoned that the footnote was enough to satisfy scientific integrity without drawing attention to the controls' existence.

The preprint was posted in January 2025, and within weeks it had been cited by several other groups. One team of clinicians at a hospital in the Netherlands used the model's predictions to guide a retrospective analysis of their own ICU patients. Another group incorporated the model into a review article on AI in critical care. The paper's high profile meant that any eventual retraction or correction would have a ripple effect across the field.

The Two Controls That Should Have Raised Red Flags

The first control, the shuffled-label test, involved randomly reassigning the mortality outcome to patients and then retraining the model. In a properly specified model, this should result in chance-level performance, because the labels no longer correspond to any real signal. The authors reported that the model's AUC dropped to 0.51, essentially random. The second control, the null feature permutation, shuffled the input features while keeping the true labels. This tests whether the model relies on any single feature that, when scrambled, would break the pattern. Again, the AUC fell to 0.49. Both results were consistent with a model that had learned genuine associations.

At first glance, these controls seem reassuring. They suggest the model was not overfitting to noise. But a deeper look reveals a subtle issue. The shuffled-label test is only meaningful if the model is trained from scratch on the shuffled data. If the authors had instead used the already-trained model and simply evaluated it on shuffled labels, the test would be meaningless. The supplementary files do not specify the training procedure for the controls, only that they were "trained under the same conditions." This ambiguity is typical of many preprints, but it is precisely the kind of detail that peer reviewers are supposed to catch.

The second control, the null feature permutation, is even more telling. If the model's performance drops to chance when features are shuffled, it suggests the model is sensitive to the joint distribution of features. But this control does not rule out all forms of leakage. For example, if the data contained duplicate patients or temporal correlations that the model exploited, shuffling features within a row would not destroy those correlations. The authors did not test for such leakage, and their footnote did not mention it.

The fact that both controls were relegated to a footnote, rather than a full section in the main text, is itself a red flag. In a field where negative controls are often seen as a mark of rigor, why would authors hide them? One possibility is that the authors felt the controls were so standard that they did not warrant discussion. Another is that they feared that highlighting the controls would invite scrutiny of other aspects of the model. Either way, the burial of the controls made it easier for reviewers and readers to overlook them.

How the Review Process Let Them Slip Through

The preprint was submitted to the Journal of Biomedical Informatics in March 2025. The journal, a well-regarded title in the field, sent it to three reviewers: a clinical intensivist, a biostatistician, and a machine learning researcher. The intensivist praised the model's performance and asked for additional analyses on a different dataset. The biostatistician raised concerns about the clinical applicability and suggested the authors include a decision curve analysis. The machine learning expert asked for more details on the model's calibration and requested that the authors report confidence intervals for their performance metrics. None of the reviewers mentioned the negative controls.

When the authors responded to the reviews, they addressed each point in a point-by-point rebuttal. They added the decision curve analysis, included calibration plots, and reported confidence intervals. In their response, they also noted, almost as an aside, that "as shown in the supplementary materials, the model's performance was robust to label shuffling and feature permutation." This was the first time the controls were mentioned in the review process, and it was buried in a single sentence. The reviewers did not follow up.

The editorial process at this journal, like many others, encourages speed. Editors are under pressure to publish high-impact papers quickly, and reviewers are asked to submit their reports within two weeks. The average time from submission to first decision at this journal was around 30 days. In such an environment, there is little incentive to dig into supplementary files, especially when the main results appear solid. The editors accepted the paper in June 2025, with only minor revisions.

The final published version, which appeared in August 2025, included the negative controls in the supplementary materials, but the main text still did not mention them. The footnote was expanded to a short paragraph, but it remained in the appendix. The paper was now peer-reviewed and published, giving it an additional layer of credibility. The buried controls were now even more deeply buried, hidden behind a paywall and a PDF that few would read.

This case is not unique. A 2022 study of 100 machine learning papers in biology found that fewer than 10% reported any negative controls, and those that did often buried them in supplementary files. The authors of that study argued that negative controls should be mandatory in the main text of any paper that claims predictive accuracy. But such requirements have not been widely adopted, in part because journals fear that adding more requirements will slow down the review process and discourage submissions.

A Replication Attempt That Uncovered the Trail

In September 2025, a team at Stanford University, led by postdoctoral researcher Elena Vasquez, attempted to replicate the model's results. Vasquez had been working on a similar problem and was curious about the preprint's high performance. She downloaded the code and data from the repository linked in the paper, which the authors had made public. She reran the main analysis and obtained an AUC of 0.93, close to the reported 0.94. But when she tried to reproduce the negative controls, she could not find them in the code.

Vasquez emailed the authors, who pointed her to the supplementary PDF. There, she found the footnote describing the controls, but the code to generate them was not included in the repository. She asked for the exact scripts, and the authors provided them after a few weeks. When she ran the scripts on her own machine, she got the same results: the model's performance dropped to chance. But she also noticed something odd. The scripts for the shuffled-label test used a fixed random seed that was different from the one used in the main training. This meant that the control was not directly comparable to the main model, because the model's weights could have been initialized differently.

Vasquez decided to run a more rigorous version of the control. She retrained the model from scratch on shuffled labels, using the same random seed as the main training. The AUC dropped to 0.50, as expected. She also ran a permutation test on the features, shuffling each feature independently across patients. The AUC again dropped to 0.50. These results were consistent with the authors' claims. But then she decided to test a different kind of leakage: temporal leakage. The ICU database contained time-stamped records, and the model had been trained on data from earlier years and tested on later years. Vasquez suspected that the model might be exploiting time trends in the data, such as changes in treatment protocols or coding practices.

To test this, she trained the model on data from 2018 to 2020 and tested it on data from 2021 to 2022. The AUC dropped to 0.78, still good but substantially lower than the reported 0.94. She also trained the model on a random subset of patients from all years and tested it on the remaining patients, which is the standard approach. The AUC was 0.94. This suggested that the model's high performance was partly due to temporal correlations, not just patient-level features. Vasquez published her findings as a commentary in the same journal, noting that the original paper's negative controls were insufficient to rule out temporal leakage.

The commentary was published in December 2025, and it quickly gained attention. The original authors responded by saying that temporal leakage was a known limitation and that they had not intended to overstate the model's generalizability. But the damage was done. The preprint had been cited over 100 times, and many of those citing papers had used the model's predictions as if they were valid. The field now had to reconsider the model's true utility.

Why Negative Results Are Systematically Undervalued

The episode illustrates a broader problem in science: negative results are systematically undervalued. Journals prefer to publish novel, positive claims that attract citations and media attention. Funding agencies reward breakthroughs, not null findings. Researchers, aware of these incentives, may downplay or hide negative controls that could cast doubt on their main results. In this case, the authors did not hide the controls entirely, but they did relegate them to a footnote, which is almost as good as hiding them.

Several studies have documented the prevalence of publication bias. A 2014 analysis of 4,600 papers in psychology found that 96% reported positive results, even though it is unlikely that all hypotheses were true. In machine learning, the bias is even more pronounced, because models are often tuned to achieve high performance on a specific dataset, and negative results are rarely reported. The result is a literature that overstates the reliability of predictive models.

There are some efforts to change this. For example, the journal PLOS ONE has a section for "Negative Results" and has been publishing null findings since 2014. The journal Royal Society Open Science offers a "Registered Reports" format, where the study is peer-reviewed before the results are known, which was introduced in 2015. Preprint servers like arXiv and bioRxiv allow authors to post negative results without fear of rejection, and since 2016, bioRxiv has seen a steady increase in preprints that report null findings. But these venues are still a minority, and the incentive structure of academia remains skewed toward positive findings.

In this case, the authors' decision to bury the controls was not necessarily malicious. They may have believed that the controls were standard and did not need discussion. They may have worried that highlighting them would invite scrutiny of other aspects of the model. But the effect was the same: the negative results were invisible to most readers, and the model's limitations were not fully appreciated until a replication attempt uncovered them.

Practical Fixes for Preprint and Peer Review

What can be done to prevent such oversights in the future? One suggestion is to require that negative controls be reported in the main text, not just in supplementary files. The journal could have a checkbox in the submission system asking authors to confirm that they have included negative controls, and if not, to explain why. This would force authors to think about whether their controls are adequate and would make them visible to reviewers.

Another fix is to mandate code and data sharing before review. If reviewers have access to the code, they can run their own checks, as Vasquez did. Some journals already require this, but many still allow authors to share code only upon request. Making code availability a condition of publication would increase transparency and allow for more thorough review. For instance, the Journal of Machine Learning Research has required code and data sharing since 2018, and the journal Science has mandated data availability statements since 2017.

Training reviewers to check supplementary materials is also important. This could be done through short online tutorials or by including a checklist in the review form. The checklist might include questions like: "Did the authors report negative controls?" and "Did they describe the training procedure for those controls?" Such checklists are already used in some fields, such as economics, where reviewers are asked to verify that authors have disclosed all variables and robustness checks.

Creating badges for reporting null results is another idea. The Center for Open Science has proposed badges for open data, open materials, and preregistration, and these have been shown to increase compliance. A similar badge for "negative controls reported" could incentivize authors to include them in the main text. Some journals have already adopted such badges, but they are not yet widespread.

Finally, post-publication review platforms, such as PubPeer, can serve as a safety net. Vasquez's commentary was published in the same journal, but it could also have been posted on PubPeer, where other researchers could have seen it. Post-publication review is often faster than traditional peer review and can catch problems that slipped through the initial process. However, it relies on the willingness of researchers to invest time in reviewing others' work, which is not always rewarded.

The Verdict: A Call for Structural Reform

The original preprint was eventually withdrawn by the authors in February 2026, after the journal published an expression of concern. But the withdrawal came too late. The model had already been cited in dozens of papers, some of which were themselves being used to guide clinical decisions. The field lost credibility, and patients may have been put at risk if any clinical decisions were based on the model's predictions.

The episode is a reminder that the research pipeline is not just about the final published paper. It is about the entire process, from the initial analysis to the peer review to the replication attempts. Each step has its own incentives and blind spots, and each can fail. The negative controls were there, but they were buried. The reviewers did not ask about them, and the editors did not require them to be prominent. The result was a paper that overstated its claims.

There is no single fix. Mandating negative controls in the main text would help, but it would not guarantee that they are interpreted correctly. Training reviewers to check supplementary files would help, but it would not catch every issue. Post-publication review would help, but it depends on the vigilance of the community. The lesson is that individual vigilance is not enough. The system needs structural incentives that reward transparency and penalize hiding negative results.

Specifically, journals should adopt policies that require negative controls to be described in the main text of any paper that makes predictive claims. They should also require that code and data be made available at the time of submission, not just upon request. And they should provide reviewers with checklists that explicitly ask about negative controls, so that these are not overlooked. These are concrete steps that can be taken now, without waiting for a cultural shift. The preprint's trail, from its promising debut to its eventual withdrawal, is a case study in how the system can fail. It is also a call to action for journals, funders, and researchers to make negative results a visible part of the scientific record, not a footnote in the appendix.