Challenges and potential for reproducible open science in the AI era

This post covers three topics: AI and open science, reproducibility, and its impact on scientific practice. It assumes a basic level of AI knowledge and becomes more complex as the reading progresses. Feel free to start reading from whichever section interests you most.

🌐 Leer este posteo en español

Summary. Many researchers use generative AI to draft manuscripts, generate code, and summarize literature. If these models are used to produce results, we can consider them part of the analysis, and they should meet the same openness and transparency standards as open science. Starting from this premise, we walk through three features of open science that these models put under tension: how open a model really is, what risks emerge when publishing or sharing data, and whether documenting their use in detail could make the analysis reproducible. Finally, we introduce multi-agent research systems: the question is not whether AI will replace researchers, but what we are doing to ensure these new tools are used responsibly and safely in each discipline.

Cover image: Janet Turra & Cambridge Diversity Fund / Better Images of AI / License CC BY 4.0

Introduction

Since ChatGPT was released to the public in 2022, large language models (LLMs) have become part of everyday life. Let’s pause for a moment to relate the well-known chatbot to what we call “Artificial Intelligence” (AI). The models behind chatbots like Claude and ChatGPT are part of so-called generative AI: AI capable of creating content, whether text, images, video, audio, or code, from a user instruction or prompt [1].

The dynamic is simple: you provide a prompt (an instruction or question we write) and the model returns an output or response. These models are trained on enormous amounts of data, and that training fixes their weights: parameters that store what was learned and that, given a prompt, assign probabilities to the possible next words. In essence, the model predicts the next word [1].

How did researchers use LLMs?

As soon as this technology became available, researchers began using generative AI for, among other things, improving manuscript quality, generating ideas and organizing content, creating code, summarizing literature, and improving their peer reviews [2], [3]. Most scientific journals took action, adopting the stance of requiring the use of LLMs to be declared in the methodology (or acknowledgments) section of the scientific article, depending on the use [4], [5], [6].

Even so, in some cases problems related to these early uses were reported. Concerns included unequal access, plagiarism, lack of rigor in peer review, and the hallucination of bibliographic citations: invented references that look real [7], [2], [8], [9]. Although these are very different problems, at the mechanism level they all involve the use of LLMs.

If the LLM is part of how the results were obtained, then it is part of the analysis. And everything that is part of the analysis should meet the standards of open science and reproducibility.

Open science in transformation

Open science is defined as a cooperative approach within the scientific community that seeks to share knowledge, results, and tools as early and as widely as possible: “as open as possible, as closed as necessary”. Its goal is to promote science that is accessible, inclusive, and transparent [10].

A 100% reproducible study is not always possible: for example, data often cannot be shared for privacy reasons.

In this post we will focus on three fronts where transparency is necessary for reproducibility [11], [12]:

1. the computational tools (open source),
2. the analyzed data (open data), and
3. the analysis procedures (open workflows).

To be open, all material should have a permissive license and be publicly accessible.

Let’s see how each of these three points can be affected by introducing LLMs into the analysis:

1. Open source: a gradient of openness

If the LLM is part of how the results were obtained, should the code be released?

We can think of the openness of these models as a gradient (Fig. 1). The most popular LLMs, such as Claude, ChatGPT, and Gemini, are “proprietary”: they were designed for commercial use, have closed licenses, and no details are known about their architecture or their training data.

Figure 1. Types of LLM according to how much information and data is available about them

Open weight vs. open source

In the middle of the gradient are the “open weight” models. In general, these models release the weights, but not the training data. According to the Open Source Initiative (OSI) definition [13], this is not enough for a model to be considered open: at minimum, the description and location of the training data, the code, and the weights must be made freely available, along with an appropriate license (for a more extensive list, check [14]). An example of the few models that meet most of these standards is OLMo (Allen Institute for AI) [15].

Is it enough to use an open weight model?

Knowing what data was used to train a model is key to understanding how it behaves and where its answers come from. Lack of transparency in training data can perpetuate biases or errors that go unnoticed, preventing researchers from making informed decisions about its use. Some authors call this open-washing: releasing only the weights as a way to avoid the scientific scrutiny and legal exposure that could come with presenting their models as fully open [16], [17], [18].

So why not directly adopt an open source model?

• Does one exist for my goal? There may not be one, or it may perform worse on the benchmarks you care about. It may also happen that, after examining the training data in detail, you end up deciding against it.
• Infrastructure. Running them locally requires good hardware. If the model has many parameters, you may have to pay for infrastructure [19].
• There aren’t that many! The supply of open weight models far exceeds that of open source models.

2. Data transparency: privacy and anonymization

Norms on how to use LLMs ethically in research, protecting participants and data privacy, are one of the needs emerging from the use of these tools [23].

The anonymization problem: what data are you going to leave available?

Anonymizing data used to mean deleting explicit identifiers —name, address, phone number. But LLMs are capable of indirectly inferring identity from context: profession, location, dates, and minor details that, combined, are enough to recognize someone. In an experiment on anonymized texts about famous people, GPT managed to re-identify a considerable share of cases, nearly triple the rate of human participants [20]. The practical conclusion: anonymization should be evaluated considering the risk of AI-assisted re-identification, and not just the removal of explicit identifiers.

Access to models: In the current landscape, would leaving data available allow those who pay for subscriptions to frontier models to draw more associations from the data? [21]

The privacy problem: what data are you sharing through the API?

When you use an LLM to process sensitive information, accessing it through third-party services can add confidentiality, data residency, and regulatory compliance issues. The data leaves your institution and becomes subject to the provider’s policies and jurisdiction. One alternative is to deploy open source or open weight models on your own infrastructure or in private environments, which allows you to keep control over the data and apply the organization’s security policies [22].

3. Transparency in the workflow: reproducibility

Deterministic vs. Probabilistic

When we carry out quantitative data analysis, reproducibility assumes that, with the same data and methods, we should obtain the same results. In other words, we are talking about a deterministic process. The use of LLMs, instead, tends to be probabilistic: parameters like the temperature make the model pick tokens with some randomness, so it doesn’t always return exactly the same thing. Even so, there are elements we can document to get closer to deterministic behavior:

What can be documented about an LLM

1. Model. The identity of the model (name, version), ideally with a Model Card [24]. It’s best to use the API rather than the web interface, which may apply hidden instructions, silently change versions, and doesn’t let you fix all the parameters. Ideally, a model released for open use would be better, to avoid it becoming unavailable [25].

2. Prompts and context. The exact text of the system prompt (or a statement that none was used) and of the user prompts. If there were multiple turns, the full history, since each exchange influences the next.

3. Model configuration:

• Parameters that alter the randomness of the output, for example, the temperature. The lower the temperature, the lower the probability that less likely tokens¹ are chosen. To better understand which parameters influence the randomness of the output, we recommend reading Wagner et al. [25].
• Parameters that modify the output, for example, the maximum number of tokens in the response.

And is that enough to make it reproducible? Not entirely: even at temperature 0, GPU inference can vary [26], [27].

If an LLM is a stochastic model, does it make sense to choose it when what I’m looking for is reproducibility? For exact computation (like doing calculations), a deterministic method would be preferable.

But I think we need to reframe the situation. Let’s take a step back. Imagine with me two hypothetical scenarios, one where LLMs are only used to create the code and one with a greater degree of autonomy, where the entire analysis is created directly using an LLM (Fig. 2):

Figure 2. How integrated the use of LLMs is in your workflow

Scenario 1 — We create the code we need with LLMs. Being responsible researchers who know the subject, we will look critically at the generated code, improve it, and be able to use it for our results just as was done before LLMs became widespread. We very likely won’t know what code the model was trained on, so it can be difficult to trace what is leading us to solve the problem one way or another and to cite the software’s authors. But even so, if the code is made available and well documented, we would be in a scenario with a high percentage of reproducibility.

Scenario 2 — The LLM solves the analysis. This scenario is a bit more radical and not very realistic in most situations. Can we directly ask the LLM to carry out an entire analysis from start to finish and trust the result?

Why would anyone be interested in obtaining reproducible results using this technology?

To try to answer this, there’s a leap we need to take: introducing agentic AI. Its use implies not only the possibility of automating a complete workflow, but delegating to the AI part of the decision-making about an analysis. While this may sound attractive, since it would offer the possibility of speeding up and boosting scientific production, it’s not that simple.

From querying an LLM to using agents

An agent is a system that observes its environment and acts on it to achieve goals [28]. Agentic AI completes predefined objectives with limited supervision over a cycle of perception, planning, action, and reflection [29], [30]. It does not replace the LLM: it coordinates it. The LLM keeps doing the work at each step, while the agentic scaffolding provides autonomy and orchestration.

An example of interest is the recently published AI Scientist, an agentic system that attempts to recreate scientific research from scratch, moving through hypothesis, experimentation, writing, and review automatically [31]. Using this methodology, they achieved the acceptance of one of the three papers generated entirely by AI at a conference workshop (not a journal).

The AI Scientist is no longer alone: in 2025–2026, a whole ecosystem of multi-agent systems for research appeared, such as Google DeepMind’s Co-Scientist [32] and FutureHouse’s Robin [33].

The issue of computational reproducibility for agentic AI is under active development [34], [35], [36].

Will AI replace researchers?

I can’t read the future, but I can offer an observation based on the information shared so far: the adoption of autonomous AI systems could boost lines of research previously unfeasible due to time or resources. On the other hand, it would open up discussions about authorship, ethics, and peer review, and would remain within reach only of those who can afford its costs.

Perhaps the question we should ask ourselves is not whether AI will replace researchers

It is likely that in the near future we will see more agents assisting in parts of the workflow. Platforms designed for AI output have already emerged, such as aiXiv [37] or the Agents4Science conference [38]. While this seems like progress, we must not lose sight of the fact that scientists’ responsibility is to maintain a certain rigor in the standards of scientific production. Defining AI-human collaboration in this context is vital [39].

However, not everyone agrees that pushing toward increasingly autonomous agents is desirable. Bengio and colleagues warn that the race to build generalist agents carries serious safety risks: misaligned systems that deceive, seek to preserve themselves, or escape human control. As an alternative, they propose a non-agentic “Scientist AI,” designed to explain the world rather than act on it. Ultimately, an AI meant to assist researchers, not replace them [28].

Reading these articles, I think that, between the lines, the discussion isn’t whether AI will replace humans, but how the scientific system will adapt to these tools. The question “Will AI replace researchers?” assumes AI’s autonomy, as if one day we would magically discover that we can delegate everything to it and let it carry out our tasks. If someday we can trust an autonomous AI at that level, it will be because we have made multiple decisions about its development, decisions that inevitably have a strong social and ethical component. Perhaps the question we should ask ourselves is: “What am I doing to ensure a responsible and safe use of AI in my discipline?”

Acknowledgments

I thank Roxana Villafane for reading this text and for her comments on it. I also thank Patricia Loto and the RSE Argentina team for the invitation to the talk that was the origin of this blog post.

About the use of AI in this post

In keeping with what is discussed above, it’s worth clarifying the role AI played in this text. The ideas, the argumentative structure, the selection of sources, and the conclusions are the author’s own. Language models (Sonnet 5 and Fable, by Anthropic) were used as editing assistants: correcting writing and spelling, verifying that citations existed and giving them a consistent format, compiling the reference list, and condensing some sections. They did not generate the arguments, and all their suggestions were reviewed by the author before being incorporated. In addition, a first translation of this English version from the original Spanish text was produced using Claude (Anthropic), and then revised by the author.

How to cite this text

If you want to reuse or reference this post, you can cite it as follows:

Short format (approx. APA): D’Andrea, F. (2026). Challenges and potential for reproducible open science in the AI era. https://florencia-dandrea.netlify.app/posts/2026-07-en.html (Archived version: https://doi.org/10.5281/zenodo.21347392)

BibTeX:

@misc{flor2026en,
  author       = {D'Andrea, Florencia},
  title        = {Challenges and potential for reproducible open science in the AI era},
  year         = {2026},
  howpublished = {Blog post},
  url          = {https://florencia-dandrea.netlify.app/posts/2026-07-en.html},
  note         = {Archived version on Zenodo: https://doi.org/10.5281/zenodo.21347392}
}

References

1. MIT News (2023). Explained: Generative AI. https://news.mit.edu/2023/explained-generative-ai-1109
2. Almarie, B. et al. (2023). Editorial – The Use of Large Language Models in Science: Opportunities and Challenges. Principles and Practice of Clinical Research, 9(1), 1–4. https://pmc.ncbi.nlm.nih.gov/articles/PMC10485814/
3. de la Torre-López, J., Ramírez, A. & Romero, J. R. (2023). Artificial intelligence to automate the systematic review of scientific literature. Computing, 105, 2171–2194. https://link.springer.com/article/10.1007/s00607-023-01181-x
4. Science. Science Journals: Editorial Policies. https://www.science.org/content/page/science-journals-editorial-policies
5. Science. Science Journals: Guidelines for the Use of Generative AI. https://www.science.org/cms/asset/86716fc6-8edc-46a4-ab20-2d0e2d4700c5/science_journals_guidelines_for_ai_use.pdf
6. Taylor & Francis. AI Policy. https://taylorandfrancis.com/our-policies/ai-policy/
7. Gao, J. & Wang, D. (2024). Quantifying the use and potential benefits of artificial intelligence in scientific research. Nature Human Behaviour, 8(12), 2281–2292. https://www.nature.com/articles/s41562-024-02020-5
8. Nature (2023). Tools such as ChatGPT threaten transparent science; here are our ground rules for their use [Editorial]. Nature, 613, 612. https://www.nature.com/articles/d41586-023-00191-1
9. Naddaf, M. & Quill, E. (2026). Hallucinated citations are polluting the scientific literature. What can be done? Nature, 652, 26–29. https://www.nature.com/articles/d41586-026-00969-z
10. UNESCO. Open Science. https://www.unesco.org/en/open-science/about
11. The Turing Way Community. Open Research. https://book.the-turing-way.org/reproducible-research/open/
12. Open Knowledge Foundation. The Open Definition. https://opendefinition.org/
13. Open Source Initiative. The Open Source AI Definition. https://opensource.org/ai/open-source-ai-definition
14. Open Source Initiative. Checklist to Evaluate Machine Learning Systems. https://opensource.org/ai/checklist
15. Ai2 – Allen Institute for AI (2024). Hello OLMo: A truly open LLM. https://allenai.org/blog/hello-olmo-a-truly-open-llm-43f7e7359222
16. Open Source Initiative. Open Weights: Not Quite What You’ve Been Told. https://opensource.org/ai/open-weights
17. Comprehensive Analysis of Transparency and Accessibility of ChatGPT, DeepSeek, and other SoTA Large Language Models (2025). arXiv:2502.18505. https://arxiv.org/abs/2502.18505
18. Liesenfeld, A. & Dingemanse, M. (2024). Rethinking open source generative AI: open-washing and the EU AI Act. FAccT ’24: Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency, 1774–1787. https://doi.org/10.1145/3630106.3659005
19. Local AI Master. AI Hardware Requirements 2026: Beginner-Friendly Guide. https://localaimaster.com/tutorials/getting-started/hardware-requirements
20. Patsakis, C. & Lykousas, N. (2023). Man vs the machine in the struggle for effective text anonymisation in the age of large language models. Scientific Reports, 13, 16026. https://www.nature.com/articles/s41598-023-42977-3
21. Staab, R., Vero, M., Balunović, M. & Vechev, M. (2024). Beyond Memorization: Violating Privacy via Inference with Large Language Models. The Twelfth International Conference on Learning Representations (ICLR 2024). arXiv:2310.07298. https://arxiv.org/abs/2310.07298
22. Dorca Josa, A. & Bleda-Bejar, M. (2024). Local LLMs: Safeguarding Data Privacy in the Age of Generative AI. A Case Study at the University of Andorra. ICERI2024 Proceedings, 7879–7888. https://doi.org/10.21125/iceri.2024.1931
23. Schroeder, H., Aubin Le Quéré, M., Randazzo, C., Mimno, D. & Schoenebeck, S. (2025). Large Language Models in Qualitative Research: Uses, Tensions, and Intentions. CHI ’25: Proceedings of the 2025 CHI Conference on Human Factors in Computing Systems. https://doi.org/10.1145/3706598.3713120
24. Mitchell, M. et al. (2019). Model Cards for Model Reporting. arXiv:1810.03993. https://arxiv.org/abs/1810.03993
25. Wagner, S., Muñoz Barón, M., Falessi, D. & Baltes, S. (2024). Towards Evaluation Guidelines for Empirical Studies involving LLMs. arXiv:2411.07668. https://arxiv.org/abs/2411.07668
26. Gond, R., Kamath, A. K., Ramjee, R. & Panwar, A. (2026). LLM-42: Enabling Determinism in LLM Inference with Verified Speculation. arXiv:2601.17768. https://arxiv.org/abs/2601.17768
27. Thinking Machines Lab. Defeating Nondeterminism in LLM Inference. https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/
28. Bengio, Y. et al. (2025). Superintelligent Agents Pose Catastrophic Risks: Can Scientist AI Offer a Safer Path? arXiv:2502.15657. https://arxiv.org/abs/2502.15657
29. IBM. What is agentic AI? https://www.ibm.com/think/topics/agentic-ai
30. Google Cloud. What is agentic AI? https://cloud.google.com/discover/what-is-agentic-ai
31. Lu, C., Lu, C., Lange, R. T., Yamada, Y., Hu, S., Foerster, J., Ha, D. & Clune, J. (2026). Towards end-to-end automation of AI research. Nature, 651(8107), 914–919. https://www.nature.com/articles/s41586-026-10265-5
32. Gottweis, J. et al. (2026). Accelerating scientific discovery with Co-Scientist. Nature. https://www.nature.com/articles/s41586-026-10644-y
33. Ghareeb, A. E. et al. (2026). A multi-agent system for automating scientific discovery (Robin). Nature. https://www.nature.com/articles/s41586-026-10652-y
34. Starace, G. et al. (2025). PaperBench: Evaluating AI’s Ability to Replicate AI Research. arXiv:2504.01848. https://arxiv.org/abs/2504.01848
35. Reproducible, Explainable, and Effective Evaluations of Agentic AI for Software Engineering (2026). arXiv:2604.01437. https://arxiv.org/abs/2604.01437
36. Siegel, Z. S., Kapoor, S., Nadgir, N., Stroebl, B. & Narayanan, A. (2024). CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmark. arXiv:2409.11363. https://arxiv.org/abs/2409.11363
37. aiXiv: A Next-Generation Open Access Ecosystem for Scientific Discovery Generated by AI Scientists (2025). arXiv:2508.15126. https://arxiv.org/abs/2508.15126
38. Bianchi, F., Queen, O., Thakkar, N., Sun, E. & Zou, J. (2025). Exploring the use of AI authors and reviewers at Agents4Science. arXiv:2511.15534. https://arxiv.org/abs/2511.15534
39. Noble, A. (2026). Thinking & Doing Science in the Age of AI. Daedalus, 155(1-2), 306–315. https://doi.org/10.1162/DAED.a.989

Footnotes

1. The smallest units of text, words or word fragments, that an LLM processes or generates at each step.