In this guest blog, researchers from University College London discuss their recent work exploring whether locally run AI models can help with one of the most time-consuming aspects of research: preparing data for analysis.

Artificial Intelligence (AI) tools such as Large Language Models (LLMs), the technology behind tools like ChatGPT, have become very good at writing computer code. Advanced AI systems such as Agentic AI can now do more than simply answer questions.

AI agents are small, specialised pieces of software that can make decisions independently or cooperatively to perform tasks.* They can write code, run it, check for mistakes and fix problems with limited human input.

These AI tools are increasingly being built into software used by researchers, such as VS Code and RStudio, two widely used applications for programming and statistical analysis. As a result, AI is becoming a valuable tool for analysing data, helping researchers work faster and more efficiently.

However, using AI with sensitive research data can be challenging. Many researchers work within Trusted Research Environments (TREs), highly secure digital environments that provide access to sensitive data for analysis. Because TREs are isolated from the internet, researchers often cannot use cloud-based AI services, which require information to be sent to computers operated by external providers.

Can local AI models provided securely within the TRE itself offer a solution? How well do they work? Our new preprint (a research paper shared publicly before it has completed formal peer review) explores these questions.

*AI agents definition adapted from: AI Insights: Agentic AI (GOV.UK)

What are open-weight models?

Open-weight models are AI models that can be freely downloaded and run on a local computer or server. This means organisations can use them without sending data to an external company.

Running AI models locally has several potential advantages:

  • It can help organisations meet data protection and governance requirements, keeping data safe and secure.
  • It avoids ongoing subscription or usage costs for cloud services.
  • It gives researchers more control over which AI systems they use, reducing reliance on a single technology provider and helping to make their work more replicable.

Like cloud-based AI tools, local models can be used through coding and data analysis software used by researchers – also known as Integrated Development Environments (IDEs) – or through standalone applications such as LM Studio and Ollama.

However, there is a practical limitation. The most powerful AI models, such as Kimi-K3, often require extremely powerful and expensive computing systems that most researchers do not have access to. As a result, many researchers are limited to smaller models that can run on ordinary desktop computers or laptops.

We believe our research is one of the first attempts to find out whether these more accessible AI models are capable of supporting real-world research.

Our paper

We tested several locally run, consumer-grade AI models on real data, focusing on one of the most time-consuming parts of research: data preparation.

Before researchers can analyse data, they often need to clean it, organise it and develop code to create new variables (groups or categories of values) from the raw data. This process can take many hours and often involves writing large amounts of computer code.

To evaluate AI performance, we compared it with work previously completed by experienced researchers using data from Next Steps, a long-running study led by the Centre for Longitudinal Studies. The study has followed the same individuals from age 14 to age 32.

The original task involved manually transforming raw survey data into a dataset that was ready for research. This required thousands of lines of code and the creation of 102 variables.

We created a benchmark, or a standardised test, called RRBench to measure how well AI systems could complete these tasks. We also made the benchmark and automated evaluation tools freely available for other researchers to use.

Our results

The best locally run AI models performed better than we expected.

The strongest models we tested successfully completed research data preparation tasks up to 87.9% of the time. This suggests they could provide a practical route to AI-assisted research, including in fields such as population health and social science.

However, results varied considerably between models. Smaller models were much more likely to make ‘silent’ errors. These are mistakes where the code appears correct and runs successfully, but contains errors; this produces incorrect results that could lead researchers to draw the wrong conclusions. This highlights the importance of choosing AI tools carefully and thoroughly checking their outputs.

At the task level, one of the strongest predictors of success was the amount of information the AI needed to process. For each task, the AI was given information describing the data and variables involved. Tasks with many variables or complicated descriptions were more likely to fail. This suggests that well-organised, searchable documentation could help AI systems perform better.

Performance might also improve by using multiple AI agents rather than a single system. For example, different agents could be given specialised roles, such as refining tasks, searching documentation, reviewing code, and checking results.

Further details can be found in our preprint.

What’s next?

We hope these findings encourage providers of TREs and other research infrastructure to offer local AI models within their secure systems. Our results suggest that these tools can be both powerful and relatively straightforward to deploy.

Data providers could also consider using local AI systems to help create “research-ready” datasets and the code needed to generate them. If implemented carefully, this could save researchers thousands of hours by reducing duplicated work across different projects. Both approaches could help maximise the scientific and policy impact of valuable research datasets.

For researchers working on personal computers, hardware remains a challenge. The highest-performing local AI models generally require large amounts of memory (RAM) and more powerful computers than many researchers currently have access to (for specific hardware requirements, see an online calculator). Although smaller models continue to improve, further investment in computing infrastructure is likely to be needed if researchers are to make full use of local AI tools.

Our benchmark, RRBench, remains available for testing future AI models as they are released. The framework is open source and can be adapted to different research areas, programming languages and tasks. In fact, any task that produces a structured output can potentially be evaluated using RRBench.

Researchers will continue to play an important role in assessing the accuracy, reliability and potential biases of AI systems. This evidence will help determine which tasks can safely be accelerated or delegated to AI, allowing researchers to spend more time on interpretation, insight and meaningful analyses.

Links

Authors

  • Mack Nixon, Yevgeniya Kovalchuk — Centre for Advanced Research Computing, University College London
  • Andy Boyd — UK Longitudinal Linkage Collaboration, University of Bristol
  • Liam Wright, Alison Fang-Wei Wu, Martin Danka, David Bann — Centre for Longitudinal Studies, University College London

GitHub repository

Editor’s note: This article was contributed by researchers from University College London. The opinions and findings presented are those of the authors and do not necessarily represent the views of DARE UK or its delivery partners.