MIT researchers use AI model to interpret and edit other neural networks

Researchers at the Massachusetts Institute of Technology (MIT) in Cambridge have used an artificial-intelligence model to read and tinker with the internal workings of another AI network, offering a new way to peek into the “black box” of AI. The findings were reported in a preprint on 2 November.

How the AI mechanic works

The team demonstrated how an AI model called GPT-6 Astra can interpret the raw internal values of some other models well enough to predict what different parts of the systems do. It can then edit the models to make improvements, including boosting their mathematical abilities.

Frontier AI models are “showing early sparks” of an ability to treat neural networks like files that they can read, reason about and change, says co-author Rikiya Takehi, a machine-learning researcher at MIT.

The results do not mean that Astra, developed by OpenAI in San Francisco, California, can explain or understand how a given AI network works. But the findings suggest that general-purpose AI models — such as ChatGPT or Claude — might be able to modify a network in useful ways, like a mechanic who can identify and fix issues in an engine that they neither designed nor fully understand.

Expert reactions

Fazl Barez, an AI-safety researcher at the University of Oxford, UK, says the preprint shows that it is possible to automate some “interpretability” methods. He cautions that the hope that better understanding models will help create safer AI systems is “largely untested.”

Barez says he would be surprised if Astra is able to read and interpret models in ways that go beyond existing human abilities, and it remains unclear if Astra is going beyond applying information and techniques learned in its training.

Marinka Zitnik, a computer scientist and bioinformatician at Harvard University in Cambridge, Massachusetts, says that if developed further, such ability to reprogram models might be an asset in developing AI for science, for example by helping to spot errors in a model ahead of real-world testing.

Why it matters

Being able to target and debug AI systems could improve artificial-intelligence safety and utility. The complex way in which a model’s features link to its outputs means neural networks are often described as black boxes, and interpretability researchers run experiments to work out, for example, which weights are activated under specific conditions.

---

Sources

• nature.com: This AI ‘mechanic’ can peer inside the black box of other neural networks | Nature

This article was drafted with AI assistance and checked against the sources above. Company claims are reported as claims. Cover image is AI-generated.