Research talk · AI agents

AutodidactAgent

An AI agent that learns a company's expert methods from its own reports and raw data files.

The problem

A simple question about a complex file

An expert needs days to answer it by hand. A general LLM knows the physics but not the team's method, because reports keep only the final numbers.

A safety engineer asks

“What was the maximum chest acceleration of the dummy in this crash test?”

ZIP 4936.zip Chrysler Town & Country (2005) 196 sensor channels
Raw file · answer not known yet
By hand
2–3 days
  • find the right channel among about 200
  • clean the signal, screen the outliers
  • filter it to the standard, take the 3 ms peak
Explorer new

Start from what the company already has

Old reports with final numbers in them, and the raw test files behind those numbers.

Explorer new

The Explorer turns them into an exercise

It reads a number in the report, writes the question that number answers, and attaches the raw file that holds the data.

Explorer new

One exercise, up close

A question, its raw file and the answer from the report. The method that gets from file to answer is what the agent must learn.

Explorer new

Hundreds of pages become an exercise bank

The same happens for every page and file. Nobody writes or labels a single exercise.

Memory

The bank becomes the agent's memory

Each circle is an exercise: its answer is known, the method to reach it is not.

Retriever

Similar exercises get linked

The Retriever compares exercises field by field (question, file, measured quantity) and links each one to its closest matches. A thicker link means more similar.

Practice

The engineer's question arrives

Its answer is unknown, so the agent practises on linked exercises, where the answer is known and every attempt can be checked.

Practice

The first practice attempt fails

The LLM agent writes and runs code on the exercise's raw file. Its result is more than 10 % off the known answer, so the attempt failed.

Practice

A linked exercise is solved instead

A neighbour two links from the question is tried from scratch and solved. Its working method is saved as a skill: a short written procedure.

Practice

The skill unlocks the harder exercise

Retried with that skill as a starting point, the exercise that failed is now solved too.

Answering

The question gets its answer

Practice stops. The agent adapts the new skill to the question's own file and answers it.

Answering

A similar question reuses the skill

This question already has a solved neighbour, so no practice is needed: the stored skill is adapted directly.

Answering

Nothing similar? It still answers

With no close solved exercise, the agent works the question out from scratch, like a plain LLM agent. The base model is never retrained, so it keeps all its general ability.

AIMBE trainer new

Five new questions, three attempts left

Some exercises are solved, some failed before, most were never tried. Which one is worth practising first?

AIMBE trainer new

Every nearby exercise gets a value

Value = chance the attempt succeeds × how strongly it links to questions not yet covered. A solved neighbour raises the chance.

AIMBE trainer new

Practise the best one, then update the values

Solved with its neighbour's skill, it covers two more questions. Exercises that only help those questions lose value; its failed neighbour gains value, because it can now reuse this skill.

AIMBE trainer new

Attempts can fail

The next-best exercise is tried from scratch and fails. The attempt is spent anyway, and the other values stay the same.

AIMBE trainer new

A failed exercise gets a second chance

The exercise that failed before now has a solved neighbour. Retried with that skill, it is solved. The budget is used up.

Answering

Then all five questions are answered

For each question, the agent adapts the skill of its best solved neighbour. The budget went only to exercises linked to these questions.

Provenalways picking the highest value reaches at least 63 % (1 − 1/e) of the best possible plan, on AIMBE's planning score

Summary

What is new, and what a company gets

What I built

  • Explorernew: builds its own practice set from reports and raw files. No manual labelling.
  • AIMBE trainernew: spends a fixed practice budget where today's questions need it, with a proven guarantee on its planning score.
  • AutodidactAgent: one agent that turns raw company data into reusable, written skills. It sits on top of the LLM (GPT-5 in the paper) without changing it.
  • Crash-QA: a new engineering benchmark built from public NHTSA crash-test data, to be released with the paper.

What a company gets

  • Uses the archive it already has: no annotation project before the first answer.
  • Learns the team's method, not just its documents: every answer comes with a readable procedure.
  • Spends practice where it counts: the budget goes only to exercises linked to current questions.
  • No fine-tuning: a newer LLM plugs in without retraining, nothing is forgotten, and the memory can be corrected by hand.
  • Not tied to crash tests: also applied to cardiology questions on ECG data.
Production

Quality, latency, cost and guardrails

Quality

  • An exercise counts as solved only within the tolerance of its known answer (10 % here, set by the user); only solved exercises lend their skill.
  • Every answer comes with its procedure, so a reviewer checks the method, not just the number.

Latency

  • Finding similar exercises and choosing what to practise need no LLM call.
  • The slow step is the agent run itself (about 15 min per GPT-5 call on these crash files); a reused skill gives it a method to start from.

Cost

  • Practice is capped by a budget of LLM calls; in the paper, at most 3 calls per question on average, practice included.
  • No fine-tuning runs: the memory is plain text and the LLM is used as it is.

Guardrails

  • Every exercise is tied to its raw file and to the known answer from the company's own report.
  • Wrong skills can be fixed or deleted by hand; practice stays within the budget and the two-link radius.

Thank you. Questions?