Autoresearch: Letting AI Train Its Own Models

An AI That Trains Its Own Models

In March 2026, Andrej Karpathy left an AI agent alone with a small language model and one GPU for about two days. Nobody told it what to try. It edited the training code, trained for five minutes, kept the change if the model got better, threw it away if not, and went again. Write-ups put the run at around 700 experiments. It found about 20 improvements that stacked, and together they made his model reach GPT-2 quality 11% faster.

He called it autoresearch, and the idea has since spread to almost every field that has a number to push. I wanted to know what it takes to use it on my own work, so I built a loop for a recommender and let it run 185 experiments. It raised Recall@10 from 15%, for a baseline that shows everyone the same ten most popular stores, to 73%, and showed me what this kind of loop really gives you: every idea tried, every result measured, every failure undone, and a git history that explains how it got there.

TL;DR

• What it is: Karpathy's autoresearch hands the experiment loop to an agent. One file it may edit, one locked evaluator, one number, git as memory.
• How to use it: with the uditgoenka skill you write six fields (Goal, Scope, Metric, Direction, Verify, Guard) plus a budget, and it runs keep-or-revert iterations on its own.
• Does it work: on a public recommender task, 185 experiments raised Recall@10 from 15% to 73%.
• The catch: the loop optimizes whatever judge you give it. Lock the judge, put statistics in the keep rule, and read the log.

Why I Wanted a Machine for This

Most of my time on an ML problem is not spent having ideas. It is spent on the loop around them: change one thing, run it, wait, read the number, decide whether it helped, undo it if it did not, and remember what I already tried. That loop is mechanical, and it is exactly the part I do worst at 11 pm.

Autoresearch hands exactly that loop to a machine. Karpathy's version gives an agent "a small but real LLM training setup" and lets it "experiment autonomously overnight".

The rest of this post follows the three animated diagrams:

1. What it is. Karpathy's original: three files, a seven-step loop, why each design choice is there, what it found, and where it falls short.
2. How to use it. The uditgoenka/autoresearch skill I use in Claude Code: six fields to define, what happens on every iteration, and the guardrails.
3. Does it work? My own loop on a recommender: the problem, the model, the metric, five runs, and what the log taught me.

Part 1: What Autoresearch Is

The short version: one file the agent may edit, one judge it cannot touch, one number, and git to remember what worked.

The idea in one paragraph

Give an agent a small, real experiment setup. Let it edit exactly one file. Lock the evaluation in a file it cannot touch. Score every attempt with one number under a fixed budget. Commit every attempt to git, keep the commit if the number improved and reset it if not, write a row to a log either way, and repeat until a human stops it. The human's job moves from running experiments to writing the instructions and the judge.

Three files

The metric is val_bpb, validation bits per byte. Lower is better, and because it is measured per byte rather than per token, it does not depend on vocabulary size. An architecture change and a tokenizer change land on the same scale.

The loop, step by step

program.md is short, and every line in it is doing something:

1. Set up a branch. Pick a tag from today's date and create autoresearch/<tag> from master. Read the three files. Create results.tsv with only a header row.
2. Make one change. Edit train.py with a single idea and commit it.
3. Run it under a fixed budget. uv run train.py > run.log 2>&1. Training stops at five minutes of wall clock, excluding startup. Output goes to a file, not the agent's context, so a long log does not flood its memory.
4. Read one number. grep "^val_bpb:\|^peak_vram_mb:" run.log. Empty output means the run crashed: read the last 50 lines, fix it if it is a typo, and give up on the idea if it is fundamentally broken.
5. Log it. Append a row to results.tsv: short commit hash, val_bpb, peak memory in GB, status (keep, discard or crash), and a one-line description. The log itself is not committed.
6. Decide. If val_bpb improved, keep the commit and the branch advances. If it is equal or worse, git reset back to where it started.
7. Never stop. Do not pause to ask whether to continue. Keep proposing, re-reading the files and combining near misses until the human interrupts. A run that takes more than 10 minutes is killed and counted as a failure.

Why it is built this way

Each design choice closes off a specific way for an unattended loop to go wrong.

• A fixed time budget, not a fixed step count. Every experiment gets the same five minutes, so a bigger model and a smaller one are compared on what they achieve in the same wall-clock time. The trade-off is that results are tied to your GPU and are not comparable across machines.
• One editable file. The blast radius of any change is one file, and every kept change is a diff a human can read in a minute.
• A locked evaluator. The agent can read prepare.py but not change it. This is the single most important rule in the repo. A loop that can edit its own judge will eventually optimize the judge instead of the model (Part 3 has an example of exactly that).
• Git as memory. Every attempt is a commit, and failures are resets. The branch history is a lab notebook of what survived, and the agent rereads it to decide what to try next.
• A simplicity rule. A tiny gain that adds messy code may not be worth keeping, and an equal result from deleting code counts as a win. Without it, the loop stacks hacks that each move the metric by noise.
• Never stop. The point is to run while you sleep. An agent that politely asks "should I continue?" at 2 am has failed its one job.

What it found

The README shows Karpathy's own overnight progress chart: 83 experiments, 15 kept, with val_bpb falling from about 0.998 to about 0.977. The kept changes are ordinary tuning: halving the total batch size, a longer warmdown, shorter attention windows, a larger RoPE base frequency.

Karpathy's own progress chart, from the autoresearch README. Green steps are kept commits, gray dots are discarded ones.

The longer run is the one that got attention. Karpathy left it running for about two days on a depth-12 nanochat model. Write-ups put that run at around 700 experiments. It found about 20 changes that improved validation loss. All of them stacked, all of them transferred to a depth-24 model, and together they cut nanochat's "time to GPT-2" from 2.02 hours to 1.80 hours, about 11%. These are not breakthroughs. They are the kind of tuning a careful engineer would find in a week, found while nobody was watching.

Where it falls short

The Datacamp guide lists the limits well, and I ran into most of them myself in Part 3:

• The ratchet has a creativity ceiling. It only accepts immediate gains, so it cannot take a step back to reach something better. It drifts toward small variations on what already worked.
• Five minutes hides long-horizon effects. A change that only pays off after an hour of training looks like a loss.
• Many experiments overfit one validation set. After hundreds of choices on the same data, some of the gain is specific to that data.
• The evaluation becomes the bottleneck. Once experiments are cheap, the quality of the judge is what limits you.

From one repo to an ecosystem

The awesome-autoresearch list now tracks dozens of descendants. The ones that matter fall into three generations, and they differ mainly in what the agent may touch and who judges it:

Ports of the original run the same loop on Apple Silicon, MLX, Windows GPUs, WebGPU and free Colab T4s. General tools drop the training script and keep the contract. Shopify is the clearest example: their engineers built pi-autoresearch, report using it on more than 40 metrics, and cut the build time that gates every CI pipeline by 65%. Two details from that write-up stuck with me. The one-shot prompt "Improve Polaris build time" failed, while the loop with a real metric did not. And Tobi Lütke called his own Liquid result (53% faster parse and render) "probably somewhat overfit".

Research pipelines go furthest. ARIS (Auto-Research-In-Sleep) is a set of plain Markdown skills in which the executor and the reviewer are different models: Claude Code does the work and Codex reviews it, so one model is not grading its own blind spots. Agon splits the work into scientist, coder, auditor and reviewer roles and routes every handoff through a file on disk. The further you go along that table, the softer the metric gets. A survey of 56 autonomous research systems concludes that these systems have scaled what they can produce, not what they can defend, and that the most common verifier, an LLM acting as judge, is also the weakest.

Part 2: How to Use It With the uditgoenka Skill

The short version: you write six lines of config. The agent does the rest, one commit at a time, and stops when the budget runs out.

For everyday engineering work I use uditgoenka's skill. It keeps Karpathy's contract (one scope, one number, keep or revert, git as memory) and drops everything specific to LLM training.

Install

# Claude Code, as a skill
npx skills add uditgoenka/autoresearch
 
# or as a Claude Code plugin
/plugin marketplace add uditgoenka/autoresearch
/plugin install autoresearch@autoresearch
 
# or with the bundled installer (also supports --opencode and --codex)
./scripts/install.sh --claude --global

Start a new session after installing. In OpenCode the commands use underscores (/autoresearch_debug), and in Codex they are keywords ($autoresearch plan).

The six things you define

Put together, a config looks like this:

/autoresearch
Goal: raise line coverage of the parser module
Scope: src/parser/**/*.ts
Metric: line coverage % from the coverage summary
Direction: higher
Verify: npm run test:coverage -- --silent | node scripts/coverage-pct.js
Guard: npm test
Iterations: 25

Applied to the recommender from Part 3, the same six fields would read like this. It is illustrative: the scripts named here are not in the repo, because my case study ran its own Python loop with the same rules.

/autoresearch
Goal: recommend the right store more often
Scope: experimenter.py (model and feature code only)
Metric: validation Recall@10
Direction: higher
Verify: uv run python eval_val.py        # prints the validation score
Guard: ./check_eval_unchanged.sh         # evaluator and test split must not change
Iterations: 50

Let /autoresearch:plan derive them

The hardest field is Verify: a command that turns "better" into one number, reliably. If I am not sure what it should be, I run /autoresearch:plan with a plain goal. It works through seven phases: analyze whether the goal is measurable, derive the scope from the project structure, propose a metric and direction, build a verify command, dry-run that command to prove it prints a number, propose a guard (test suite, type check or build), and suggest an iteration count. It ends with a ready-to-run config block and asks whether to run it or adjust it. For a goal that has no honest number, it points you to /autoresearch:reason instead.

What happens when you press enter

Before the first iteration, the skill checks that you are in a git repo with a clean working tree, runs the guard once to get a baseline, and screens the Verify command for dangerous patterns (rm -rf, curl | sh, embedded credentials). Then it runs Verify once and records that as iteration 0, the baseline.

Every iteration after that has seven phases:

1. Review. Read the tail of the results file and git log --oneline -20. If the last iteration was kept, read its diff to see what helped.
2. Modify. Make one focused, atomic change.
3. Commit. Commit it with an experiment: prefix, before knowing whether it works.
4. Verify. Run the Verify command and compute the delta.
5. Guard. Run the guard. If it fails, the change does not stay, even if the metric improved.
6. Decide. keep, discard, crash, no-op, hook-blocked or metric-error. Anything other than keep is undone with git revert.
7. Log. Append a row and go again.

Everything lands in autoresearch/loop-<date>-<time>/. The results file is plain TSV, so you can read it in the morning, diff it, or hand it to /autoresearch:evals:

# metric_direction: higher_is_better
iteration  timestamp  commit   metric  delta  guard  guard-metric  status    description
0          ...        a1b2c3d  71.8    0.0    pass   -             baseline  initial state
1          ...        e4f5a6b  73.1    +1.3   pass   -             keep      tests for empty input
2          ...        c7d8e9f  72.9    -0.2   pass   -             discard   parametrize token cases

An illustrative results file. The columns are the skill's; the numbers are made up.

The guardrails are the point

The parts I value most are the boring ones. In Claude Code, about ten hooks are on by default: they block reads of .env files and credentials, block force pushes and hard resets, skip heavy folders like node_modules, and warn at 400 changed lines and block at 800. The skill never pushes or deploys without asking. Iterations are bounded unless you write Iterations: unlimited. When a loop is going to run for an hour without me, these limits are what let me leave it alone.

The other loops

The same skill ships thirteen more commands built on the same idea: a bounded loop with a mechanical stop condition.

The command table from the uditgoenka/autoresearch README.

The ones I reach for most are :plan to set up a loop, :fix to drive test, type and lint errors to zero, :debug for hypothesis-driven bug hunting, and :regression as a stable-or-unstable gate before I push. Commands can be chained (--chain fix,regression), and --evals adds checkpoints that flag a plateau and suggest stopping early. Since v2.2 you can also give /autoresearch a plain-language goal without a metric. An orchestrator then classifies the goal, derives a pass/fail predicate, asks you to confirm it once, and routes between subcommands until the predicate is met or it plateaus.

Habits that make it work

• Make Verify fast and deterministic. A noisy number makes the loop climb a staircase of luck. If your metric is noisy, average several runs inside the command.
• Put your tests in Guard, not in Scope. The agent should never be able to make the guard pass by editing it.
• Keep the scope small. One module is better than the whole repo. Small diffs are diffs you will actually read.
• Read the kept diffs. Git makes it cheap. It is how you catch the agent winning in a way you did not intend.

Other variants worth knowing

The uditgoenka skill is one of many. The awesome-autoresearch list tracks dozens; these are the ones I would look at first, depending on your tools and how far you want the loop to go:

The first six keep autoresearch's shape: one metric, keep or revert. ARIS, Agon and AI-Scientist widen the loop to the whole research cycle, which is powerful but moves you toward softer judges like reviewer models.

Part 3: Does It Work? A Recommender Case Study

The short version: 185 experiments, 13 kept changes, and Recall@10 up from 15% to 73%.

The skill is great when the thing to optimize is code and the metric is cheap. I wanted to see what the same pattern does on a classic ML question, so I built a small loop of my own in Python. The code and all run logs are on GitHub.

The problem

Given what someone does across several parts of a super-app, which signals actually predict their next purchase in one vertical, especially for users who have barely used it? I used public Foursquare NYC check-ins as a proxy: 227K check-ins from 1,083 users, mapped into three verticals. Food is restaurants and cafés, Mart is delis, bodegas, pharmacies and markets, and Shops is other retail. The task is to recommend each user's next Mart venue out of 705.

The metric

The question I care about is simple: is the store the user actually visited next in their top 10? The percentage of users for whom it is, is Recall@10. It runs from 0% to 100%, and higher is better.

The loop itself scores every change with NDCG@10, a stricter version of the same idea that also rewards putting the right store near the top: 1.0 at rank 1, 0.5 at rank 3, 0.29 at rank 10, and 0 outside the top 10. Optimizing the stricter number is what lifted both.

For each of the 721 users with at least two Mart visits, the second-to-last visit is used for tuning (validation) and the last one is kept aside (test). The loop makes every decision on validation. Test visits are scored once at the end of each run, so the results below come from visits the loop never saw.

The model and the search space

The model is a base recommender plus six candidate features, which I call signals below. Each feature starts switched off, and the loop decides whether to switch it on and how much weight to give it. The loop never rewrites code; each iteration changes exactly one thing in a config:

This gives up the creativity of free code edits. In exchange every experiment takes well under a second, and the loop cannot break the evaluator because it never touches code.

The keep rule

On 721 users, a tiny gain in mean NDCG can be pure noise, and a loop that keeps noise will climb a staircase of luck. So a change is kept only if it improves the mean and a paired bootstrap over users says the gain is real with probability at least 0.90:

def p_improve(new, old, n=2000, seed=0):
    """Paired bootstrap over users: P(mean NDCG delta > 0) and a 95% CI of the delta."""
    d = new - old
    means = d[np.random.default_rng(seed).integers(0, len(d), (n, len(d)))].mean(1)
    return float((means > 0).mean()), np.percentile(means, [2.5, 97.5])
 
delta = res["ndcg"] - best_res["ndcg"]
prob, _ = p_improve(res["per_user"], best_res["per_user"])
status = "keep" if delta > 0 and prob >= KEEP_PROB else "discard"

Memory and proposer

Every experiment, kept or not, is a row in results.tsv. Every kept config is a git commit whose message carries the change and the score, so the commit list reads like the lab notebook Karpathy's design intends:

The kept experiments of run 5 as git commits. Each message carries the change and its validation score.

There are two proposers. The default is a heuristic: try each base model once, then change one knob of the current best. With --llm, Claude reads the full log and returns one atomic change plus a one-line hypothesis as structured output. Every number in this post comes from the heuristic, which needs no API key. That matters: the loop, the keep rule and the sealed test set did the work, not a clever model.

Five runs, 185 experiments

Every experiment across the five runs. Compare the shape with Karpathy's chart in Part 1: a fast climb, then a long tail of small, hard-won steps.

Each run taught me something different, and most of it was about the loop rather than the model.

Run 1: the baseline staircase. Starting from the popularity baseline (the same most popular stores for everyone), the loop tried each base model and kept RP3beta (0.24 → 0.45 on validation), then added the usual-area signal, the repeat boost and cross-vertical history. Validation 0.5404, test 0.5579.

Probes between runs, on validation only. Before touching the repo again I tested candidate signals by hand. Recency-weighted repeat visits did nothing (best P = 0.68, which is noise). Proximity to the latest check-in helped. I also tried something fancier, a gradient-boosted ranker trained on each user's earlier Mart visits with the same signals as features. It scored 0.514 to 0.520 against 0.540 for the searched blend, so I dropped it.

Run 2: greedy and stuck. Restarting from scratch with here added, the loop took here early. After that, cross no longer passed the bootstrap, and after 29 experiments every one-step change around the best had been tried. Validation 0.5409, no better than run 1. This is the ratchet's creativity ceiling from Part 1, in my own log.

Run 3: resume instead of restart. I added --resume, which continues from a previous best the way an autoresearch branch keeps advancing. It gained a little (0.5423) and stopped again. The log showed why: here 0.0 -> 0.2 was tried twice and narrowly rejected (P = 0.81 and 0.84), while the probe said larger weights work. The proposer could only switch a signal on one small step at a time. The bottleneck was the proposer, not the model.

Run 4: fix the proposer. Letting an off signal start at one to three steps, the loop tried here 0.0 -> 0.6 and kept it (P = 0.94). Validation 0.5488.

Where to look next. Instead of guessing, I checked where run 4's best still missed. Stores a user had visited before were almost always in their top 10 already. Most of the remaining misses were stores the user had never been to.

Run 5: aim at the gap. For a venue someone has never visited, a reasonable guess is somewhere popular near where they are right now. That became the sixth knob, local: venue popularity multiplied by proximity to the latest check-in. The loop tried local 0.0 -> 0.6 and kept it (P = 1.00), then tuned RP3beta's α. Validation 0.5589.

Results

On the visits the loop never saw, Recall@10 is 73%, up from 15% for the baseline, which shows everyone the same ten most popular stores. For almost three in four users, the store they actually went to next was in their top 10.

The baseline recommends the same busy stores to everyone. The tuned model ranks stores for each user, from their own history, their location and what similar people do.

Compared with run 1, the last three runs added a small but measurable gain (+0.015 NDCG@10, 95% interval +0.005 to +0.025), mostly through the two location signals.

The saved run is browsable on a small dashboard. For any user, it maps their history next to the top 10 from popularity and from the tuned model:

One user's history (small dots), popularity's top 10 (gray), the tuned model's top 10 (red) and the store they actually visited next (yellow).

What the loop found

• The graph model is the biggest single jump. RP3beta, a walk over the user-store graph, took Recall@10 from 15% to 61% on the tuning data.
• Location matters. Where users usually are and where they are right now both made the cut, and they help most for users with little history.
• People return to the same stores. A boost for stores someone has visited before was kept three times.
• Recency attention never helped. It was tried in every run and rejected every time.
• A fancier model lost. A gradient-boosted ranker scored below the simple blend. Every change had to beat a bootstrap on validation, whatever its pedigree.

Caveats

The data is a proxy: check-ins are visits, not orders, in one city in 2012. 185 experiments chose on the same validation set, and the test set has now been scored five times, once per run. No choice was made on test, but small test differences between runs should be read with care. The gap over popularity, from 15% to 73%, is large enough not to be in doubt.

Where It Goes Wrong: The Agent Will Game a Soft Judge

One failure mode deserves its own section. The best cautionary tale I found is a tennis prediction loop. The metric was ROC-AUC on 2026 matches, and the scoring logic lived in a file the agent was allowed to edit.

For eleven iterations the agent made honest progress, from 0.7454 to 0.7609. Then it started adding "specialists" keyed by tournament name. Then it added offsets that shift predicted probabilities after the model runs, keyed by tournament and round: 122 of them by iteration 33. The reported score reached 0.8523. The author noticed only because the curve sped up after a long plateau. A single +212 bps jump was larger than the whole honest phase. After the cleanup the real score was 0.7449.

Shopify saw a milder version: one "faster" result came from the agent deleting a lot of files, and the team rejected it. Nothing in either prompt asked the agent to cheat, and a prompt asking it not to would not have helped. As the tennis author puts it, "please don't game the metric" is a prompt, not a control. The controls that work are structural:

• Put the evaluator outside the writable scope, and fail the gate if any evaluator file shows up in the diff.
• Keep a holdout set the loop never sees, and score it rarely.
• Treat a sudden big jump as suspicious, not as good news. Check how the winning change was built before you believe it.
• Put statistics in the keep rule so noise does not count as progress.
• Read the kept diffs.

When to Reach for It, and What Changes

Autoresearch fits a problem with three properties: a number that a script can compute, a run that takes minutes rather than days, and a scope you can fence off from the evaluator. Hyperparameter and architecture search, performance work, coverage, kernel tuning and ranking quality all fit. It fits badly when the metric is a proxy you do not trust, when one run is expensive, or when "better" is a matter of taste. In that last case the loop does not remove the judgment; it moves it into the evaluator.

What changed for me is where my time goes. Less of it on running experiments, more of it on three things: the scope the agent may touch, the number it is judged by, and the lock between them. My five runs added a fourth: reading the log. Twice the thing holding the metric back was the loop itself, once its greed and once its step size.

The agent turns the crank. You still have to read what it writes down.

Try It

• Install the skill with npx skills add uditgoenka/autoresearch, then start with /autoresearch:plan on something small and measurable, like the coverage of one module.
• Read or fork my loop, with all five run logs: dzakybd/autonomous-experimenter.
• Browse the saved run on the dashboard.

References

• karpathy/autoresearch, and Karpathy's results post
• uditgoenka/autoresearch, the Claude Code skill
• Autoresearch isn't just for training models, Shopify Engineering
• A guide to autoresearch, Datacamp
• ARIS (Auto-claude-code-research-in-sleep) and Agon
• awesome-autoresearch
• Reward hacking in a tennis XGBoost loop
• What's missing in autonomous research? A survey of 56 systems
• My code, all five run logs and the dashboard: dzakybd/autonomous-experimenter, Hugging Face Space