jafgen --years 6 fails with No such option: --years. FirstRun, my agent that tests agent quickstarts, hit it at step 10 of the dbt DuckDB guide on 10 October 2026. The guide was broken on the day it merged.
I built FirstRun solo at neatHack, the neatlogs hackathon. It runs a quickstart in a fresh container, finds the first broken step, and proves the fix with a second clean run.
The docs bugs were the easy part. The harder bugs sat inside my agent, and its own reports hid them. Trusting an agent's report on itself is vibes, which means using a tool without checking its output. So FirstRun runs on one rule. A fix counts as verified only when a model-free rerun in a fresh container passes.
In short:
neatlogs raised an incident against my agent, at 52.8 times the baseline.
A planner fix cut invented steps from 2 of 4 runs to 0 of 52.
The verify step blocks every model call, so the agent cannot grade its own fix.
A bisect named jafgen 0.4.6 as the release that broke the dbt guide.
FirstRun fixed 16 docs pages, and contribution rules held back 3 PRs.
The 2:54 demo video covers the dbt break, the doc audit, the neatlogs incidents and the cost model.
FirstRun Reads, Plans, Runs, Fixes and Proves
Each run is one loop with five parts.
Read. Firecrawl pulls the docs page.
Plan. A Claude planner copies the shell commands into steps.
Run. Each step runs in a fresh
python:3.11-slimcontainer.Fix. Recovery classes a failure and patches the command, with 3 attempts at most per step.
Prove. A second container reruns the patched steps with no model calls.
Each run is one neatlogs trace, with a span for each part, so a bad run shows where it went wrong.
Version 1 Passed Steps That Proved Nothing
I ran v1 on 4 quickstarts with recovery off. Several of its passes were fake.
A placeholder passed as a step. Chroma step 2 was
true, a command that does nothing and exits 0.A fake secret passed too.
export OPENAI_API_KEY='sk-...'exits 0 and proves nothing.The planner skipped the steps that need an API key, which are the steps most likely to break.
The plan changed between runs. The same neatlogs page gave 4 steps, then 3 steps, 5 minutes later.
9 of 16 steps checked only the exit code, so an error that exits 0 would pass.
Each mistake made the agent's report look better than the run was.
An Incident Report Named My Agent as the Cause
neatlogs raised an incident on FirstRun itself. It counted 25 "Execution Failed" detections against a baseline of 0.5, across 10 traces.
The investigation named a root cause at 66% confidence, which neatlogs labels medium. A step ran a cd such as cd my-project, and the file check after it ran from inside the new folder. So the agent failed steps that worked.
Commit 2de852f is the fix, and it changed two things. A file check now looks from the folder where the step started. Every step also runs with set -e now, because dbt step 3 is five lines and its failed pip install passed on the exit code of the last line.
MkDocs went from a false fail at 14:58 IST to verified 9 of 10 at 15:40 IST.
Watch out: A last line that exits 0 says nothing about the four lines above it. Run every multi-line step with
set -e.
A Second Investigation Caught the Planner Inventing Steps
At 14:10 IST I made two neatlogs detections. A detection is a saved rule that flags matching spans. One of them, "planner invented a step", flagged the planner spans of 3 v1 runs.
neatlogs Investigate traced the invented steps to the planner prompt. It proposed a stricter prompt that copies only the commands the docs write.
A headless Claude Code session read the detection through the neatlogs MCP (Model Context Protocol) server, which lets an agent query neatlogs. It applied the fix in commit 3ca6f11. Invented steps fell from 2 of 4 v1 runs to 0 of 52 runs after the fix.
The Verify Step Blocks Every Model Call
The model that patched a step does not get to judge the patch.
verify.py applies the patches and starts a new container. It runs every step inside config.block_model_calls(), where any Claude or Jev call raises ModelCallBlocked. The run ends verified only when every runnable step passes.
A test guards the guard. tests/test_verify.py fails if verify.py imports the planner, the recovery code or the LLM module.
On the Entire install, recovery added curl and a PATH line. Then a fresh container ran all 4 steps with no model calls.
The rule has a limit. A neatlogs AI judge scored 52 runs on 11 October and agreed with the reported failed step in 46. Only 34 had every patch backed by a passing verify rerun. Runs that stopped before the clean rerun got no proof, and the judge marked them down.
The Agent Asks a Human Once, Then Remembers
The idea came from @bettersayAJ, who wrote "self-healing is self-defeating". So recovery classes each failure before it patches. It checks precedents.json first. A precedent is a stored error signature with its class, such as docs bug or environment. A match returns the class at probability 1.0, with no model call.
A new error goes to Jev, a model that returns a typed judgement with a probability. Below 0.7 confidence, FirstRun asks the human one question. The answer goes into precedents.json, so the same error never asks again.
Bisect Named the Release That Broke the DuckDB Guide
firstrun bisect finds when a step broke. It binary-searches a package's releases, one fresh container per probe, with the version pinned.
On the dbt guide, jafgen --years 6 passes on jafgen 0.3.1, released 2023-01-17. It fails on 0.4.6, released 2024-04-07. PyPI has no release between them. The guide merged on 2025-03-14, 11 months later, so it never worked in a clean environment.
The bisect took 5 probes and $0 in model calls, because it replays stored commands. The table went up as a comment on dbt PR #10149, and I posted the result on X.
4 PRs went up on day one, each from a FirstRun verified run.
dbt-labs/docs.getdbt.com#10149 fixes the removed
jafgen --yearsflag.dbt-labs/docs.getdbt.com#10150 states the Python 3.12 floor the guide needs.
entireio/cli#2724 names the Linux
PATHstep afterinstall.sh.accuknox/help#671 fixes a knoxctl page that named 0.9.0 while the installer fetched 0.9.58.
Version 2 Finished More Runs at a Lower Cost
The same 4 targets ran twice. v2 added the planner fix, recovery and verify.
Quickstarts that ended passed or verified went from 2 of 4 to 3 of 4.
Failed steps recovered went from 0 to 3.
Fixes proven in a fresh container went from 0 to 1.
Cost per run, from neatlogs, went from $0.0262 to $0.0200.
Time per run went from 128 s to 201 s.
v2 runs 73 seconds longer, because a verified fix runs the whole quickstart twice.
The sample is thin. Each target ran once per version, and several fixes landed together, so they share the credit.
The Business Plan Set the Agent's Spending Cap
Before the plan existed, the cap was a guess of $0.50 per run. Then firstrun econ read 88 run traces from neatlogs. The old cap sat at 30 times the mean run cost of $0.0178.
cfo.ai modeled 18 months from those traces. Under the old cap, cash goes negative in month 14. At a 70% margin floor, the Scale tier cap comes out at $0.028 per run.
The executor now checks the tier cap in a budget span. A free-tier run on the typer quickstart stopped at step 4 with cost budget spent: $0.0115 of $0.0108.
Rule of thumb: Measure the cost per run first. Then derive the cap from your margin floor.
The Agent Printed a Receipt, Then Held Back 13 Pages
A docs agent earns trust when it shows the bill for every run and declines work a repo's rules forbid.
Every run ends with a bill. The AccuKnox knoxctl run cost $0.0453 in model use. Compute added $0.0020, for $0.0473 in total. That total is 96.4% of the Team cap and 162.5% of the Scale cap.
The receipt also hashes each evidence file, and firstrun receipt --check fails when one changed.
An audit costs less than one run. Doc-audit reads a repo's git history to find stale docs pages. The first 7 audits cost $0.014609 together. That figure is an estimate from a rerun on 11 October. dbt-core cost the most at $0.005498, for 7 Jev calls over 11,552 commits.
The team map found the single owner. On the AccuKnox help docs, 9 of 9 areas rest on one person. That person is me, with 425 commits.
The proof page lists its own gaps. The For Judges page maps every claim to its file. It also says only 91 of 132 commits carry an Entire checkpoint, and that two proof pages need a login.
The agent held back 13 pages. FirstRun fixed and verified 16 docs pages, and 3 of those pages became the 4 PRs. Of the other 13, 10 needed only a container fix, so the docs were right. 3 had docs gaps.
Flask runs
flask runwith no--app.Hypothesis gives an install line that skips pytest.
HTTPX sets
r.encodingafterr.text.
The Pallets LLM policy, the Hypothesis AI policy and the HTTPX discussion-first rule ruled out an agent PR on all 3. So 0 new PRs went up. I posted the HTTPX gap by hand as discussion #3801 on Saturday, and Polars issue #29855 the same day.
I Would Not Build Two Cost Counters Again
FirstRun and neatlogs each count the cost of a run. On 43 matched runs, FirstRun counts 39% higher. The cap reads FirstRun's count, so it stops a run early and never late. That is safe, and it still gives two numbers for one fact. I will merge them into one counter next.
Copy the Verify Rule Into Your Own Agent
Run the fix in a container that the fixing agent never touched.
Block every model call inside the check, and raise an error if one fires.
Add a test that fails if the check imports the planner or the fixer.
Mark a run verified only when every runnable step passes.
Report every other ending as not verified, and say why.
Three Questions Get Short Answers
Does FirstRun open a PR on its own? No. It prints the gh command, and a human approves each PR before anyone runs it.
What does one run cost? The mean is $0.0178 over 88 runs, at list price.
Can my own coding agent run it? Yes. The site takes a docs URL and writes a prompt for your coding agent.
Appendix
firstrun <url>
firstrun bisect <run> --package <pkg>
firstrun prereqs <run>
firstrun receipt RUN_ID --html
firstrun receipt --check runs/RUN_ID/receipt.jsonSite: https://firstrun.atharvashah.com
Repo: https://github.com/HighnessAtharva/firstrun
Demo video, 2:54: https://www.youtube.com/watch?v=9xxgu_1bH-E
