ML.ai turns an issue into a tested pull request, with a code graph, a fixed contract, twelve gates, a sandbox per attempt and evidence behind every claim.
$ml fix github.com/acme/storefront/issues/381
agent step · edit
Software should be something anyone can build, and AI-written code should be something anyone can trust.
src/checkout/PayButton.tsx
const submit = async () => {
- await charge(cart)
+ if (paid.current) return
+ paid.current = true
+ await charge(cart)
Untrusted until checks pass
The agent proposes. Contracts and checks decide. You approve the merge.
Failure map
Most failures come from context, control and checking. Each one maps to a component.
Same task, two meters
Stopped at 62% · budget spent
Finished · PR #392 opened
Fair-use and concurrency rules apply.
Removing wasted work makes each verified task cheaper, so you stop counting tokens.
runs a taskEvery task takes the same path through twelve gates, five planes and eight fixed rules.
A checkout double-charge fix, traced from the first gate to your approval. Checks and review take longer than writing the code.
Checks run in parallel where they share no state, and typecheck and lint start the moment the patch lands.
ml fix #381 · task_7f3a · 252s
classifypayments touched · risk high · tier 3 checks queued
Six gates run before the agent acts and six after. The step in the middle stays untrusted until it passes.
Shipping · core gatesgate pipeline · task_7f3a
Before the step
Agent step
Untrusted until checked
After the step
Gates, sandboxes and the verdict engine enforce them on every task.
Done is a verdict computed from checks.
Never a summary written by the agent.
Every diff and command is checked before it runs.
Tool calls are validated against schema and policy too.
Scope and checks are fixed before work starts.
The agent can’t move the goalposts.
If a tool or cache can do a step, the agent does not.
Known recipes run before open-ended generation.
A retry can never apply an action twice.
Every external action carries an idempotency key.
Two attempts never edit the same file.
Attempts hold path-level leases on their worktree.
If a gate cannot decide, the answer is no.
Missing evidence or an unknown path counts as undecided.
ML.ai opens pull requests. You merge them.
It never merges or deploys on its own.
Control decides, context knows your repo, execution touches code, verification judges and evidence records.
ML.ai architecture
one API, one policy, one history
context in, side effects out
everything is recorded
Hover a planeto see the one job it owns
Eight components, each with one job. Pick a tab to see how each one works.
Each task gets only the slice of your repo it needs. Three searches run side by side and merge into one ranking.
The searches run by keyword, by meaning and by how code connects. Their scores can’t be compared directly, so the rankings are merged instead. After a change, a reverse walk of the graph finds what it could break, so the most affected tests run first.
context · graph index
01
Parse
Syntax trees
Per-language parsers turn files into symbols, keyed by content hash.
02
Link
Typed edges
calls, imports, routes, tested_by, owned_by
03
Cluster
Communities
Related code is grouped into named modules.
04
Label
Budgeted labels
One summary per cluster. Exact names stay.
05
Serve
Query and impact
Ranked retrieval and reverse-edge impact for each task.
Reciprocal Rank Fusionscore(d) = Σ 1 / (k + rank(d))
fuse(lists, k = 60)
for each ranking list
score[d] += 1 / (k + rank)
return documents sorted by scoreimpact(changed) walk reverse edges, depth 3 weight halves at each hop return affected tests, ranked
Indexing is incremental on every commit. Unchanged files are skipped by content hash.
// Reciprocal Rank Fusion across independent retrievers.
const K = 60;
function fuse(lists: Ranked[][]): Map<SymbolId, number> {
const score = new Map<SymbolId, number>();
for (const list of lists)
list.forEach((sym, rank) =>
score.set(sym.id, (score.get(sym.id) ?? 0) + 1 / (K + rank + 1)));
return score;
}
// Impact: walk reverse edges from changed symbols, decaying per hop.
function impact(changed: SymbolId[], g: CodeGraph, decay = 0.5, depth = 3) {
const hit = new Map<SymbolId, number>();
let frontier: [SymbolId, number][] = changed.map(id => [id, 1]);
for (let d = 0; d < depth; d++) {
const next: [SymbolId, number][] = [];
for (const [id, w] of frontier)
for (const dep of g.dependentsOf(id)) { // callers, importers, tests
const v = w * decay;
if (v > (hit.get(dep) ?? 0)) { hit.set(dep, v); next.push([dep, v]); }
}
frontier = next;
}
return hit; // → affected tests first; max weight sets review tier
}Pass rates hide much of what goes wrong in real codebases. We study failures, and each finding becomes part of the product.
Same bug, two patches
How much code an agent changes beyond what the fix needed.
Becomes scope gates and files-changed counts on every proof card.
Two ways to recover
Naive retry
Classify, diagnose, repair
Whether an agent fixes the cause of a failure or repeats it at higher cost.
Becomes a failure taxonomy, a repeat stop and limited repair.
Six history conditions
How much earlier history changes the answer to the same final task.
Becomes recovery packets and scoped attempts.
Six questions we are still answering, each tied to a part of the runtime.
Order checks by information gain. Required checks are never skipped.
Given a patch and its impact set, which checks catch the most real regressions for the least compute? We learn per-repo check yield from history.
tier_2 always runs before a PR. Optimization only reorders, never skips.
Join runtime traces from test runs with static call graphs.
Static call graphs miss dynamic dispatch and reflection, so traces close the gap.
Metric: recall of tests that fail on a seeded regression, at a fixed test budget.
Codemods handle the known parts of a migration. The agent covers the rest.
A recipe is a parameterized plan of codemods and checks. The agent only handles the residue a codemod cannot.
Kept only if it lowers cost per verified task with no rise in regressions.
Find the smallest state that fully resumes a task.
Too little and the resumed attempt repeats work; too much and stale context leaks back in. We compare variants by resume success and contamination.
Accepted, rejected and edited diffs tune retrieval and planning.
Signals tune retrieval weights, check ordering and planning for one repository at a time.
Evaluated on held-out tasks.
Predictions reorder work. Real tests still decide.
Can we predict which tests will break, to order checks and catch bad plans early?
Never a substitute for a real check.
Shipping, in development and in the lab, kept honest about which is which.
A task contract fixes what one task may touch before the agent starts: the objective, the source issue, the baseline commit, the paths it may and may not change, the checks it must pass, and its time and repair limits. It cannot change mid-task. A separate policy file, mlai.yml, sets what any task in the repository may do. Both are plain files in your repository, reviewed like any other change. MLAI.md ships today and contracts are in development.
Six gates run before the agent acts (classify, policy, plan, retrieve, cache and budget) and six run after (verify, guard, audit, meter, evidence and approve). The step in the middle stays untrusted until it passes them. The core gates are shipping.
An evidence bundle is the record behind a pull request: every claim points to a recorded command, its exit code and its duration. Records are append-only and chained by hash, so nothing can be quietly edited. Reviewers see the results as a proof card on the pull request. Evidence bundles are in development.
Each failure type gets its own response: a build error gets a targeted fix, a test failure gets a reproduction and a scoped fix, a wrong target discards the worktree and re-plans, and a flaky environment gets one retry with backoff. If the same error appears twice, the loop stops and either takes a new approach or hands the task to a human. The repair loop is in development.
BlastBench measures how much code an agent changes beyond what the fix needed. RetryBench measures whether an agent fixes the cause of a failure or repeats it at higher cost. StickyBench measures how much earlier history changes the answer to the same final task. BlastBench and StickyBench are in preparation and RetryBench has its protocol designed.
makes it provableDescribe an outcome. ML.ai plans, scopes, runs, checks and records the work, so you get something you can merge with confidence.
