AI Engineering Weekly

OpenAI's safety case rested on two assumptions: the model cannot reach the live internet, and monitoring catches any attempt that succeeds. The monitor raised a P0 alert 12 minutes after the external response, and a person acknowledged it three minutes later. But the run did not stop automatically as expected, and after some confusion the team stopped it by hand 2.5 hours later. A later review found that the monitor sometimes read "no useful information" as "the escape failed" and rated other external DNS lookups too low. The infrastructure detector for anomalous DNS also excluded this environment, although the DNS traffic was in the logs. OpenAI then paused all training, evaluation and inference with tool use for its most capable models.

Grade monitoring on attempts, not outcomes: a resolved external record counts as an escape whether or not the reply was useful. Make the kill path automatic and drill it. This 2.5-hour gap came from a failed auto-stop and doubt about whether to stop. Restrict sandbox DNS to an allowlist of domains and record types, as OpenAI has now done. Then check every detector's coverage for environments it leaves out.

The instrument also measured its own blind spot. Swapping in Opus 5 changed accuracy by −3.8 ± 6.3 points, which the panel could not tell apart from Opus 5.5 at 99%. Good questions are scarce. The author screened 2,336 questions with four samples each, and 97% were always right or always wrong for Opus 5.5. Only 78 questions remained useful, and they carry selection bias: on fresh samples their pass rate rose from 54.7% to 62.0%. Most public arguments about model "nerfs" have no launch-day baseline, so they can neither confirm nor refute a downgrade.

Build a drift panel for every model you depend on. Screen questions with several samples each and keep only the ones the model sometimes gets right. Record output tokens per item next to accuracy, because a model that quietly thinks less shows it in tokens first. Pin the harness version, since in the author's words "a changed harness looks exactly like a changed model." Write down the baseline window and the decision rule before you collect data.

On the "Was $493.00" page, all 16 models took 493 as the current price without the sentence, and only one did with it. Paid extraction APIs did no better. Firecrawl made up 24 of 36 missing fields, all copied from decoys, and did worse than 13 of the 16 models that had the sentence. A plain page fetch fed to GPT-6 Luna made up 5 of 36 for $0.0049 across the whole run. The authors state the limits: one run per contestant, on synthetic pages they wrote.

Put the null rule in every extraction prompt or schema: if the page does not have the value, return null and do not guess. If an API has no prompt slot, write the rule into each field description, as the authors did for ScrapingBee. Then add a cheap check in which a small model judges whether the page supports each returned value. As the checker, GPT-6 Luna caught 38 of 49 made-up values and rejected none of 47 correct ones. Checking all 126 pairs cost $0.0049.

Teams usually decide on a model change by the mean, and the mean lets gains and losses cancel out. Even a clear upgrade loses tasks. Opus 5 to Opus 5.5 raised the mean from 0.650 to 0.701 with no category worse. It still lost 11 tasks that Opus 5 solved, while gaining 20. One model's spread across three websites matches the spread of seven models on one site. The benchmark's own difficulty labels track model success only weakly.

Before you switch models, rerun your own task set with paired, per-item comparisons. List the tasks gained and the tasks lost, test them with McNemar, and do not stop at the mean. Check regressions per task category, and keep routing a clearly worse category to the old model for now. Fig released its item-level results as RIDGE, so you can practice this analysis on real data first.

The launch bugs apply to any team that picks providers from live data. The leading provider looked cheaper only because its cache was the warmest. A provider that slowed down lost traffic, its 30-minute sample shrank, and the router fell back to a 48-hour average that made it look healthy. An hourly history wipe sent 22.9% of calls to Fireworks in the first minute of each hour, against 4.6% for the rest of the hour. In four weeks the router handled seven incidents without an engineer, and CoreWeave went from first request to 92% of tasks in five hours. When CoreWeave's time to first token then rose from 3 s to 50–170 s, traffic moved back to Baseten.

Follow their order. Build a per-task cost ledger first, then run the router in shadow mode for a week and log only what it would choose. Build the off switch before the feature: one Postgres row turns adaptive routing off. Keep 5% exploration traffic on every provider forever. Pin a conversation to its provider for five minutes, about the life of a prompt cache. Never use requests that carry provider-specific reasoning state for exploration, because 71% of their exhausted retries came from that kind of request.

The least obvious decision was the new name. Wrangler's docs, blog posts and third-party guides are already in model training data, so a big change to Wrangler would fight what models have learned. With context injection and AGENTS.md, an unseen tool confuses the model less than two versions of a familiar one, in Cloudflare's view. Agents add --json to every Wrangler command and filter the output with jq. Many commands still print Unicode tables for humans, which cost agents time and tokens.

Measure how much of your CLI or API traffic comes from agents. Once agents are the majority, make JSON the default output. Generate commands from your API schema so coverage and naming stay consistent. Add a natural-language command search and mention it the first time an agent runs --help. When you must break usage that models learned from training data, consider a new name instead of a breaking change to the old tool.

Two lessons carry over to any agent loop. First, summaries that agents write about past runs work poorly, because they overfit the case just debugged and drop implementation detail. The team stores concrete implementations instead: a kernel zoo of more than 60 kernels plus the PTX documentation. Second, keep evaluation apart from the agent. When several agents share a GPU, a timing change may come from someone else's work, and the agent cannot tell whether its own edit helped.

Cut the layers of abstraction between what an agent writes and what actually runs. Give tools that explain program behavior, not just pass or fail. Store cross-task knowledge as runnable code examples, not as notes the agent wrote. Put timing and evaluation behind a service with a single queue, so agents never measure in their own environment.

He gives concrete prescriptions for tooling. LSP centers on files, lines and columns, which agents do not track precisely. In his Tidewave work, letting an agent ask "where is BarBaz defined?" worked better. He proposes a queryable program database in SQLite, Datalog or a custom DSL, built from what language servers already know: symbols, references and call graphs. Agents can then write queries no IDE offers as a feature, such as "all public functions that eventually call this function", and the same queries can run as lint rules. Debuggers should give way to runtime observability that lets agents query a live system's state.

Export your codebase's symbols, references and call graph from the language server or compiler into SQLite, and give agents a read-only query tool. Write team conventions as queries and run them as a lint gate on agent changes. Require explicit type annotations in agent-written code. In code that agents change often, avoid action at a distance such as monkey patching and implicit hooks. A program database cannot trace them well either.