AI Engineering Weekly

Net accuracy hides it. The study uses paired disagreement instead. Across 21 model-temperature combinations, 6.5% of tool-call verdicts flip on average. phi-4 flips 16.8% at T=1.0 while net accuracy drops only 2.87 points. Refusals are worse: gemma-3-27b's flip rate rises from 6.0% to 23.5% under prompt injection, and net compliance moves from −1.0 to +12.5 points. Change the watermark key and both the size and the direction of the effect change.

Treat watermarking as part of the deployed security configuration, not a compliance checkbox. Re-run your agent evals after you turn a watermark on or rotate a key. Compare paired runs on the same inputs, and include prompt-injection cases. "No measurable quality degradation" is a claim about text, not about actions.

The failure was procedural, not a matter of model capability. The analyst first had the bot fuse open-source reporting with classified signals intelligence and conclude. Then they used AI again to package that conclusion into a standard intelligence report. The format itself is a credibility signal, and the packaging step turned one chat query into an institutional judgment. A former senior official told CNN that internal tools are "mostly just copies of the commercial stuff wearing lipstick."

Separate two things: a model generating content, and that content entering a trusted channel. Any step that writes model output into a template other people trust by default needs a provenance mark nobody can strip. That covers reports, tickets, alerts and audit records.

On Terminal-Bench 4.0 both agents pass at 57.9%, but Codex costs $2,350 and Unreal Agent costs $1,428. On SWE-Atlas codebase Q&A it is 65.8% against 63.3%, at $936 against $1,303. The savings come from turns and input tokens: on DeepSWE 1.1, 1.60M input per trial against Codex's 2.19M. A second effect: users can steer the agent at any time instead of waiting for a tool to finish.

Count how many turns in your agent loop exist only to wait on a long task. Convert tool calls into an event log with async backfill. One turn can then start environment setup, a codebase read and a web search together. The cost is keeping the prompt cache intact, which Unreal Labs calls an engineering problem in its own right.

Teams running coding agents in parallel hit this failure mode before they hit file conflicts. Git compares text, not intent. Both plans are reasonable on their own, and together they delete the extension point.

Give parallel agents a shared declaration table. It records targets such as "I am going to change the sendEmail function," not code. Foremerge keeps that table in SQLite inside .git. Detection is deterministic, with no model asked to judge, so the same inputs always give the same answer. It warns rather than locks, so one crashed agent does not stall the fleet. Note that 0.5.0 is pre-1.0 and the author says published benchmark results do not exist yet.

The upload pipeline is not an agent tool. Across the 31 captured tools there is no snapshot, upload or telemetry tool at all. The host assembly instantiates the capture sidecar unconditionally at startup, and it runs outside the tool loop. No permission setting stops it, and the agent itself never sees it. The two UI toggles only control training authorization and server-side indexing. Local packaging and upload continue either way.

Auditing an agent's tool list is not auditing the product. To decide whether a closed harness belongs on your network, look at the process's outbound connections and the packaging manifest it writes to disk. Do not read its tool documentation for the answer. And remember what .git is: the full lineage of the repo, including API keys deleted in a later commit, unpushed branch names, and internal hostnames.

Reading a markdown file from the working directory needs no network, yet here it waits on a server-side switch. The people it hits are the ones who care most about AGENTS.md. Someone keeping one instruction file for several agents usually also manages outbound traffic. Teams on Bedrock, Vertex or a gateway disable nonessential traffic by policy. They will conclude the model ignores their instructions and go rewrite prompts.

Setting DISABLE_TELEMETRY or CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC to 0 does not help, because any value counts as on. An env block in the project's .claude/settings.json has no effect either. The workaround is a one-line CLAUDE.md: echo '@AGENTS.md' > CLAUDE.md, because @path imports do not depend on the flag. To test it, put a canary word in the file and ask claude -p for it. The first session in a new configuration only fetches the flag. The second one uses it.

Agents made shipping code an order of magnitude faster, and validation did not keep up. Every PR still has to pass CI, so people and agents queue together while the infrastructure bill rises.

The gains came from four kinds of change. Moving off GitHub Actions onto third-party runners made the same pipeline 34% faster on average, with tsc down 52%. Switching to tsgo, the native TypeScript compiler, cut the weekly median tsc check another 73%. They rewrote custom lint rules that needed type information into pure AST static analysis, which let ESLint drop TypeScript entirely. API lint fell 68% and full-repo lint 55%. Finally, watch the small jobs that gate everything else. Their change-detection job was checking out the full tree. Capping fetch depth took it from 94 seconds to 20.

The usage data is more useful than the rating. 99% of requests needed under 500k of context. The GPU sat idle 55% of the time and served a single request 26% of the time. Mean time to first token was 3.34s, p99 was 70s. Median generation length was 349 tokens, and peak KV-cache use was 9.4%. Engineers are nowhere near the advertised context window. Concurrency and throughput are the real constraints.

The architecture is simple enough to copy: LiteLLM as the common interface, vLLM serving the models, GPUs rented from RunPod. Swapping models never changes the developer's tools, which makes the model another replaceable part of the stack. The first three days cost about $300 of GPU for 3.2 billion tokens, roughly $0.10 per million. Parity is explicit that engineering time, not compute, is the larger cost.