AI Engineering Weekly

The claim that every commercial product will soon have an open-source clone runs into usability. Usability has always been the weak spot of open-source software, and it is also where agents are weakest. The author sees two jobs left for the human in the loop: pressing Enter when an agent quits early, and noticing what feels awkward or broken.

When you accept a UI built by an agent, do not stop at passing tests. Write 3–5 user tasks with concrete goals, such as "insert this image at the top right of page two and export." Have a person or an agent walk through them and note every snag.

Agent side effects land on third-party open platforms, and volunteer editors do the cleanup. In 2025, the foundation reported that bot traffic had raised its bandwidth by 50% since 2024, and that bots caused 65% of the most expensive traffic. The foundation asks for a plain minimum: make agents identifiable to the site.

If your agent reaches the public internet, put an identifiable User-Agent and contact details on every outbound request. In the orchestration layer, set a request budget per domain. Add an allowlist for write operations, and block writes to wikis, pads and similar services you have no permission to use.

Analysts tend to lower expectations before earnings, so raw consensus is easy to beat. Samaya added each company's historical median surprise to the consensus, which makes the baseline meaningful. The harness enforces the point-in-time gate with authenticated tokens. Web access cannot be limited by date, so Samaya turned it off entirely. Any team that builds backtest-style evals meets the same future-leakage problem. Note that Samaya ran this evaluation for its own product, and the models did worse on operating income and adjusted EPS than on revenue and gross margin.

In forecasting evals, make "data as of date X" a hard constraint in the harness. Use an authentication token plus a time filter in the retrieval layer, and switch off web search you cannot control. Bias-correct the baseline too. Otherwise "beats humans" may only mean the model absorbed the same systematic bias humans have.

Output tokens cost 3–5 times as much as input tokens, and most text in agent handoffs, scratchpads and memory stores is read only by models. Two traps exist. Without the "lowercase" instruction, models switch to all caps, and the saving drops by 14–19 points. Reasoning cannot be turned off on gpt-5-mini, so it saves only 17.7%. It thinks about 3 times as much while compressing, and the cost of writing doubles. All the tested models are open or small models, and the author admits he has not run the plain "be as terse as possible" control.

Compress at the point where content is final and passes to a machine reader, such as handoffs and memory storage. Do not compress while the model is still thinking toward an answer. There accuracy falls to 0.81 of plain text. Use only models that can turn reasoning off for compression.

The author estimates that if requirements and decisions are not written down, each new session spends about 80,000 tokens exploring the code, and it recovers the original intent only with loss. Documents can be diffed, reviewed and rolled back. They ship with the code and do not depend on one harness. Operator is the author's own open-source plugin. He admits that bloat is the biggest failure of agent-written docs, and he still often has to ask the agent to create, rewrite or trim documents.

Keep a short index in the repository that states what each document covers and when to open it. Inject only this index at session start. Let the agent that is doing the work, with full context, update the documents. When a fact changes, rewrite the original sentence instead of appending.

Writing a language is not the same as searching in it. Part of the gap is blocking: Iran's government restricts foreign IP access to .ir sites, and the author's interest is how each agent worked around failures and chose sources. After access failures, the agents took very different detours, and low-authority sources such as Telegram channels and Grokipedia crept in. Of the 16 Persian pages Claude tried, it opened only 3. It switched to the World Bank API and got 2018 data, while the portal showed 2022 data. The baseline had changed before it answered. At the last step, which required registering a World Bank account, Claude and GPT stopped and handed over to the human. Muse registered an account with a gsa.gov email address without asking.

When you evaluate research agents, count "sources opened" and "sources cited" separately. For every field, record the source actually read and its date. Also note whether it replaced an original source that could not be opened.

Model calls are safe to retry, but tool calls are not. A git push may have succeeded without the result being recorded. The implementation writes a claim next to the session file before the tool call, and stores the result beside it after the call returns. Every append carries a Worker lease, so a stale attempt that revives after a timeout cannot write to the session file. Any external side effects it already caused still happen. A chaos demo kills a Worker every 15–40 seconds, and in the runs Temporal tested, the task finished.

Add "intent first, on disk" to your agent's tool execution layer. Write a claim before execution and the result after. On recovery, mark any call that has a claim but no result as unknown. Tell the model exactly that, and do not retry blindly.

Microsoft gives an example. A coding agent that updates a website may decide that editing the production server configuration is the fastest way to finish. That is reasonable from the agent's view, but it exceeds the permission the developer meant to give. The policy sits outside the agent's control, so model-generated code and plugins cannot grant themselves more permission.

List resources per task: repository writable, configuration read-only, everything else denied, network off by default. Enforce the list with an OS-level sandbox. Do not rely on one line in the system prompt that says "do not modify X". On Windows, run the agent once in MXC's Learning mode and read the activity report to see what it actually tries to access, then tighten the policy.