You download an open-weight model onto a laptop – two or three billion parameters, no cloud, no per-token bill. You give it a real task: go through a folder, fix a bug in a script, build a comparison out of two spreadsheets. The first answer reads sensibly. Then the model starts calling the same tool over and over, announces after the twentieth repetition that the task is done although no file was ever written, or simply goes quiet halfway through. The conclusion looks obvious: the model is too small.
A paper from October 1 (arXiv:2610.02001, authors based in Beijing – University of Science and Technology Beijing and Honor Device) argues that this conclusion is often wrong. The usual culprit is the harness the model runs inside.
A model does one thing: predict the continuation of text. To do anything in the world – read a file, save a change, run a command – it needs a program around it. That program is called an agent harness. It assembles the text into one whole (the instruction, the conversation history, the tool definitions), decides when to call a tool, feeds the result back into the conversation, and makes sure the loop ends. The same model under two different harnesses behaves like two different models. Mingbird measures how large that difference gets.
The biggest problem is not the reasoning itself but what the model is handed on the way in. Before it sees the first sentence of the task, the harness pushes a catalogue of tools at it: a few dozen JSON descriptions, each with parameter names. That is several thousand tokens, and a token is the smallest piece of text the model processes; the context window is its entire budget of attention. The more text lands before the task, the less attention is left for the task itself, and small models lose accuracy long before the window fills up. Mingbird counts that overhead in bytes and holds it to a hard limit: the factory starting prompt is 797 tokens, pinned by a unit test, and every new feature must add zero – if something is added, something else has to be shortened or dropped.
The design follows the observed failures point by point. Three of the ten mechanisms show the thinking:
– Instead of one large tool catalogue, the harness first classifies the type of task and loads only the tools in that category, as a flat list. The cost: the classifier can misclassify and hide a tool the task needs, so the model can ask for a reload.
– The model cannot declare itself finished. Before the harness accepts “done”, it goes back to the original task text and makes the model check, point by point, that everything was delivered; it also runs the task’s own checks and pushes real failures back into the conversation – file name, line number, error text. That costs one extra model turn.
– Loops have a signature: the tool plus its normalised arguments. Six identical calls, or fifteen in a row with no output, and the harness interrupts with escalating pressure. In one reported case a 4-billion-parameter model made 156 tool calls and produced nothing; after the fix the same configuration finished in 15 calls with three files.
The measurement. The authors built their own test set: four harnesses, four open-weight models (2 to 35 billion parameters) and 18 real tasks, scored automatically and deterministically – a file on disk counts, not the style of the answer. Mingbird reaches 0.886, goose 0.631, opencode 0.479, agent-mini 0.405; all 288 measurement cells are public in the repository. On τ²-bench, an external benchmark of 278 tasks where the model talks to a simulated user, Mingbird scores 0.856 against 0.791 and 0.737. The same 18 tasks run on a top-tier model score between 0.997 and 0.478 depending on the harness. So this is not only about small models, although well-built harnesses stay within 0.072 of each other.
An honest caveat the authors put up front: attributing the result to individual mechanisms is directional in their work, not evidential. Replicating the same arm on the same night moves its mean by up to 0.069 – the size of every single-trial difference they report. The one properly paired comparison gives +0.10 for the executable completion guards. In other words: the mechanism stack works, but how much each piece contributes is not nailed down. The LRAB test set is the authors’ own, too – a measurement made by the tool’s creators, documented and reproducible, but not independent.
What is left for someone who keeps a model on their own disk? Mostly a change of question. When a small model flounders on a task, check first how much text it received before the task, whether it is allowed to declare victory without evidence, and whether anything is watching for repeats – instead of immediately looking for a bigger model. Mingbird’s code is Apache-2.0, available as a desktop application and a command-line tool, and the model, the tools, the memory and speech recognition all run locally; the only outbound call is to a search provider, if the user connects one.
We wrote about how a harness shapes an agent’s behaviour when DeepSeek Harness came out: https://ai.quailoop.com/deepseek-harness-open-source-agent-framework-where-everything-is-a-plugin-166k-stars-in-3-days/
This content was generated using artificial intelligence and reviewed by a human editor.
Sources:
– Mingbird: A Local-First Agent Harness Enabling Small Open Models to Complete Real Tasks (arXiv:2610.02001, 1 Oct 2026) – https://arxiv.org/abs/2610.02001
– Code, test protocol and all 288 results: https://github.com/Mingbird/Mingbird-agent
– τ²-bench, the external benchmark used in the paper (arXiv:2506.07982) – https://arxiv.org/abs/2506.07982
Polish version: https://ai.quailoop.com/pl-mingbird-gdy-maly-model-lokalny-sobie-nie-radzi-wina-lezy-zwykle-w-szkielecie-a-nie-w-modelu/
Leave a Reply
You must be logged in to post a comment.