Is Your Library Agentic Enough? The Same Scaffolding Helped Big Models and Broke Small Ones
Hugging Face open-sourced agent-eval, a benchmark that measures the path an agent walks through your library: not just whether the final answer is right, but how many turns, tokens, and errors it took. Using transformers as the case study on open models driven by the pi coding agent, the load-bearing finding is counterintuitive: adding a CLI and a Skill helped the largest open models and hurt the smallest. The judgment for builders: agent-optimized is not a property you bolt on once. Ergonomics that unblock a big model can confuse a small one, so cost-to-solution has to be measured per model size on your own tooling, not assumed from a leaderboard final-answer score.
Summary
On June 18 Hugging Face open-sourced agent-eval, a harness for benchmarking a library for agentic use. It asks a different question than most benchmarks. Not whether an agent got the answer right with your library, but how much work it took and what path it walked to get there. Turns, tokens, errors, which files it read, how it finally found the answer, all captured and scored.
The thing to hold onto is the finding it surfaced. Hugging Face used transformers as the case study and planned to add an agent-optimized layer to it: a CLI, a Skill, and a set of task-specific examples. Intuition said this would make agents’ lives easier. The benchmark said something else. The same scaffolding helped the largest open models and hurt the smallest. In their own words, it is something they would have shipped on faith.
What happened
The transformers team’s motivation is concrete. Coding agents increasingly work the library directly instead of going through you: describe a task, and the agent picks the library, writes the calls, runs them, and debugs its own mistakes. When the library gets in the way, the agent will happily bypass it and rewrite the logic from scratch. So a library now has to be correct, fast, and drivable by an agent. A clunky API or stale docs is an annoyance to a human and a longer, more expensive path to an agent.
Their intuition was that a CLI, a Skill, and a few self-contained examples would dramatically simplify usage. The same recipe had just been applied to the hf CLI, where agents used 1.3 to 1.8 times (and up to 6 times) fewer tokens. But before adding several thousand lines to a codebase as widely used as transformers, they wanted evidence, not a hunch.
So they built agent-eval. The tasks are concrete: agents use transformers to classify text, caption images, and transcribe audio, not to contribute code to it. Each task runs under three tiers: bare (just pip install transformers), clone (the full source checked out in the working directory), and skill (a packaged Skill that loads the CLI’s docs and task examples into context). The full sweep of model × revision × task is fanned out across Hugging Face Jobs, one Job per combination, running in parallel on identical hardware so the comparison stays fair at scale. Every run is driven by pi, Mario Zechner’s coding-agent CLI, which needs only an HF_TOKEN to serve an open model.
Scoring goes well past the final answer. The harness scores several axes: match % (did the final answer contain the expected result, per task, case-insensitive substring, regex, or exact); median time and median tokens (new vs. cached vs. generated); error rate, with a guard that flags runs producing nothing (zero output tokens, no tool calls, no answer) so silent failures don’t masquerade as a “0”; and marker adoption.
A marker is the load-bearing design here. It is a named pattern the profile matches against a run, a one-line label for a behavior you care about, checked against the shell commands the agent ran, the code it wrote, the files it read, or its final answer. For transformers the two that matter most are cli (the agent invoked the command-line tool instead of writing Python) and pipeline (it reached for the high-level pipeline() API). Watching how those markers move tells you whether your change actually shifted the agent’s behavior.
Why it matters
The real finding lives in the split between large and small models. For the largest open models, completion saturates near 100% on common tasks, so match % stops being informative, and what matters is the effort it took to get there. Hugging Face fixed one strong model and varied the transformers revision, from v5.8.0 and v5.9.0 up to the commit that introduces the CLI and Skill. The result has two faces. On the skill tier the commit cut the time the big models spent, because they reach for the CLI instead of debugging Python. On the clone tier the same commit raised token consumption, because roughly a third of runs first read the newly shipped CLI implementation and examples, lifting median input from ~4k to ~6.4k tokens. One change buys the big models less time at the cost of more tokens, a tradeoff worth knowing before merging the PR.
For small models the experiment flips: hold the revision, sweep the model. Here match % matters more than tokens, because you want to see which models can’t reliably handle the tool calls at all. The result is stark: the skill tier lifts match % for the larger models and drops it for the smaller ones. On Qwen3-4B the Skill barely changed the match rate yet fanned the cost distribution wide open, almost all of it from the clone tier, where the agent read the newly shipped CLI source in bulk and its median new tokens jumped from ~2.4k to ~23k, with time and output skyrocketing for no gain in accuracy.
Qwen3-14B went further, with the Skill breaking correctness outright. Its overall match rate fell from 67% on bare to 43% with the Skill, and on the simplest task, sentiment classification, it collapsed from 100% on clone to 0% with the Skill. The traces show why: the model mistook the CLI for a tool it could call directly, like web-search. But a Skill is not an executable tool. It is documentation in context, and the CLI only ever runs from the shell. In 39 of its 56 Skill runs the model either emitted a never-registered transformers(command="classify") call or, finding nothing like it among its read/bash/edit/write tools, gave up. It didn’t even fall back to the one-line pipeline() that scored 100%. It declared the task impossible.
This is exactly what the harness was built to catch: the same change that sped up the big models broke the small ones, counterintuitive, and likely to have shipped as-is.
Builder impact
If you ship a library or tool in the agent era, the judgment to take away is that agent-optimized is not a property you bolt on once and forget. Ergonomics that unblock a big model can confuse a small one. A new affordance is fewer pitfalls for a strong model and a larger surface to get wrong for a weak one, because small models lean on memorized API patterns, reproducing pipeline() snippets they saw in training data, and a new concept is more room to be wrong.
So cost-to-solution (turns, tokens, errors) has to be measured per model size on your own tooling, not inferred from a leaderboard final-answer score. A change that looks clean on a big model can be a net negative on the smaller tier your real users actually run. Agent-facing APIs should be evaluated across model sizes. That is the rule maintainers should adopt.
It also hints at a fix: rather than hand-write a Skill and check it after the fact, generate and validate one against the weaker models up front, keeping it only when it measurably helps the smaller ones. That is the idea behind Upskill, which turns a strong model’s solution into a Skill only when it helps the smaller ones.
Shipping it is simple. agent-eval is one CLI: install it, run a suite, fan it out across model × revision on HF Jobs, and publish the report as a Hugging Face Space. It is profile-based and built to be portable, so you point it at your own library, define a few tasks and their expected answers, and get the same report. Code and tasks are in the repo, traces on the Hub.
What to ignore
Don’t read this as a leaderboard of open models. Hugging Face measured the cost curve of one library change across model sizes, not which model is strongest. Models are the variable. When Qwen3-14B drops to 0% on one tier, don’t conclude the model is bad. That is the skill scaffolding failing to mesh with that size, and on another tier (clone) it scores 100%.
Don’t take the token-spike number as the final word on the CLI either. Hugging Face flags the caveat themselves: their experiment is one-off, each agent rediscovers the CLI from scratch and pays the discovery cost every time, so the measurement is close to a worst case. In real use an agent learns the interface once and solves task after task in the same session, amortizing the cost, and the day-to-day token bump is much smaller.
Finally, the harness currently covers only deterministic tasks with an exact match. Model-as-a-judge and the rest are next steps. It runs a coding agent with bypassed permissions and executes the code you point it at. That security line, trusted local use only, is not one to skip.
FAQ
Why is checking only the final answer not enough to evaluate how well an agent uses a tool?
Because two agents can produce the same correct label at an order-of-magnitude difference in cost. One writes a 40-line Python script, debugs a shape error, re-runs twice, then prints the result; the other types a one-line transformers classify and is done. If you only check the final string you are blind to turns, tokens, latency, and failure rate, and you cannot see whether a change you shipped (a CLI, better error messages, a Skill) actually helped the agent. agent-eval captures all of those process metrics, down to the native trace of every command.
Is a Skill an executable tool, and why do small models misuse it?
It is not. A Skill is documentation loaded into the agent's context, and the transformers CLI is only ever meant to run from the shell via bash. Reading Qwen3-14B's traces, Hugging Face found that in 39 of its 56 Skill runs the model either emitted a never-registered transformers(command="classify") tool call or, finding nothing like it among its read/bash/edit/write tools, concluded it could not run a model. Rather than fall back to the one-line pipeline() that scored 100%, it declared the task impossible. That is a small model mistaking the Skill for a directly callable agent tool.
Big models spent more tokens on the clone tier. Did the CLI backfire?
Not backfire, but a tradeoff worth knowing before merging. The commit that introduces the CLI also ships its implementation and the cli/agentic/*.py examples into the repo, so on the clone tier the agent has a full transformers checkout in front of it and roughly a third of runs read that new surface to learn the interface, raising median input from ~4k to ~6.4k tokens. In exchange, big models reach for the CLI instead of debugging Python, cutting time and turns. That reading cost also amortizes across a real multi-turn session, so the bump Hugging Face measured in one-off runs is closer to a worst case.
Can I use this harness on my own library, and what should I watch for?
Yes. agent-eval is profile-based, where a profile is the small plugin that teaches the harness how to build and drive a given library. Point it at your own library, define a few tasks and their expected answers, and you get the same report. One red line: it runs a coding agent with bypassed permissions and executes code from whatever revision you point it at, and traces can contain prompts, output, and local paths. It is for trusted local use only; read SECURITY.md before pointing it at code you didn't write or sharing results.
Sources
No official primary source available; this analysis is based on reliable secondary reporting (named outlets, cross-confirmed).