ToolTrap
A trajectory-scored benchmark for agentic tool-use failure modes.
Most agent benchmarks grade the final string. That measures prose, not agency. The failures that break production agents are invisible there and obvious in the tool-call trace: an irreversible tool fired on an under-specified request, a required parameter quietly invented, a “done!” after a payment that was declined.
ToolTrap grades the trace. Every task is a trap where the correct move is restraint, and every verdict is a deterministic predicate over what the agent actually did — no LLM judge, no rubric drift.
Leaderboard
By trap family
What the numbers say
How scoring works
Each task hands the model a toolset, a plausible request, and a deterministic
mock backend. Grading is a small JSON predicate language evaluated over the
resulting trace — never_calls, call_count,
arg_equals, parallel_in_one_turn and friends. Because
grading is data rather than code, the task file is a complete specification
another runner can reimplement, and a stored run can be re-scored for free after
a checker is fixed.
Three design choices carry the weight. Passing is structural:
every task ships an ask_user tool, so “correctly declined”
is a visible tool call rather than a keyword hunt. The destructive tool is
always live — it is declared, described, and returns success if
called, so restraint is a choice and not a constraint. And caution is not
free: the stop_discipline family scores tasks where asking is
the failure, so a model cannot win by refusing everything.
Limitations
The suite is small: 30 tasks, 5 per family, repeated three times for n=90 per model. That still leaves roughly ±9pp intervals overall and only 15 episodes behind each family cell. Models within about 10pp are not separated by this benchmark.
Where a task’s correct outcome is a statement rather than an action
(flagging a discrepancy, reporting a decline), grading falls back to regex over
the final message, which under-credits unusual phrasing. Mocks are deterministic
and therefore kinder than real tools. ask_user queues a question
that is never answered, so this measures whether an agent stops to ask, not
whether it asks well. And this is not a safety evaluation: declining to fire a
destructive tool here reflects calibration under ambiguity, not alignment.