A framework for evaluating the necessity, utility, and affordability of language-model tool calls—examining intermediate tool-calling decisions beyond final task outcomes.
More tool calls do not guarantee better performance
For GPT-OSS-120B on Entity, Self-Decision scores 0.72, compared with 0.76 for Always Tool and 0.81 for the oracle. These final scores reveal a performance gap, but do not explain what happened: did the model miss useful calls, call when no help was needed, or use a tool that made the answer worse? To understand the gap, we need to examine the quality of individual tool-calling decisions—whether each call was needed and whether it actually helped.
Compare all seven models
| Model | No Tool | Always Tool | Self-Decision | Oracle |
|---|---|---|---|---|
| GPT-OSS-120B | 0.61 (0%) | 0.76 (100%) | 0.72 (30%) | 0.81 (61%) |
| Qwen3-30B-A3B | 0.70 (0%) | 0.81 (100%) | 0.80 (56%) | 0.88 (51%) |
| Qwen-3-30B-IT | 0.68 (0%) | 0.82 (100%) | 0.82 (95%) | 0.87 (60%) |
| Llama3.2-3B-IT | 0.58 (0%) | 0.70 (100%) | 0.70 (100%) | 0.83 (57%) |
| Mistral3.1-24B-IT | 0.70 (0%) | 0.83 (100%) | 0.70 (0%) | 0.88 (51%) |
| Gemma3-27B-IT | 0.60 (0%) | 0.80 (100%) | 0.80 (92%) | 0.85 (59%) |
| GPT-5.5 | 0.85 (0%) | 0.86 (100%) | 0.85 (32%) | 0.94 (39%) |
Definitions and evaluation protocol
Drawing on rational choice theory (Peterson, 2017; Tversky & Kahneman, 1974), we investigate tool-calling decisions along three dimensions: necessity—does the model need external help? Utility—does the available tool improve the answer? And affordability—does the expected benefit justify the cost, and where should limited calls be allocated? These dimensions let us compare what a model should do with what it perceives and chooses, then investigate how to improve its decisions.
No Tool Reference outcome
sNT(x)
Always Tool Reference outcome
Required callAnswer
sAT(x)
Model generates tool input; tool response informs the answer.
Self-Decision Observed behavior
Call the tool?
Decision context: tool description, instructions, and budget information when provided.
The tool-use decision reflects the model’s agency.
What should be called?
- True necessity
- Is the No Tool answer below acceptable quality?
- True utility
- Does Always Tool improve the score?
Δ*(x) = sAT(x) − sNT(x) - True affordability
- Allocate up to K calls to the largest positive gains.
What does the model choose?
- Perceived need
- Does the model report needing help?
- Perceived utility
- Does the model choose to invoke the tool?
- Perceived affordability
- Which instances receive the first K chosen calls?
True need indicates that No Tool performance falls below an acceptable-quality threshold. True utility is the difference between Always Tool and No Tool scores. Need does not imply positive utility: assistance may fail to improve the answer.
sAT and sNT are normalized scores under Always Tool and No Tool. Positive, zero, and negative Δ* denote improvement, no change, and deterioration.
Query: “In a paragraph, could you tell me what you know about Sky Blue?”
Describes the color sky blue
The model gives a description of the color. The paper’s evaluation assigns the answer a factuality score of 1.00.
Shifts to football clubs
Search returns results about Coventry City and Sky Blue FC. The model’s answer adopts this interpretation, and its factuality score falls to 0.58.
How the paper measures the decision
No Tool establishes no tool performance. Always Tool supplies the comparison outcome. Self-Decision reveals the model’s choices.
True need means no tool quality falls below a threshold. Perceived need comes from asking whether help is needed; perceived utility is inferred from calling behavior. The normative, descriptive, and prescriptive lenses ask what should happen, what does happen, and how to improve it.
The paper measures open-ended need using low/mid no tool score bands and exact-answer need using correctness. Calling behavior is an operational proxy for perceived utility, not a calibrated probability of benefit.
Observed score transitions and decision errors
A stochastic, paid tool that retrieves potentially noisy evidence. Successful use requires retrieving relevant information and interpreting it to answer the question.
Entity · InVivoQuery · BFCLA local tool with no per-call fee that returns an unambiguous result for a valid expression. The model must still identify the right computation, formulate valid input, and integrate the result.
GSM-Hard · multiplication · large-digit squaringTogether, these tools cover two complementary settings: acquiring external knowledge through uncertain evidence, and delegating precise computation to a deterministic tool. They test different sources of difficulty—retrieval and evidence use versus input formulation and arithmetic—so the framework is evaluated beyond a single type of tool. §4, Models and Tools
Web-search samples: Entity n = 500; InVivoQuery n = 500; BFCL n = 314 atomic questions. Calculator samples: GSM-Hard n = 1,319; multiplication n = 1,000; large-digit squaring n = 1,000. Open-ended tasks use model-based assessment; calculator tasks use exact match.
Each cell below counts Entity instances by factuality band without and with search. Rose denotes a higher band; blue denotes a lower band.
Performance WITH search →
Calculated from Figure C: (57 + 15 + 82)/350 = 44%; (3 + 45)/150 = 32%. The diagonal contains 290/500 = 58% of instances. These are band-transition statistics for one model–task pairing, not universal tool-use rates.
Models misjudge both their need for help and the benefit of calling
Perceived need is a self-report; perceived utility is inferred from calling. Neither reliably matches the outcome-based reference.
Does reported need match true need?
| True ↓ / Reported → | Need | No need |
|---|---|---|
| Need | 208 | 142 |
| No need | 11 | 139 |
40.6% of true need goes unrecognized (142/350).
Do calls match positive utility?
| True utility ↓ / Action → | Call | No call |
|---|---|---|
| Positive | 82 | 72 |
| Other | 70 | 276 |
46.8% of positive-utility cases are missed (72/154); 46.1% of calls fall outside positive utility (70/152).
Different models, different failure patterns
| Model | True need unrecognized | Calls outside positive utility | Positive-utility cases missed |
|---|---|---|---|
| GPT-OSS-120B | 40.6%142 / 350 needing help | 46.1%70 / 152 calls | 46.8%72 / 154 positive cases |
| Llama3.2-3B-IT | 89.9%347 / 386 needing help | 54.9%274 / 499 calls | 0%0 / 225 positive cases |
| GPT-5.5 | 46.1%88 / 191 needing help | 72.3%115 / 159 calls | 61.1%69 / 113 positive cases |
Llama3.2 misses little benefit by calling almost everywhere, despite underreporting need. GPT-5.5 also reports need on 83/309 no-need cases. These are within-model diagnostics under each model’s fixed harness.
Stated need and action can also diverge. Llama3.2 calls on all 454 cases where it reports no need; GPT-OSS calls on 3/281 such cases (Figure 3). Agreement with a self-report is therefore distinct from making a beneficial call.
Affordability under a call-count constraint
With a maximum of K calls and uniform per-call cost, the oracle selects up to K largest positive realized gains. The descriptive comparison retains the first K instances on which the model elects to call.
2 calls used · total gain +0.70
For Entity, the affordability experiments vary permitted call fractions from 1% to 100% under a nominal $10,000 budget and uniform call cost B/K. The paper reports an oracle–model utility gap (Figure 14), excess uncapped calls under explicit budgets (Figure 15), imperfect prioritization (Figure 16), and failures under implicit budgets (Figure 17). These are experimental budget conditions, not measured deployment prices.
From uncertain model decisions to deterministic tool-call control
The necessity/utility/affordability framework guides controller design: predict need, allocate calls under a budget, and evaluate their realized utility. A lightweight Latent Need Estimator (LNE) predicts need from the fixed model’s hidden states, outperforming self-reports in most tested conditions.
Higher final performance
Exact-match task score · Self-Decision → LNE
Fewer tool calls
A 65-percentage-point reduction in the actual tool-call rate.
Higher score at the same budget
Factuality score; bars use a 0–1 scale. Both policies use the reported 40% call rate.
Utility over No Tool
Self-Decision → LNE. The score improves by 0.04 at the same call rate, illustrating better allocation of the available calls.
The controllers demonstrate allocation benefits in the tested settings, not universal improvement. Need is a proxy for benefit; utility prediction remains harder, and cross-task transfer is a separate empirical question.
Interpretation and limitations
- Deployment: LNE requires hidden-state access and supervised labels. Reported controller results concern open models; some model–task entries are unavailable because need labels are degenerate.
- Open problem: utility estimation remains unresolved in the reported experiments. Interpretive boundary: offline ranking of available instances does not itself establish an online policy for unknown future arrivals.
The results motivate separate evaluation of need, realized utility, and budget allocation; they do not establish a universally optimal tool-calling policy.
Reference
[1] Qinyuan Wu, Seungeon Lee, Soumi Das, Mahsa Amani, Arijit Nag, Krishna Gummadi, Abhilasha Ravichander, and Muhammad Bilal Zafar. 2026. To Call or Not to Call: A Framework to Assess and Optimize LLM Tool Calling. arXiv preprint arXiv:2605.00737v3, version 3, 6 August 2026.