Small-model builders: a 542M model learned six tools. It did not learn what a tool declaration is.

Small-model builders: fixed tool use and declaration reading are different capabilities, and in these experiments the observed boundary lies within the 350B-to-4T pretraining interval. A better fine-tune did not buy it back.

I trained Drummer, a 542M-parameter language model, to call tools. On tools present in its training corpus, it scored 116/120 on a sealed, execution-verified test. That sounds like tool use. It is, with an important qualifier: it is tool use over a fixed catalog.

When I declared a tool in the prompt but withheld it from training, Drummer scored 0/24.

That result reproduced. Then it survived controls for the fine-tuning corpus, parameter count, instruction tuning, model family, pretraining budgets up to 350B, and finally function-name memorization. The last control took three jobs, two of which failed in their first minute, and closed the most plausible remaining loophole. Masking did not recover declaration reading and decisively harmed trained-tool execution: 88/120 versus 116/120, with 31 paired losses against 3 gains. A post-hoc 4-to-9 shift in correct-name emissions is unregistered, one-seed evidence from five discordant pairs; the likelier mechanism is more copying and less recall, not recovered capability.

The short version is that a small model can become very good at operating a set of known tools without learning to read a new declaration. Those are not two points on one smooth curve. In these experiments they separate cleanly.

The result that started this

Drummer is a decoder-only model I trained from scratch. The current foundation has 542,310,720 parameters and has seen 15.85B tokens, and every tools caller here was fine-tuned from its 10.85B-token checkpoint. A later refit on the 15.85B continuation, at a gentler learning rate, also made 0 of 24 valid calls. It is a research model, not a product, and its general benchmark scores are ordinary for its size and training budget. That is useful here. There is less hidden competence to confuse with the thing being measured.

The tools fine-tune uses declared tool descriptions and execution-verified targets. This worked much better than I expected on the catalog it saw during training. The 126M model went from 5/120 to 107/120 on the sealed execution test. The 542M model reached 116/120, and across its three tests it called no tool on any of the 100 slots where none was wanted.

Then I tested transfer. The prompt declared tools the model had never seen in training, including their names, descriptions, and argument schemas. Twenty-four sealed slots required one of those held-out tools. Drummer selected and executed the right call in 0/24.

This was not a formatting failure. On those 24 slots Drummer declined 8 times and asked a clarifying question 6 times; it named the declared tool 4 times, always with an incomplete argument set, named some other tool 3 times, and produced unparseable output 3 times. It never produced one schema-valid call to a tool it had not been trained on. Forcing its output into the tool-call grammar did not rescue it: under constrained decoding it made one valid call naming the declared tool in 24, and none was correct.

Given a request to move a date back six days with dates.shift declared in the prompt, its best answers used the right name and dropped the days argument. It behaved as if the declaration were a label to copy rather than a schema to fill.

That gave me two competing explanations:

1. The corpus was too small and repetitive. Drummer had learned six tool names rather than the relation among a request, a declaration, and a call.
2. Reading an unseen declaration is an in-context instruction-following capability that a 10.85B-token pretraining budget never produced. A fine-tune can specialize an existing capability, but it cannot cheaply install the missing substrate.

The rest of the work was an attempt to make those explanations disagree.

Same fine-tune, different pretraining histories

I applied the same LoRA corpus and scoring procedure to models near Drummer's size. The held-out tools were absent from fine-tuning for every arm. Rendering matched training, decoding was greedy, and every arm saw the same sealed slots. "Valid call naming the unseen tool" counts a schema-valid proposal that names the held-out tool; invalid output that happens to contain the right name does not count here.

Every Drummer caller here, including the masking arm, used the 10.85B-token parent. All three models learned the trained catalog. Only the models with much larger pretraining budgets transferred to tools declared only in context.

On tool selection, one-sided Fisher tests against Drummer gave p = 0.00011 for Qwen and p = 0.00030 for SmolLM2.

The error pattern matters as much as the totals. On one dates.shift slot, Qwen returned the right unseen tool and the right declared argument names, but put the wrong sign on the day offset. That is an arithmetic error after reading the declaration. Drummer's output on that slot was not valid JSON: it reached for the trained calculate tool and pasted the request text into its arguments. That was not its typical miss. The four times it did copy the declared name dates.shift, it left out the required days argument: it used the label without filling the schema. Those failures should not share a label just because both calls fail execution.

Instruction tuning was not the missing ingredient

Both reference models above were instruction-tuned, so perhaps the comparison had measured post-training rather than pretraining volume.

I ran the base Qwen2.5-0.5B model with the same corpus and recipe. It selected the unseen tool in 8/24 slots and executed 7/24 exactly. The Instruct sibling selected 11/24 and also executed 7/24 exactly. Base against Instruct on selection was one-sided p = 0.28 (two-sided 0.56), with no separation on the graded score.

A base model with no instruction tuning read unseen declarations eight times where Drummer read them none. Instruction tuning is therefore not a sufficient explanation for the gap, and more instruction data is not the cheap repair this result first appears to suggest.

One scoring correction belongs in the record. The base Qwen generation config did not stop at the end-of-turn token used by its chat template, so valid objects were followed by repeated text. I changed the scorer to consume the first complete JSON object, which is what the executor consumes, for every arm. Re-scoring both Instruct arms reproduced their existing rows exactly: 75/96 overall, with 7/24 and 8/24 held out. The correction rescued a real harness mismatch without moving the controls. The base Qwen arm was also run on two backends, Apple-GPU and CPU: greedy decoding produced different text on 45 of 96 slots and different correctness on 5, with totals of 73 and 72 and the same 7/24 held-out exact count on both. The gap-model zeros below rest on the CPU backend alone.

Looking for the threshold

At this point the threshold was somewhere between Drummer's budget and SmolLM2's ~4T. That interval was too wide to guide a training decision. Buying a 100B-token run to discover that it still fails would be an expensive way to obtain the number zero.

Three existing model families already occupied the gap. I fine-tuned OPT-350M, Pythia-410M, and BLOOM-560M on the same corpus and scored the same sealed test.

All three gap models learned the trained tools. All three failed completely on held-out tools. They also came from different model families, which makes an architecture-specific account less plausible. Among the models tested, the observed capability boundary lies within the 350B-to-~4T interval.

That is a range, not a scaling law. I did not train matched checkpoints across it, so I cannot locate a clean transition or claim pretraining volume is the only difference among every row. I can say that parameter count does not explain the result, instruction tuning does not explain it, several architectures fail below 350B, and the models that succeed have seen roughly 4T or more.

For Drummer, that is enough to price out the repair. Reaching 350B tokens is about 32 times the caller's 10.85B-token parent (22 times the current foundation's 15.85B), and the evidence says it buys none of the target capability. Four trillion is roughly 370 times the parent's budget, about six months on four H200s. This project is not going to train through that uncertainty interval to find out where the first nonzero lives.

The last loophole: maybe the names were the curriculum

The original corpus used the same small set of tool names in every record. A model could minimize loss by memorizing those identifiers. In that setup, copying a name from a new declaration is never rewarded during training.

I preregistered a masking arm before running it, following the function masking of Lin et al., 2024 (Hammer, arXiv:2410.04587). Each training record received fresh opaque aliases for every tool name and argument key. The aliases appeared in both the declaration and the target, while descriptions and schema types stayed intact. A model could no longer solve training by storing six global names. It had to connect the current request to the current declaration and copy that record's identifiers.

The registered held-out metric and decision bands were fixed in advance. The metric is the sealed test's held-out score as every table above reports it: a schema-valid call naming the held-out tool that executes correctly. The band text said "selection" without spelling that out, and I want to be plain that the raw name count discussed later was never the registered metric:

• 6 or more of 24: masking recovers declaration reading.
• 0 to 2 of 24: masking does not recover it.
• 3 to 5 of 24: equivocal, requiring a second seed.

The trained-tool do-no-harm tolerance was also fixed: within four slots of 116/120 on the sealed execution test and within four of 28/28 on the transfer test's trained group.

The masking attempt took three jobs. Two failed in their first minute because the experiment exposed mismatches between the training profile and the package validator. The successful arm used 11,054 training records, three passes, 210 updates, and 573,217 target tokens per pass. The unmasked comparison used the same 11,054 records, three passes, 204 updates, and 552,312 target tokens per pass. Aliases tokenize a little longer, so the masked arm received 3.8% more target tokens. That favors the masked arm, which is the useful direction for a null result.

The preregistered result was 0/24.

Masking did not recover declaration reading: the preregistered primary result was 0/24. It also decisively failed the do-no-harm check. Sealed execution fell from 116/120 to 88/120, with 31 paired losses against 3 gains (p = 7.7e-07), against an incumbent re-run whose 120 outputs were byte-identical to its original qualification; invalid responses rose from 2 to 21, precision fell from 0.971 to 0.784 and coverage from 0.972 to 0.708. On the transfer test's trained group it scored 26/28 against 28/28, but the execution harm is decisive.

That is the confirmatory result. It closes the original question for this experiment.

Post-hoc: an output shift under harmful masking

The registered score is not the whole output, but the next part is exploratory and stays that way.

On the same 24 held-out slots, counting the name in the emitted proposal whether or not the proposal was valid, the unmasked and masked models emitted:

There were five discordant pairs for correct-name selection, all five favoring the masked model. One-sided exact McNemar p = 0.0312, two-sided 0.0625. That is the smallest p-value five discordant pairs can produce, the direction was chosen after the data, and it was one of several post-hoc looks at the same output. Invented or unparseable outputs fell from 5 to 0.

The tempting version is that masking worked. It did not, by the metric I registered. Correct name selection was not the registered outcome, this is one seed, and five discordant pairs is a thin base for a new claim.

The post-hoc output distribution shifted: correct-name emissions rose from 4 to 9 and invented or unparseable outputs fell from 5 to 0. This does not establish improved capability. Under masking the model produced that signature more often, which reads as more copying and less recall; exact held-out execution remained 0/24. All nine correct-name outputs were invalid: seven omitted the required days field and two supplied both keys with values that fail the schema, producing calls like:

{"name":"dates.shift","arguments":{"date":"2004-10-12"}}

The name shift is compatible with more copying and less recall, not with recovered declaration reading. Argument construction remained at zero. The comparison that settles the reading is the gap models under the same raw count: OPT-350M emitted the correct unseen name 2 times, Pythia-410M 8 and BLOOM-560M 10, every one of them invalid and every one scoring 0/24 exact. Drummer's own unmasked baseline shows the same shape: all four of its correct-name outputs omitted days too. Masking made that generic below-threshold signature more frequent; it did not create it. Emitting the right name with a broken argument set is what a model does when it copies; it is not what a model does when it reads.

If name selection is studied again, it should be preregistered as a separate diagnostic outcome. A second seed could test whether the 4-to-9 shift replicates, but it would not change the original preregistered 0/24 null or the observed execution harm.

This distinction is slightly annoying, which is a good reason to keep it. The point of preregistration is not to prevent looking at interesting output. It is to prevent an interesting output from quietly replacing the question that produced a null result.

What I think the experiments establish

Drummer can learn a fixed catalog well. The jump from 5/120 to 107/120 at 126M, then 116/120 at 542M, is real and useful. If an application has a stable set of local tools, a small fine-tuned model can be an effective constrained caller.

That does not mean it can read declarations.

Across these arms, held-out declaration use is associated with pretraining budget, while parameter count and instruction tuning do not explain the observed pattern. Every tested model at 350B or below scored 0/24; the tested models at roughly 4T and 18T succeeded on 8 to 11 of 24 under the same fine-tune. This is an observational 350B-to-4T interval across model families, not a scaling law. Masking left exact unseen calls at 0/24 and decisively harmed trained-tool execution; its 4-to-9 correct-name shift is post-hoc, unregistered, one-seed evidence from five discordant pairs.

The practical mistake would be to call the first result "tool use" without naming the boundary, then assume a larger tool corpus will turn it into open-ended function calling. For Drummer, more tool rows are not the first repair. The missing capability appears upstream of the tool fine-tune.

Limits

This is a small study with a small sealed set. Twenty-four held-out slots can distinguish zero from the reference arms here, but it cannot map a smooth scaling curve.

The corpus declares about six tools. Published tool-calling recipes often use hundreds or thousands. These results show what this corpus could and could not elicit; they do not show that no sufficiently broad synthetic tool curriculum could produce partial transfer at 542M.

The pretraining-budget comparison is controlled at the fine-tuning stage, not at pretraining. Architectures, token mixtures, tokenizers, and training recipes differ across model families. The base-versus-Instruct Qwen comparison removes one major confound, but the 350B-to-4T interval is still observational across families.

The held-out set is two tools, twelve prompts each, and the reference arms' successes are concentrated on one of them. Counting schema-valid selections and exact calls: on dates.shift, Qwen-Instruct 10 and 7 of 12, SmolLM2 8 and 6, Qwen base 7 and 6; on text.replace, 1 and 0, 2 and 2, 1 and 1 (Apple-GPU run; the CPU re-run splits 6 and 5, 2 and 2). dates.shift is lexically transparent from prompts like "roll 2029-04-19 back 6 days", so the reference arms' reading may be as shallow as mapping "days" to the tool whose name says dates. The effective sample for "reads an unseen declaration" is closer to two tools than to twenty-four slots, and the statistics above treat twelve templated variants as independent.

Token volume is also collinear with corpus era. The arms that succeed are 2024 models whose pretraining and post-training corpora contain instruction-shaped and schema-shaped text; the 180B-to-350B arms are 2022-era models trained before such data was common. A model trained on 350B tokens of 2024-era data might sit on the other side of the interval. The base-versus-Instruct comparison removes post-training but not pretraining content, and with 8 of 24 in the base arm it would have needed 15 or more in the Instruct arm to detect a difference at all; "no separation" there is absence of evidence.

The masking observation is post-hoc, one-seed, and based on five discordant pairs. It is a reason to run a named confirmatory test, not a finding to build a system around.

The preregistration is a repository commit: the bands were committed 38 seconds before the first submission and the amendment 34 seconds before the successful one, and the commits were pushed days later. Nothing was seen in between, since the two failed jobs produced no model output, but there is no external timestamp, and a reader is entitled to treat that as weaker than a public registration.

Finally, 0/24 does not show that the model ignored every part of the prompt. The masked arm emitted the correct name 9 times rather than 4, but that post-hoc shift is also consistent with more copying and less recall. It did not produce one exact, executable held-out call, and masking substantially harmed trained-tool execution.

Where this leaves Drummer

The open-tool line stops here. I am not going to train a research model through an interval whose lower end has already produced zero; the 15.85B continuation, refit and tested, scored zero too.

The fixed-catalog result remains. A 542M model that reliably selects among known tools, asks when an argument is missing, and declines when no tool fits still has practical uses. It is simply a different system from one that can accept an arbitrary schema at runtime.

That difference was the result.

Back to the demo · Model card