The Tokenizer Nobody Read: Finding 50 MB in a 28-Layer Rust LLM Proxy

The data plane of the mesh is a Rust proxy: a 28-stage middleware pipeline that every model call traverses — routing, failover, budgets, guardrails, caching, audit. Two headline numbers described its performance for months: 11 MB resident memory, 1.7 ms median latency. Both were wrong, and not by a little. This is the story of a memory-profiling program that started by refusing to trust its own baselines, found a 36.7 MB tokenizer table that nothing consumed, removed it with a guard of about a dozen lines, and watched the production process settle from roughly 60 MB of retained memory to roughly 11 MB. It is also a story about the measurements that said no — because in performance work, the negative results are the ones that keep you honest.

Rule one: check the provenance before you optimize anything

The first act of the program was not a flamegraph. It was reading the fine print under the two numbers everyone quoted.

The "11 MB RSS" figure came from ps on a macOS laptop, during a soak at 18.3 requests per second against a mock provider with no TLS — a different operating system, a different allocator, and an environment table that recorded a thin-LTO build while the shipped profile is fat-LTO. The "1.7 ms p50" came from a load test where the semantic cache was enabled at high similarity and the request bodies were near-identical — a cache-hit benchmark wearing a proxy benchmark's name. The giveaway was sitting in the same run's own table: p95 was about 52 ms, which is the mock upstream's mandatory 50 ms hop. The median measured the cache. The tail measured the mock.

Neither number was fabricated. Each was a real measurement of the wrong thing, promoted to a headline without its conditions. So the program's first deliverable was a rule: every number carries a provenance class, stated once and attached everywhere.

• [container] — a Linux benchmark container. Attribution-grade: ratios, slopes, deltas, and call-tree shapes transfer. Absolute megabytes and milliseconds do not.
• [jetson] — a Jetson Orin Nano at 15 W, the deployed-edge platform. Platform-grade absolutes for edge conditions.
• [prod] — the live production VPS, read-only observation of the real deployment.

Everything below carries one of those tags. Where a measurement was confounded, the label stays on it — several runs in the committed evidence are marked saturation-confounded or repeatability-FAIL, kept as they ran rather than re-run until they looked clean.

The measurement that mattered: warm is 4x cold, and it never comes back

The centerpiece measurement of the reconnaissance phase was almost embarrassingly simple: memory at cold idle, then memory after exactly 100 small requests, then memory after sustained load, then memory after the load stops.

Warm idle was four times cold idle after one hundred requests. And the post-load number did not recede — it sat at or above every mid-run sample. Live request buffers die with their requests; pages retained by the allocator do not. That persistence was the discriminator: the memory story was retention, not throughput.

The obvious suspect was the request path's double body buffer (the pipeline buffers each body twice, by design — one stage needs unmasked content). So we made the hypothesis falsifiable: if live double-buffered bodies explained the growth, the 1 KB-to-32 KB delta should be about 2 x 31 KB x 150 in-flight, around 9 MB. Measured: 67 MB across unequal VU counts, 123 MB in the VU-matched comparison — both runs the evidence labels saturation-confounded, and keeps labeled. Off by 7-14x, and the growth outlived every buffer. Refuted. The copy cost is real and worth fixing on hygiene grounds, but it is not where the memory went.

Attribution: the profiler names a surprising suspect

Heap profiling (heaptrack) and CPU profiling (perf) against the loaded proxy [container] produced two matching answers.

On the heap side, the single largest retained site was the tokenizer: construction of an o200k_base BPE encoder — a 36.73 MB table, out of 41.42 MB total retained heap at the peak. That one allocation was 60.4 percent of the entire cold-to-warm step, against a next-largest named site of 1.4 MB.

On the CPU side, the top self-cost frame was BPE encoding at 25.31 percent — with a caveat the committed evidence states in bold, because the first profile was taken under an accidental stress test: the load generator offered far more concurrency than a default per-provider bulkhead would admit, so 97.2 percent of offered requests were being rejected. The 25 percent share was measured over a mostly-rejected mix, and the tokenizer was running even for requests the proxy went on to reject. On a later clean profile, the tokenizer stage's cost resolved to 17 percent of per-request latency at 1 KB bodies — the confounded number was directionally right and quantitatively wrong, which is exactly why the label stayed attached.

The tokenizer nobody read

Here is what the code was actually doing, and every clause is load-bearing.

The proxy's budget stage estimates each request's token count before forwarding, so that spend limits can be enforced pre-flight. To do that it builds a BPE tokenizer — the 36.73 MB table — lazily, on the first request rather than at startup, and then runs an encode over every request body. The estimate feeds exactly two consumers: a per-workflow budget manager, and the tenant budget hierarchy.

Both consumers are off by default. Budgets ship disabled. There was no gate above the estimate.

So on a default configuration, the proxy built a 36.73 MB table on the request path and BPE-encoded every body — the largest single stage cost in the 28-stage pipeline, by 4x at 1 KB bodies and 35x at 32 KB — to produce a number that the finalizer provably never read. We traced every reader of the estimate through the tree to confirm it: with both consumers disabled, the value is dropped on the floor, always.

The fix is the least dramatic diff of the program: compute the estimate if and only if a consumer will read it. One middleware file changed. The guard itself is about a dozen lines; the full change, tests included, is +195/-10 in that one file. The estimate's carrier keeps its presence semantics — downstream code sees the same shape, with a zero inside it when nothing consumed it — and the consuming paths are byte-for-byte unchanged, which the budgets-enabled control run confirmed by reproducing the pre-fix memory step to within 0.3 percent.

What the fix delivered [container]

Two details in that table deserve their own sentences.

First, the memory saving was 54.7 MB — larger than the 36.7 MB table itself. Removing a scattered 37 MB allocation also removes the allocator arenas and transparent-hugepage padding that grew around it; the isolation report had flagged its 60.4 percent attribution as a floor, and this is that floor being cashed.

Second, the CPU forecast missed by 2.2x, and the correction is a lesson worth exporting. The isolation had divided a wall-clock per-stage latency gap by a CPU-time denominator. A paired same-session probe measured the CPU fraction of that stage gap at 37.7 percent (1 KB) and 48.3 percent (32 KB); applying the correction predicts -6.4 and -29.9 percent against the measured -6.9 and -29.0. The standing rule the program adopted: a stage-latency gap is a latency budget, not a CPU budget. If you quote one as a CPU share, measure the conversion factor first.

The production witness — including the confound

The deployment that motivated the memory work is a container with a 512 MB limit on the production VPS, where the pre-fix process had been observed carrying about 60 MB of anonymous memory — most of it swapped out cold, which is what a memory-pressured kernel does with a tokenizer table nobody reads. The proxy there fronts the platform's model traffic, including the live consumer product (@PersonalMeshBot on Telegram).

Same host, same container limits, near-identical uptime, old binary versus fixed binary [prod]:

And now the caveat, because the piece's credibility is the caveat: the new process served zero completions requests during that soak window. The comparison confounds two explanations — the table is absent because the fix gates it, and the table is absent because nothing asked for it. What the soak witnesses cleanly is that the fixed binary holds a flat ~11 MB on the production host for almost three days with no growth, where the same host previously carried ~60 MB over the same duration. The mechanism claim rests on the container evidence — the budgets-on control that reproduces the old behavior on demand — not on the soak. The traffic-path witness completes on the first organic burst of production traffic, and the evidence file says exactly that.

The A/B that said no: keep glibc

Before the tokenizer attribution landed, the leading theory for the warm-idle plateau was the allocator — glibc arenas retaining freed pages. So the program ran the full experiment: five arms, three runs each, one operating point [container]. glibc, glibc with arena_max=2, jemalloc, tuned jemalloc, mimalloc.

Both replacement allocators moved every memory column the wrong way and every CPU column the right way. That trade is the finding — and for a memory-bound 512 MB deployment, it is a losing one. The recommendation the evidence supports: keep glibc as the default, carry arena_max=2 as an operator knob rather than a dependency.

The deeper result is the one that redirected the program: the ~61 MB warm-up step survived all five arms. Three allocators with three unrelated retention policies all faulted in the same order of magnitude after the same 100 requests. That is what a workload property looks like, not an allocator property — and it pointed the search back at the workload, where the tokenizer was waiting.

One more discipline note, because it is the part most benchmark posts quietly skip: three of the five triples failed the harness's own repeatability gate — the replacement allocators purge on decay timers asynchronous to the sampling schedule, so their memory readings genuinely spread. Those failures are recorded in the committed evidence as failures. The headline deltas above are 8-18x the worst spread involved; the deltas that were inside the noise are labeled inside the noise. No triple was re-run to green.

The trigger that fired after the fix

The pipeline's dispatch machinery — the boxed futures and cloned route shells that 28 stages of tower-style layering produce — was the pre-program favorite for the latency budget. The measurements demoted it: static arithmetic put the boxing cost at single-digit microseconds per request, and the first profile showed the machinery at about 6.3 percent of self cost while tokenization and memcpy dominated. Rather than argue, the program ratified a rule: enumerate the dispatch frames, and if they measure above 5 percent of self cost on a clean-load profile, the de-boxing migration is approved; below, it stays deferred.

Then the tokenizer fix shrank the denominator. On the clean-load edge profile [jetson] — 78,044 samples, zero lost, 90,000 of 90,000 requests returning 200 — the enumerated frame set measured 7.11 percent. The trigger fired, and fired because of the earlier win: removing the tokenizer's share of CPU is what pushed the machinery's share over the line. The fix that demoted the migration is what armed it. The migration proceeded with the threshold rule as its warrant — approved by a measurement, not a mood — and what it measured is the next piece in this series.

A finding inside that profile worth passing on: the Box::pin frames every Rust performance thread worries about measured 0.00 percent — fully inlined by the optimized build, their cost folded into callers and the allocator. If you go hunting for boxing overhead by symbol name, you will conclude it is free. Enumerate the surrounding machinery instead.

Where the platform honestly stands

The re-baseline on deployed-edge conditions [jetson] — 15 W power mode, default governor, the configuration the live node actually runs — replaced the debunked numbers with these:

• Cold idle: 19.0 MB RSS / 17.6 MB PSS. Warm idle: 24.5 MB RSS. The warm step, post-fix, is +5.5 MB — down from the ~61 MB the old binary would have paid.
• Added latency: +1.91 ms p50 over a direct-to-mock baseline at 500 req/s, measured open-loop with coordinated-omission correction on both sides of the subtraction.
• The original aspirational targets — 6-8 MB idle, sub-millisecond added p50 — are not met, and the evidence says so in those words. What remains in cold idle is mostly the binary's mapped text and read-only data, including an embedded 3.6 MB tokenizer vocabulary; the remaining path to single-digit megabytes is binary-layout work, not runtime tuning. Revising the targets is a strategy call; the numbers above are the evidence base for it.

Two smaller levers from the same program, each with its honest label. Per-request logging at the default level was writing 3,273 bytes and 3 lines per request — 98.2 MB per minute-long window at 500 req/s, or 5.89 GB per hour; demoting three per-request events to debug took it to zero bytes, and the evidence file explicitly declines to claim a latency win, because the same-session control showed the harness cannot resolve one at this rate. And the inbound listener was serving with Nagle's algorithm live while the upstream client had it disabled; the one-line fix was justified by the asymmetry and the traffic shape, not by a histogram — the committed write-up says that too.

What travels

If you run an LLM proxy, check whether you are tokenizing for a budget nobody enabled. The shape of this bug — an expensive estimate computed unconditionally, feeding a feature that defaults off — is common enough that we found it load-bearing in a codebase with 3,000-plus tests and review on every change. Grep for your estimator's call site. Look above it for the gate. If the gate is below, you have the same bug.

The rest of what transfers is method, not code:

1. Baselines have provenance. Audit it first. Both of ours dissolved on contact with their own conditions.
2. Tag every number with its environment class, and let attribution-grade and platform-grade numbers do different jobs.
3. Make hypotheses falsifiable before profiling. The double-buffer theory died in one arithmetic line, which saved a week.
4. Keep the confounded runs, labeled. A repeatability failure you kept is evidence; one you re-ran to green is fiction.
5. A stage-latency gap is not a CPU share. Measure the conversion.
6. Write the decision rule down before the measurement. The de-boxing threshold turned a taste argument into a one-line verdict — in both directions.

The mesh's coordination layer gets the attention, but the data plane is where these disciplines compound: the same measurement-first approach runs through the platform's provider key rotation and cost-aware routing, applied here to the proxy that carries them. More of the program's evidence will become posts as the remaining gates close.

If you are building agent infrastructure and this kind of engineering matters to you, subscribe — the next piece in this series covers what the dispatch-machinery migration measured before and after.