Every AI model you pay for counts your text in tokens. The price sheet is per million tokens, the context window (the most text a model can work with in one call) is a number of tokens, and the chunk size in your document search (the size of the pieces your documents are cut into) is set in tokens. None of those is a fixed amount of text. A token is whatever the model's tokenizer, a separate program its maker trained once before the model itself, decided it is. Nobody in your company chose it, and it sets what your budget buys.
What an LLM token is, and why you are billed in it.
A large language model (LLM) does not read letters or words. It reads a sequence of numbered symbols drawn from a fixed vocabulary, and something has to turn your text into those numbers first. That something is the tokenizer. It is trained separately from the model, on its own body of text, with its own method, and in most models today that method is byte-pair encoding (BPE). The model's maker picks the tokenizer early, freezes it, and trains the model on top of it.
In practice, a token is a short piece of text: often a whole common word, sometimes a fragment of a longer or rarer word, a few digits or a punctuation mark. Once the tokenizer is frozen, every number an operator manages is counted in its output.
How much text is a token? It depends on the tokenizer and on the text. On the training data of the paper this note draws on, a token averaged about 4.5 bytes (in plain unaccented text, one byte is one character), which the authors use as a working constant for that data.[1] It is an average for one mixture, not a law. Code, tables, names, numbers and other languages all tokenize at their own rates, which is why a budget written in tokens does not transfer cleanly from one kind of content to another.
The price sheet shows one of the two numbers.
The practical consequence lands when you compare vendors. A model with a lower price per million tokens is only cheaper for you if its tokenizer cuts your text into a similar number of tokens. If it cuts the same page into more pieces, part of the discount disappears before the first call, and the price sheet will never show it. Our note on rerunning your model math makes the case for recomputing costs when prices move; the tokenizer is one more input to that recomputation, and an easy one to leave out.
Why French is the open question.
Byte-pair encoding builds its vocabulary from the text the tokenizer was trained on: it starts from single characters and repeatedly merges the pairs that occur most often, so strings that are common in that text end up as single tokens and rarer strings are spelled out in several pieces. Which strings count as common depends entirely on what the training text contained.
That is why this note exists for a Quebec business. If a tokenizer's training text held less French than English, a French page will tend to break into more tokens than its English twin, and a token budget sized on English content will buy less French text than it appears to. The direction follows from the mechanism. The size of the gap does not.
Measure it on your own content.
You do not need us, or anyone, to publish a ratio. The model you already pay for will tell you, one call at a time: API responses report how many input and output tokens each call used. A small sample is enough to replace a rule of thumb with your own number.
- Pick a sample of real content in each language you serve.
Use the pages, emails and documents you actually send to the model, not test sentences. If you publish both languages, take the same pages in English and in French, so the content is held constant and only the language changes.
- Count tokens with the model you use.
Send each item through the same model with the same short instruction, and record the input token count the response reports. Then send the instruction alone and subtract its count, so your instructions are not counted as content. To measure output, ask for the same answer in each language and record the output token count. Do not borrow a count from another vendor's tokenizer: it measures a different unit.
- Divide by something you control.
Keep two numbers per language. Tokens per page, measured on the same pages in both languages, tells you what serving that content costs. Tokens per thousand characters turns a document's length into a context or chunk budget before you send it. A monthly forecast is then pages per month × tokens per page × price per token, worked out separately for input and output. Those numbers, not a conversion you read online, are what your context budgets, chunk sizes and cost forecasts should use.
target: tokens per page and per thousand characters, per language, per model - Redo it when you change models.
A new model family usually brings a new tokenizer, and a new tokenizer gives a different count for the same page. These numbers belong to the pair: your content and the vendor's tokenizer.
If you run a document search, the same measurement applies to your chunks. A 512-token chunk holds a different amount of French than English, so the chunk size you tuned on one language is not automatically right for the other. Our chunking note covers how to tune it; tune it per language if your corpus is bilingual.
Token counts do not travel between models.
Because the tokenizer is a separate, frozen stage, it also decides which models can work together, often more than their quality does. Two techniques show it.
Distillation, training a small model to imitate a large one, normally hands the small model the large model's prediction over its vocabulary, so the two have to share that vocabulary. Breaking that assumption takes extra machinery, which is what the paper we draw on sets out to build.[1]
Speculative decoding, a common way to make a large model answer faster, has a small model draft a few tokens ahead and the large model check them. The explainer we cite puts the constraint plainly:
Most businesses never touch either technique. The version that reaches them is simpler: every limit you wrote in tokens is tied to one tokenizer. Switch vendors or model families and the chunk sizes, context budgets, per-call caps and monthly token quotas all need re-measuring, because the same documents now produce different counts. Treat a model change as a change of unit, not just a change of price.
Vocabulary size is a bill too.
A tokenizer's vocabulary size is paid for in parameters and in storage, and the paper we draw on puts plain arithmetic on both.[1] This section matters most to teams that train or distill models; for everyone else it explains why vocabulary choices are made by the people who pay for them.
≈ 100K vocabulary
≈ 260 symbols
Two things in the figure are arithmetic rather than results, and that is what makes them worth repeating. With about 1.28 billion parameters in the model's layers, a vocabulary of about 100,000 tokens adds roughly half a billion more, just to turn tokens in and out. And storing a teacher model's full predictions for offline distillation is impractical at that vocabulary size, which is why teams truncate them to the top few hundred; a byte vocabulary needs no truncation at all.
Models that skip the tokenizer and read raw bytes are an active research direction. The evidence on them is not ready for a budget: the paper's headline comparisons are asymptotic, the limits of fitted curves rather than scores of trained models, and the authors state that their byte models "remain significantly expensive at inference time", because they emit several symbols for every token a token model emits.[1] For now, they change nothing in a budget.
The paper's most useful lesson for an operator is a negative one. Its byte models beat the token model on validation loss, the curve researchers watch during training, while scoring worse on the downstream benchmarks.[1] A cheap, continuous proxy metric stopped tracking quality as soon as the models being compared were tokenized differently. The same failure appears whenever a convenient score is compared across things that differ in structure, which is the argument of our note on AI judges: check the proxy against the real outcome before you trust it across a boundary.
What this does not claim.
No French-to-English ratio. The mechanism says French may cost more tokens on tokenizers trained mostly on English. We have not measured how much, on which models, or for what content. Measuring it on our own French-language corpus is planned; until then, the number the method above gives you is the only one worth trusting.
One academic source, not verified by us. The parameter and storage figures come from one preprint whose numbers are the authors'. They are arithmetic on vocabulary sizes, which is why we use them, and we do not repeat its asymptotic accuracy headlines.
We train nothing. Acceleratech builds on hosted and open-weight models; we do not train or distill them. The vocabulary section explains the economics of the people who do, because their choices become your unit.
If your team is budgeting AI work in two languages, or comparing vendors on price, the contact form is the fastest way in. Tell us what you run, and we will send back a written read on where your token costs actually come from, free.