An AI API can charge the same rate for every language and still cost more to use in some of them. The difference comes from how much text fits into a token.

In one published comparison, 1,012 English passages take 26,873 tokens. Their Tamil translations take 53,199 with the same tokenizer—almost twice as many. Hindi takes 42,276, Spanish 35,379, and Japanese 44,559.

The price per million tokens hasn’t changed. The number of tokens needed to carry the material has.

Compare the same material first

A fair comparison needs translations of the same content. Counting a thousand English words and a thousand Japanese characters would measure two different quantities of information.

The figures here come from Anri Lombard’s tokentax dataset, which tokenizes aligned passages from FLORES+. We checked its published totals for twelve languages and recalculated the ratios. We did not rerun the tokenizers or generate the translations ourselves.

Start with o200k_base, the encoding the dataset labels “GPT-5 / GPT-4o.” A tokenizer breaks text into pieces the model can process. A piece might be a common word, part of a word, punctuation, or something smaller; a token is not a language-independent unit of meaning.

Language Tokens for the 1,012 passages Relative to English
English 26,873 1.00×
Spanish 35,379 1.32×
German 35,224 1.31×
Russian 38,188 1.42×
Arabic 37,105 1.38×
Hindi 42,276 1.57×
Bengali 45,592 1.70×
Tamil 53,199 1.98×
Telugu 51,849 1.93×
Mandarin, simplified script 33,552 1.25×
Japanese 44,559 1.66×
Korean 39,566 1.47×

These are corpus totals. “Tamil uses 1.98 times as many tokens” means that the Tamil passages in this dataset, with this encoding, use that ratio. It is not a conversion constant for every Tamil prompt.

What does that do to a bill?

Use an illustrative input rate of $5 per million tokens. This is a fixed price assumption to isolate the effect of token counts, rather than a quote for every model using this encoding.

The calculation is simply:

input cost = input tokens × price per million ÷ 1,000,000

Normalize the dataset to 1,000 passages, and the English input costs about 13.3 cents. Hindi costs 20.9 cents, and Tamil 26.3 cents. Across one million passages of the same average length and composition, those amounts become approximately $132.77, $208.87, and $262.84.

The absolute numbers are small because these passages are short: the English average is about 26.6 tokens. A product processing longer documents would need its own sample. The ratios describe this sample’s variable text, so adding a large, fixed system prompt would also dilute their effect on the total input bill.

Then there is output. An application that generates long replies may spend more on output than input, and generated answers are not guaranteed to be equally verbose across languages. The measurements above contain translated source text, not model-generated responses. They establish an input-cost difference; they do not establish a whole-conversation cost multiplier.

The tokenizer can change the answer dramatically

The same dataset includes cl100k_base, an older OpenAI encoding. Its results show why a striking historical number needs a model and a date attached.

Language Older cl100k_base: tokens relative to English o200k_base: tokens relative to English
Hindi 4.76× 1.57×
Bengali 5.80× 1.70×
Tamil 7.64× 1.98×
Telugu 8.29× 1.93×

For Telugu, the actual count falls from 225,263 to 51,849 tokens, a reduction of about 77%. English changes much less, from 27,182 to 26,873.

An eightfold gap is therefore a real result for one encoding in this corpus. Repeating it as a general claim about today’s AI would miss a large improvement visible in the same data.

The result can change across model families too. With the dataset’s Qwen3.8 tokenizer, simplified Chinese takes 25,078 tokens against English’s 27,530—about 9% fewer. English is not always the shortest representation.

Tamil illustrates a different spread. Its token count relative to each tokenizer’s own English baseline is 1.98× with o200k_base, 2.59× with the Qwen tokenizer, and 1.47× with Gemma 4. These are tokenization comparisons. A lower count alone says nothing about answer accuracy, the model’s price per token, or whether it handles the task well.

Should you translate everything into English?

The token counts alone do not justify that step.

Consider a workflow that translates a user’s question into English, sends it to a model, then translates the answer back. It now has translation work on both sides. Those steps can add charges and delay, and their effect on the answer needs checking. This dataset did not test that workflow, so it cannot tell us whether the final cost or quality improves.

A useful comparison would price complete tasks: the native-language request and answer on one side, and all translation, generation, and retry costs on the other. Both sides would need to meet the same quality requirement.

The same distinction applies to consumer subscriptions. These token counts do not imply that a Tamil-speaking ChatGPT subscriber gets a higher monthly invoice, nor do they establish a particular reduction in message allowance. API token billing and a subscription’s limits are separate questions. OpenAI’s Plus documentation, for example, explicitly separates the subscription from API billing.

A usable estimate for a multilingual product

Start with actual material from each language your application serves. Keep the task and meaning comparable, then count it with the exact tokenizer for the intended model. Measure the model’s generated output separately and include the system prompt, tools, retries, and any cached input in the cost calculation.

Track tokens per completed task alongside quality. A character count can be convenient for storage limits or a text box; it is a weak substitute for the quantity on an API invoice.

For the published sample here, the answer is clear: language changes the amount of billable input, sometimes substantially. How much it changes an application’s bill depends on the tokenizer and the rest of the work that application performs.

Data and limits

This article uses the tokentax file at commit 0efd8729c9e50d69d751976d2473a83d4b157d83, checked October 18, 2026. It records 1,012 matched FLORES+ devtest segments per language, dataset revision e707e62e762a, encoded sentence by sentence without chat templates or added special tokens. The source does not record an explicit Unicode-normalization step.

The Qwen and Gemma entries identify Qwen/Qwen3.8-27B-FP8 and google/gemma-4-12b-it, but do not pin their tokenizer revisions; the exact tiktoken run version is also missing. That limits exact reproduction. The available aggregates cannot tell us the spread of differences between individual passages, and a translation corpus may behave differently from chat, code, or specialist documents.

Our twelve-language data extract retains the raw counts and provenance. The selected data is adapted from tokentax/FLORES+ under CC BY-SA 4.0; the calculation script reproduces the ratios and illustrative costs.