Token

What it is

In the context of Artificial Intelligence and Large Language Models (LLM), a token is the fundamental unit of measurement. Unlike humans, who read complete words, AI models break text down into smaller units called tokens in order to process them mathematically.

Imagine you are explaining to a computer system how to interpret a sentence. If the system had to learn every variation of every word (run, running, ran), the database would be infinite. By dividing words into subunits, AI can be much more efficient. A token can be an entire word (for short, common words like "house"), a part of a word (like the suffix "-ly"), or even just a punctuation mark or a space.

For an SME, understanding the concept of a token is crucial because it is the "currency" of the AI world. It is through tokens that providers (such as OpenAI or Anthropic) bill for the service and it is through them that a model's short-term memory capacity is measured.

How it works

The process of transforming text into tokens is called tokenization. When you send a request (prompt) to an AI, the following flow occurs:

1. Segmentation: The system analyzes your sentence and breaks it down. On average, for the English language, 1000 tokens are equivalent to about 750 words. However, other languages, by having more accents and inflections, may sometimes consume slightly more tokens to express the same idea.
2. Numerical Conversion: Each unique token has a numerical ID associated with it in a fixed dictionary of the model. The AI doesn't see letters; it sees sequences of numbers.
3. Processing: The model processes these numbers, predicts what the most likely next number (token) in the sequence will be, and converts it back into readable text for the user.

Tokenization efficiency varies between models. Newer models are more optimized for non-English languages, meaning they can represent text with fewer tokens than older models, making the operation cheaper and faster.

When to use

Although you do not "use" tokens directly (it is the model that uses them), you should keep this concept in mind in three critical situations in your business management:

1. Cost Planning: If you are building a customer support chatbot that consults 200-page technical manuals, the API cost will be calculated by the volume of tokens sent and received. Ignoring this can lead to billing surprises at the end of the month.
2. Context Management: All models have a maximum limit of tokens they can "read" at once (the so-called context window). If your document exceeds that limit, the AI will simply ignore the beginning of the text or lose the thread.
3. Performance Optimization: In applications that require fast responses, reducing the number of unnecessary tokens in the prompt (direct instructions without flourishes) makes the response faster and more economical.

Common errors

1. Confusing Tokens with Words: This is the most frequent error. Thinking that 1,000 tokens are 1,000 words can lead to an under-budgeting of 25% to 30% in operating costs.
2. Ignoring Punctuation and Spaces: Every comma, extra space, or line break counts as a token or part of one. Poorly formatted documents or those with many unnecessary spaces are more expensive to process.
3. Not considering output in billing: Many managers only look at the size of the question (input), but forget that the AI's response (output) is also charged, often at a higher price per token than the input.
4. Using outdated models for non-English languages: Very old models may be inefficient at tokenizing special characters, spending 2 or 3 tokens for something that modern models solve with just 1.

Practical example for an SME

A travel agency in Faro decides to automate the screening of complaint emails and quote requests using the GPT-4o API.

• Scenario A: The agency sends the full history of 50 emails exchanged with the client to give context to the AI. This totals 15,000 tokens. The cost per 1,000 tokens is low, but multiplied by hundreds of clients daily, the value scales.
• Scenario B: The agency uses a prior summarization technique, filtering only the key points of each email before sending it to the AI. The volume drops to 2,000 tokens.

By understanding what tokens are, the agency can reduce its operating costs by 85% while maintaining the same quality of response, simply by knowing that the AI is taxed by the "quantity of pieces" of text and not by the number of files.

Frequently Asked Questions

Q: Can I control how many tokens the model uses in the response? A: Yes, almost all interfaces and APIs allow you to set a maximum limit of output tokens (max_tokens) to avoid excessively long responses and unexpected costs.

Q: Do images also count as tokens? A: Yes, in multi-modal models (which process image and text), images are converted into an equivalent value in tokens based on their resolution and detail.

Q: Why does programming code use fewer tokens than literary text? A: Actually, code can use many tokens due to spaces and special characters, but since the structure is very repetitive, modern tokenizers are highly optimized to recognize common code patterns as a single unit.

• 01The word 'house' is generally 1 token, but 'unconstitutionally' can be divided into 4 or 5 tokens.
• 02A 128k token limit allows the AI to 'read' an entire book before answering your question.
• 03Reducing excess formalism in prompts helps save tokens and lower the monthly API bill.

Want to use Token in your company?

30 minutes, free, no commitment. We map where it fits.

Free AI diagnosis