Chunking

What it is

In the context of applied artificial intelligence, chunking is the act of dividing a long text into smaller pieces (chunks). If we imagine a company's database as an encyclopedia, chunking is the process of deciding whether we will index that information by chapters, pages, or paragraphs.

This process is vital because most large language models (LLMs) have a limit on immediate memory — the so-called context window. Furthermore, when a RAG (Retrieval-Augmented Generation) system tries to find an answer in a 100-page document, it cannot send the entire file to the model without losing precision or wasting unnecessary resources. Chunking ensures that the AI receives only the exact fragment containing the answer, maintaining the necessary context for the information to make sense.

How it works

Chunking is not just cutting text every 500 characters. For the result to be useful, the division must respect the semantic structure of the content. There are several approaches:

1. Character or token division: This is the simplest form, where the text is cut upon reaching a fixed limit. The risk is cutting a sentence or a word in half, losing meaning.
2. Recursive division: The system tries to divide the text by paragraphs; if the paragraph is too large, it divides by sentences; if the sentence is massive, it divides by words. It is the standard method for maintaining coherence.
3. Structural division: Uses the document's logic, such as headers (H1, H2), table cells, or sections of a PDF file, to ensure a topic is not separated from its explanation.
4. Overlap: To prevent context from being lost between two fragments, a small part of the end of the first chunk is usually repeated at the beginning of the second. This serves as a semantic 'glue'.

After division, each chunk is transformed into a numerical vector (embedding) and stored in a vector database, becoming ready to be searched.

When to use

Chunking is mandatory in almost all AI projects involving a company's own documents. It should be applied when:

• Implementing a RAG system: So that the customer support bot or legal assistant finds the exact clause without reading the entire contract every time.
• Dealing with technical documents: Machine manuals or parts catalogs where information is dense and granular.
• Working with cost limits: Sending less text (only the relevant chunks) to the OpenAI or Anthropic API drastically reduces the monthly bill.
• Precision is needed: When the answer to a question depends on a specific detail hidden on page 45 of a financial report.

Common errors

The most frequent error in SMEs is using an inadequate chunk size. If the chunk is too small (e.g., just one sentence), the AI loses context. If it is too large (e.g., 3 pages), the answer becomes vague and the system retrieves too much 'noise' that is irrelevant to the user's question.

Another error is ignoring the file structure. Dividing an Excel file or a price table in a purely textual way destroys the relationship between columns and rows, making the data unreadable for the AI. Finally, not using overlap is a common technical error: if the answer to a question starts at the end of one block and ends at the beginning of another, the system may fail to capture the complete information.

Practical example for an SME

Imagine a mold manufacturing company in Marinha Grande that has a 200-page procedure manual on machine maintenance and occupational safety. If a worker asks the AI assistant: 'What is the tightening torque for the bolt on injection molder X?', the system does not read the entire PDF.

With good chunking, the manual was previously divided into blocks of about 300 words, respecting section titles. The system identifies the chunk that talks specifically about 'Injection Molder X' and 'Head Maintenance'. It extracts that paragraph, sends it to the LLM, and responds in seconds: 'The torque is 50Nm'. Without chunking, the system would send too much information, take longer, and could confuse the torque of Injection Molder X with that of Machine Y described on another page.

Frequently Asked Questions

Q: What is the ideal chunk size? A: There is no magic number, but for administrative text, blocks between 500 to 1000 tokens with 10-15% overlap are usually a good starting point.

Q: Does chunking alter my original document? A: No. Chunking is just a way to organize information in the query database. Your original document remains intact on the server or in the cloud.

Q: Can I chunk images or tables? A: Yes, but it requires different techniques. Tables should be converted to formats like Markdown or JSON before being divided, and images require textual description (captioning) or multi-modal models.

Q: Why can't I just give the entire document to the AI if it has a large context window? A: Although models like Gemini or GPT-4o accept a lot of data, putting excessive and irrelevant information in the prompt causes 'lost in the middle' (the AI ignores central details) and increases operating costs unnecessarily.

• 01Dividing a service provision contract by individual clauses for fast legal analysis.
• 02Fragmenting a technical instruction manual into 800-character sections with 100-character overlap.
• 03Separating monthly meeting transcripts by agenda topics for easier searching.
• 04Indexing a catalog of 5000 products by dividing it into categories and individual technical sheets.

Want to use Chunking in your company?

30 minutes, free, no commitment. We map where it fits.

Free AI diagnosis