BM25

What it is

BM25 (Best Matching 25) is the evolution of the classic TF-IDF algorithm and remains the gold standard for so-called "lexical search" or keyword-based search. Unlike more recent vector-based Artificial Intelligence models (which attempt to understand meaning), BM25 focuses on exact term matching. In the context of an SME organizing its knowledge database, BM25 is the engine that ensures that when searching for "electricity bill", documents containing exactly those words appear at the top of the list.

Despite being decades old, it is extraordinarily robust. It is the basis for systems like Elasticsearch and Solr, and continues to be an indispensable component in modern RAG (Retrieval-Augmented Generation) systems, as it can capture specific details (such as serial numbers or product references) that complex language models sometimes ignore.

How it works

BM25 works through a mathematical formula that evaluates the relevance of a document for a specific query based on three fundamental pillars:

1. Term Frequency (TF): The more times a word appears in a document, the higher its score. However, BM25 applies what we call "saturation". This means the difference between a word appearing 1 time or 2 times is highly valued, but the difference between appearing 100 or 101 times is almost irrelevant. This prevents documents that repeat the same word over and over (spam) from dominating the results.

2. Inverse Document Frequency (IDF): The algorithm penalizes words that appear in almost all documents in the database (such as "the", "of", "that") and gives much more weight to rare and distinctive words (such as "transmutes" or "SKU-990").

3. Document Length Normalization: Longer documents naturally have more words, which increases the probability of containing search terms by pure chance. BM25 adjusts the score so that a 500-page manual does not unfairly beat a 2-page quick guide just by having more text volume.

When to use

For an SME, BM25 should be the first choice, or at least a central component, in several practical scenarios:

• Product Catalogs: If a customer searches for the exact reference "XYZ-123", BM25 is the most reliable method to find that product. Meaning-based AI models (embeddings) might incorrectly suggest similar products but with different references.
• Document Management Systems (DMS): For searching contracts by customer name, VAT number, or address, lexical precision is superior to semantic precision.
• Internal Search Engines: When speed is critical and hardware is limited. BM25 is extremely computationally lightweight compared to Deep Learning models.
• Hybrid Search: Currently, the best practice is to combine BM25 with vector search. BM25 ensures exact words are found, while AI handles synonyms and context.

Common errors

1. Ignoring Hybrid Search: Many companies abandon BM25 entirely to use only vector databases. The result? The system stops finding exact technical terms or product codes, frustrating users. BM25 and AI should work together.
2. Lack of Text Processing (Stemming): A common error is not configuring BM25 to understand that "cars" and "car" have the same root. It is essential to apply an analyzer that removes suffixes for the search to be effective.
3. Not configuring Stopwords: If common language words are not filtered, the search engine loses efficiency, although BM25 (via IDF) already attempts to mitigate this issue.
4. Relying solely on BM25 for synonyms: BM25 is blind to meanings. If you search for "spending" and the document says "expenses", BM25 will not find it. This is where it fails in isolation.

Practical example for an SME

Imagine a distribution company for electrical material with 50,000 catalog references. A maintenance technician is in the field and needs to find the manual for a specific circuit breaker. He searches for "Siemens 5SY41 Circuit Breaker".

• Without BM25 (Simple database search): The system might return a thousand results containing "Siemens" or "Circuit Breaker", without relevance ordering, forcing the technician to search manually.
• With BM25: The system identifies that "5SY41" is the rarest and most important term (high IDF). It immediately puts the manuals containing that exact reference at the top, even if the document is short. Term saturation ensures that a general catalog mentioning the word "circuit breaker" 500 times does not appear before the specific manual the technician needs.
• With Hybrid Search (BM25 + RAG): The system finds the manual (via BM25) and the AI chatbot reads the content to answer: "For that circuit breaker, the recommended tightening torque is 2.5 Nm".

Frequently Asked Questions

Q: Is BM25 better than vector search (Embeddings)?
A: It's neither better nor worse, it's different. BM25 is excellent for exact words and IDs. Vector search is excellent for concepts and synonyms. The ideal for an SME is to use both in hybrid mode.

Q: Do I need powerful servers to run BM25?
A: On the contrary. It is an extremely efficient algorithm that runs on standard hardware with millisecond response times, even with millions of documents.

Q: Does BM25 work well in different languages?
A: Yes, as long as the search engine (such as Elasticsearch) is configured with an appropriate language analyzer to handle word inflections.

Q: Can I use BM25 in a RAG system?
A: You should. Integrating BM25 into the RAG 'retrieval' step drastically reduces hallucinations by ensuring the AI receives technically correct documents based on precise terms.

• 01Retrieving a specific contract for client 'Sonae' in a database with thousands of PDF files.
• 02Filtering parts in an e-commerce catalog using the manufacturer reference (e.g., 'REF-9920-X').
• 03Ranking results of an internal FAQ so that articles with the exact terms of the question appear first.
• 04Lexical component of a RAG system to ensure technical terms are not ignored by the AI.

Want to use BM25 in your company?

30 minutes, free, no commitment. We map where it fits.

Free AI diagnosis