Quantization

What is it

In the traditional software world, we are used to applications growing as features increase. In Artificial Intelligence, the size of a model is determined by the number of parameters and the numerical precision of each one. Quantization is the engineering technique that allows these models to be compressed by converting data from a high-precision format (such as complex 32-bit decimal numbers) to simpler and shorter formats (such as 8-bit or 4-bit integers).

Imagine you are trying to share a cooking recipe over the phone. Instead of saying 'add 124.567 grams of flour', you just say 'add 125 grams'. You lost a tiny bit of precision, but the message was transmitted much faster and the final result — the cake — is virtually identical. In AI, Quantization does exactly this with the billions of calculations the model performs: it simplifies the numbers so the computer needs less memory and less energy to process them.

How it works

AI models, especially LLMs (Large Language Models), are composed of 'weights'. Originally, these weights are stored in a format called FP32 (Floating Point 32-bit). Each weight occupies 4 bytes of memory. A 7 billion parameter model (7B) in FP32 would occupy about 28GB of RAM, making it impossible to run on a common computer or an economical server.

Quantization maps these continuous values to a discrete set of values. If we move from 16 bits to 4 bits, we are reducing the model size by 4 times. The same 7B model now only occupies 5GB or 6GB of RAM.

There are two main moments to apply this technique:

1. Post-Training Quantization (PTQ): The model is already trained and we apply compression at the end. It is the most common and practical method for SMEs.
2. Quantization-Aware Training (QAT): The model is trained already 'knowing' it will be compressed. It is more accurate but much more expensive and time-consuming to execute.

When to use

Quantization is essential for any company that wants to move from the prototype phase to production with controlled costs. It should be used when:

• You want to reduce infrastructure costs: It allows running powerful models on servers with cheaper GPUs (or even just CPUs).
• You need low latency: Quantized models respond faster because mathematical calculations between integers are processed more efficiently by hardware.
• Local privacy (Edge AI): If your company needs to run AI inside office computers or on mobile devices without sending data to the cloud, Quantization is what allows the model to fit into the memory of those devices.
• Scalability: To support hundreds of simultaneous users without the cloud bill skyrocketing exponentially.

Common mistakes

The most frequent error is excessive quantization. If you compress a model from 16-bit to 1-bit or 2-bit, the model's 'stupidity' increases drastically. It starts to confuse concepts, lose grammatical coherence, or invent facts (hallucinations) more frequently. It is necessary to find the sweet spot, which is usually around 4-bit or 5-bit for modern language models.

Another mistake is ignoring the loss of nuances in technical domains. If your SME operates in a very specific area (e.g., Portuguese tax law or industrial molds), a highly quantized model may lose the ability to distinguish between very similar technical terms that a 'full' model would understand.

Finally, not testing the model after compression. You should never assume the quantized model will behave exactly like the original without performing validation tests (evals).

Practical example for an SME

Consider a legal consulting firm in Braga that wants to implement a system for analyzing internal contracts. If they use the Llama 3 model (8B parameters) in its original version, they would need a server with a high-end GPU (like an A100), which costs thousands of euros or hundreds of euros per month in cloud rental.

By applying Quantization to 4-bit (INT4), the technical team can run the same model on a Mac Studio or a common Linux server with a gaming graphics card (RTX 3060/4060).

The result: the company gets 95% of the original model's intelligence but with a 75% reduction in memory costs and double the response speed. This transforms a financially unfeasible project into a profitable internal tool that runs on the company's own servers, ensuring no contract leaves the local network.

Frequently Asked Questions

Q: Do I lose much quality using quantized models? A: It depends on the compression level. At 8-bit, the loss is imperceptible. At 4-bit, the precision loss is minimal (usually 1-3%) for general tasks, which largely compensates for the speed gain.

Q: Can I quantize my own data? A: Quantization applies to the weights of the AI model, not your customer data. However, the process can be applied to models that have been fine-tuned with your data.

Q: What is the difference between INT8 and FP16? A: FP16 uses 16 real bits (decimals), while INT8 uses 8 integer bits. INT8 is half the size and much faster, but less precise in complex mathematical calculations.

Q: Do I need special hardware to run quantized models? A: On the contrary. Quantization allows you to use less powerful hardware. However, some chip architectures (like Apple Silicon or the new NVIDIA ones) have specific instructions to further accelerate the calculation of quantized numbers.

• 01Reducing a 28GB model to 6GB to run on a local company server.
• 02Using the GGUF format to run AI on a salesperson's laptop without internet access.
• 03Implementing a customer service assistant that responds in milliseconds using INT8.
• 04Compressing a computer vision network to detect defects in factory parts using a Raspberry Pi.

Want to use Quantization in your company?

30 minutes, free, no commitment. We map where it fits.

Free AI diagnosis