LoRA

What it is

LoRA, or Low-Rank Adaptation, is an AI model engineering technique designed to solve a critical problem: the cost and complexity of adapting large models (like GPT-4 or Llama) to specific domains. Instead of retraining the entire model — which would require infrastructure costing tens of thousands of euros and weeks of work — LoRA acts as a lightweight 'additional layer' that fits over the original model.

Imagine you have a powerful truck engine. Instead of opening the engine and changing the pistons to make it perform better in the mountains, LoRA is equivalent to adding a small optimization chip that only changes the fuel injection instructions. The original engine remains intact, but the behavior changes drastically for the desired function.

How it works

To understand LoRA, we need to talk about parameter matrices. A modern AI model has billions of parameters (the 'weights' that determine how it processes information). During a traditional fine-tuning process, all these parameters would be altered.

In LoRA, the model's original weights are 'frozen' — they become immutable. Instead of trying to tinker with them, researchers discovered that the change needed to specialize a model typically has a 'low rank'. Mathematically, this means we don't need to touch every number in a giant matrix; we can decompose that change into two much smaller matrices.

During training, only these tiny matrices are updated. When the model is used to generate a response (inference), the LoRA matrices are mathematically added to the original weights in an instantaneous process. The result is an extremely small final file (a mere 50MB to 200MB, compared to the 30GB or 100GB of a full model) that contains all the 'specialized knowledge' required for the SME.

When to use

LoRA is the ideal tool when your company has specific data that you don't want or cannot send to external providers like OpenAI, or when you need a very rigid behavior that simple 'prompt engineering' cannot guarantee.

Use LoRA when:

1. Tone and Brand Customization: You want the AI to write exactly with the style, vocabulary, and standards of your company, without fail.
2. Strict Technical Knowledge: The model needs to understand industry-specific technical terminology (e.g., industrial molds, Portuguese tax legislation) that was not sufficiently covered in base training.
3. Hardware Limitations: Your company wants to run models locally on its own servers or in the cloud with modest GPUs. LoRA allows training 70-billion-parameter models on hardware that would otherwise be impossible.
4. Task Multiplicity: If you have ten different clients and each needs a different writing style, you can have a single 'heavy' base model and ten small LoRA files that you swap in milliseconds depending on the client being served.

Common errors

The most frequent mistake is confusing LoRA with RAG (Retrieval-Augmented Generation). LoRA is used to teach style, format, and logic, not to teach mutable facts. If you want the AI to know your stock price today, use RAG. If you want the AI to learn how to format audit reports following specific ISO standards of your factory, use LoRA.

Another error is 'Overfitting'. Since LoRA focuses on a few variables, if the training dataset is too small or repetitive, the model may start 'memorizing' answers instead of learning the pattern. This makes the AI rigid and unable to handle questions slightly different from those in the training.

Finally, there is data quality neglect. Since LoRA is very efficient at extracting patterns, if you give it 100 examples of poorly written customer support emails with grammatical errors, the model will learn to write exactly with those errors, systematically replicating the poor quality.

Practical example for an SME

Imagine a legal consultancy in Lisbon specialized in real estate law. They use an open-source model (like Llama 3) to help draft contract drafts.

However, the base model writes in a very generic Portuguese and sometimes uses Brazilian legal terms. The SME then decides to create a LoRA. They gather 500 of their best contracts drafted over the last 5 years, anonymize the data, and train a LoRA for a few hours on a GPU rented for a few euros.

The result? Now, whenever the lawyer asks for a draft, the model uses the LoRA matrices to apply the specific clause structure of that firm, respecting updated Portuguese legal terminology and the formal tone preferred by the partners. The base model was not changed, but the LoRA 'filter' transformed it into a virtual intern who already knows the house rules.

Frequently asked questions

Q: Do I need senior developers to create a LoRA? A: Currently, there are already 'low-code' tools and platforms that automate LoRA training. The essential part is not knowing the math behind the matrices, but having a clean, high-quality dataset.

Q: Is it better to use a LoRA or do full Fine-Tuning? A: For 99% of SMEs, LoRA is better. Full fine-tuning is prohibitively expensive, requires hundreds of times more memory, and often suffers from 'catastrophic forgetting', where the model forgets how to speak normally because it was altered too much.

Q: Can I use several LoRAs at the same time? A: Yes. One of the great advantages is modularity. You can load one LoRA for 'writing style' and another for 'technical knowledge' on top of the same base model simultaneously.

Q: Does LoRA replace RAG? A: No. They are complementary. LoRA trains the 'brain' to think in a certain way; RAG gives it the 'books' to consult facts in real time.

• 01Adjusting a Llama 3 model to write business proposals with the specific tone of voice of a marketing agency.
• 02Training a lightweight layer so an AI recognizes and uses technical jargon from the Portuguese glass industry.
• 03Specializing an image generation model to create furniture prototypes following a national brand's catalog.
• 04Creating an assistant that automatically formats JSON files following the exact structure of the company's ERP.

Quer usar LoRA na sua empresa?

30 minutos, gratuito, sem compromisso. Dizemos onde encaixa.

Diagnóstico IA gratuito