# LoRA (Low Rank Adaptation) Fine-Tuning for LLMs #71 Oct 6, 2025 Training https://www.aibutsimple.com/p/lora-low-rank-adaptation-fine-tuning-for-llms Large Language Models (LLMs) have proven incredibly useful for a variety of applications in our day-to-day lives. But to create and train a new model from scratch becomes very expensive as models scale in size and often requires supercomputers to do so. To tackle this, rather than build a new model every time we’re faced with a new problem, we can tweak an existing one. This technique is called fine-tuning, and it adapts a model to new tasks at a fraction of the cost of full training. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/2a258d7d-2aa2-4f2f-8682-b049f3ad86c7/FT_Example.png) Fine-tuning an existing model allows us to change only certain features to our necessity. Although fine-tuning is a comparatively more efficient way to make practical models, it can also become far too computationally expensive when dealing with large-scale models that house hundreds of billions of parameters. This leads us to Low-Rank Adaptation (LoRA), a popular technique that makes fine-tuning even more efficient. It was introduced in a 2021 paper titled “[LoRA: Low-Rank Adaptation of Large Language Models](https://arxiv.org/abs/2106.09685?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=lora-low-rank-adaptation-fine-tuning-for-llms).” --- ### Find out why 100K+ engineers read The Code twice a week. [![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/a15346a7-b9a1-4213-83b8-93791f7042b3/Get_curated_tech_news__top_research_papers___resources_in_2_short_weekly_emails_-_V6.jpg)](https://codenewsletter.ai/forms/14166360-de71-46c4-8722-878d417fab5c) That engineer who always knows what's next? This is their secret. Here's how you can get ahead too: - Sign up for The Code - tech newsletter read by 100K+ engineers - Get latest tech news, top research papers & resources - Become 10X more valuable [Join 100k+ engineers](https://codenewsletter.ai/forms/14166360-de71-46c4-8722-878d417fab5c) --- Not only does LoRA adapt an already existing model, it saves on memory by ignoring a majority of its parameters while doing so. Instead of updating every weight in the network, LoRA introduces small adapter layers to the original model. During training, only these new adapter layers are updated, whilst the rest of the model remains untouched. For instance, take a standard neural network. Standard fine-tuning will update all nodes, while LoRA only updates a fraction of the nodes. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/20709d7d-0c1d-45cb-8ab4-6980cbc52587/LoRA__2_.png) The diagram above highlights which nodes are updated in standard fine-tuning versus a LoRA implementation below. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/e2da81b5-2d8c-4793-bb36-a12b5a0cef9a/LoRA__3_.png) The top neural network is majorly unchanged but has a new additional layer right at the end. This is the effective implementation of LoRA, where most of a network remains unchanged, and only minor and specific adaptations are required to adapt the purpose of the network for something else. ### LoRA For LLMs Inside an LLM, computation happens in large-dimension matrices, which transform text representations into meaningful patterns. Now, instead of retraining a large-dimension matrix during fine-tuning, LoRA adds two smaller latent matrices, A and B ([low-rank decompositions](https://arxiv.org/pdf/2106.09685?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=lora-low-rank-adaptation-fine-tuning-for-llms)), that capture only the most important variations needed to tune the model. This allows the model to specialize in new domains, such as medical or legal language, without the full cost of retraining. It also makes it possible to store and swap multiple fine-tuned adapters with minimal overhead, meaning one base model can power a wide range of applications simply by loading the right adaptive layers. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/300e3be0-d0d1-4afd-80f5-582a056ef404/LoRA_Matrices.png) The matrix equation above shows how LoRA adapts larger weights within an LLM. W represents the original weight matrix, which remains frozen during training. Instead of retraining W, LoRA introduces two small adapter matrices, A and B. The product of these two smaller matrices creates a compact adjustment matrix ΔW, which represents the learned change needed to adapt the model for a new task. This adjustment is added back to the initial W, resulting in an adapted weight matrix. - The use of A and B can be inserted within specific parts of the model, especially linear layers in transformer-based LLMs. This makes up an adapter layer. Due to the fact that only adapters are trained, the number of parameters that need to be updated drops drastically, sometimes even a thousand-fold reduction. This reduces both computational costs and the amount of memory required, allowing fine-tuning to happen on a single GPU rather than a larger cluster. ### LoRA For Industry Use One major benefit of LoRA is modularity. Instead of training an entirely new model for each new task, you can swap in different adapters to customize the model for that specific task. This efficiency also transforms how organizations manage models. Instead of maintaining several massive, fine-tuned models, companies can hold a single base model with multiple adapters. Each adapter specializes the model for a different client, task, or language. Swapping adapters is as simple as loading a small file, saving enormous resources. - Think of it as the difference between carrying multiple textbooks versus one textbook with color-coded sticky notes. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/78048391-3d12-4944-80ab-0454b8ea204e/Untitled_design__4_.png) It is helpful to place LoRA in context with other fine-tuning methods. Full fine-tuning retrains every parameter, which is powerful but expensive. Adapter tuning adds extra neural layers on top of the model, but these are often heavier than LoRA’s A and B matrices, which can be added to existing layers. Prompt tuning and prefix tuning instead add learned tokens at the input level, which are lightweight but less effective for complex problems. LoRA strikes a balance; it is lightweight like prompt methods but nearly as powerful as full fine-tuning. LoRA’s versatility makes it valuable across a wide range of industries. In healthcare, providers can fine-tune base models to better interpret medical records or clinical notes without retraining an entire system. In the legal sector, firms can train adapters to understand case law, contracts, and courtroom terminology, producing outputs that align with domain-specific language. Financial institutions use LoRA to adapt models for tasks like compliance reporting, fraud detection, or risk analysis, where precision and regulatory language matter. In education, specialized adapters can turn a single base model into multiple tutors—one trained on mathematics, another on history, and another on language learning. Even in creative industries, LoRA enables adapters that mimic particular authors, generate scripts in specific genres, or bring fictional characters to life in dialogue systems. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/cb38f317-70d7-4cc0-9950-e3bd347661be/Untitled_design__5_.png) Smaller-scale uses are just as compelling. Even individuals may one day own personalized adapters that reflect their writing style, preferences, and routines. Despite its strengths, LoRA is not without limitations. Since it builds on a frozen base model, LoRA cannot introduce entirely new capabilities that the original model does not already contain. Its performance may also fall slightly short of full fine-tuning when dealing with highly complex or specialized tasks. LoRA also requires careful selection of which modules to adapt—targeting the wrong layers can lead to underwhelming results. Nonetheless, these trade-offs are usually outweighed by its efficiency and practicality. The impact of LoRA goes beyond technical efficiency. By lowering the barrier to entry, it enables startups, researchers, and even hobbyists to work with models that were once accessible only to tech giants. LoRA makes adaptation affordable and scalable. --- ### [Level Up Your Career With AI, But Simple PRO](https://www.aibutsimple.com/upgrade?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=lora-low-rank-adaptation-fine-tuning-for-llms) ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/281697a8-c915-435c-a68b-43374b671698/aibutsimple_academy.png) Supporting us by purchasing a PRO membership will get you: - Exclusive Hands-on Tutorials With Code - Full Math Explanations and Beautiful Visualizations - Ad-Free Reading for an Uninterrupted Learning Experience [Break into AI or sharpen your edge by joining today!](https://www.aibutsimple.com/upgrade?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=lora-low-rank-adaptation-fine-tuning-for-llms) --- LoRA may also intersect with advances in hardware and distributed training. As GPUs become more affordable and cloud platforms optimize for AI workloads, lightweight training methods like LoRA will only grow in importance. To conclude, LoRA represents a shift in how we consider scaling when it comes to LLMs. Bigger models are more powerful, but smarter adaptation strategies make them more useful at a general level. By lowering costs and making the adaptation of models more affordable, LoRA democratizes fine-tuning, allowing start-ups, researchers, and smaller teams to innovate with AI. --- Here’s a special thanks to our [biggest supporters](https://www.aibutsimple.com/upgrade?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=lora-low-rank-adaptation-fine-tuning-for-llms): - Sushant Waidande - Sai Krishna Pashapu - Bob Boncella --- If you enjoy our content, consider supporting us so we can keep doing what we do. Please share this with a friend! Want to reach 50000+ ML engineers? Let’s work together: [WORK WITH US 📈](https://www.passionfroot.me/aibutsimple?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=lora-low-rank-adaptation-fine-tuning-for-llms) Feedback, inquiries? Send us an email at [edwin@aibutsimple.com](mailto:edwin@aibutsimple.com). If you like, you can also donate to our team to push out better newsletters every week! [Support us ☕](https://ko-fi.com/aibutsimple?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=lora-low-rank-adaptation-fine-tuning-for-llms) --- That’s it for this week’s issue of AI, but simple. See you next week! —AI, but simple team