# GPU Optimization for Machine Learning, Simply Explained #83 Dec 29, 2025 Inference https://www.aibutsimple.com/p/gpu-optimization-for-machine-learning-simply-explained Machine Learning (ML) revolves around one central operation: matrix multiplication. Depending on the size of these matrices, the computation can be decently fast, but that’s a rare phenomenon in many applications—where matrices can have millions of entries. Graphical Processing Units (GPUs), originally intended for graphical display and gaming, solve the problem of slow computation with the ability of parallel processing. Instead of computing repetitive tasks with limited parallelism (although not none), which is the strategy of Central Processing Units (CPUs), GPUs split these tasks into different threads and complete them simultaneously. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/e2c7016e-5c74-4a7c-9199-d93f406d3086/GPU_optimization_ML__1_.png) Companies like NVIDIA, AMD, and Intel compete to create more capable GPUs, which are especially in demand due to the field of machine learning, where processing loads of millions of data points is a routine task. In this issue, we’ll break down: 1. The infrastructure of a GPU (where its parallel processing comes from) 2. Methods of optimizing its performance --- ### [Level Up Your Career With AI, But Simple PRO](https://www.aibutsimple.com/upgrade?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=gpu-optimization-for-machine-learning-simply-explained) ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/281697a8-c915-435c-a68b-43374b671698/aibutsimple_academy.png) Supporting us by purchasing a PRO membership will get you: - Exclusive Hands-on Tutorials With Code - Full Math Explanations and Beautiful Visualizations - Ad-Free Reading for an Uninterrupted Learning Experience [Break into AI or sharpen your edge by joining today!](https://www.aibutsimple.com/upgrade?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=gpu-optimization-for-machine-learning-simply-explained) --- ### The Components of a GPU A modern GPU contains thousands of small processing cores organized into larger units called Streaming Multiprocessors (SMs). These SMs are similar to weaker CPU cores. A modern CPU has between 8-16 powerful cores, but on the other hand, a high-end GPU like NVIDIA's H100 possesses over 16,000 cores, equating to 132 SMs. This produces a massive compute power advantage over regular CPUs. These SMs have an infrastructure similar to [Russian Nesting Dolls](https://en.wikipedia.org/wiki/Matryoshka_doll?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=gpu-optimization-for-machine-learning-simply-explained), each with inner-working parts. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/d65969a9-1ea9-46fd-860c-38df20199efa/GPU_optimization_ML.png) Thread: Threads are the smallest unit of execution. A single thread performs one computation or step in a task. For example, this could be computing a single entry in the output of a matrix multiplication. Warp: A warp is a group of 32 threads that are jointly executed. Think of these as packages of threads. Warps are the fundamental execution unit in NVIDIA GPUs. All 32 threads in a warp execute the same instruction at the same time, just on different data. Block: Blocks are collections of threads (organized into warps) that can cooperate by sharing data across different execution timelines. A block might contain 256 threads, which would be 8 warps. An SM can directly house blocks, so every thread within those blocks is executed on the same SM. Grid: The entire collection of blocks that make up your computation. As you can see, GPUs have a very high level of abstraction, housing many layers of interconnected pieces that enable parallel processing. Parallel processing is only possible due to different warps being able to handle different tasks simultaneously, instead of sequentially. ### Sharing Resources and Memory Similar to organizing compute resources, GPUs arrange available memory into distinct but collaborative layers. Each resource serves to provide stepping stones for future layers to build on. Registers: Tiny and fast storage locations inside a GPU/CPU. Each thread has its own private registers (no communication with other threads) where it actively works with data during computation. Access is nearly instantaneous, taking 1 cycle, which is typically under 1 nanosecond. - Cycles are the basic time steps that a processor uses to do work. Think of it as one tick on the processor’s clock. However, space is extremely limited because registers are built directly into the processing cores themselves using expensive, high-speed transistors. For example, each SM might have only 65,536 32-bit registers total, which must be divided among all active threads. If each thread needs 32 registers and you want 2048 active threads per SM, all of the available registers are exhausted. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/c177ebf5-eccc-4177-a5dd-ed63419619b9/GPU_optimization_ML__2_.png) Alternatively, you could sacrifice memory to create more active threads or more memory for fewer threads, as illustrated by the image above. Shared Memory: When threads need to exchange data or reuse common values, they move data from their registers to shared memory where other threads can access it. Access ranges around 20-30 cycles, which is slower than accessing from registers but still fast because shared memory sits physically close to the processing cores on the same SM. Global Memory: The main GPU memory (VRAM) that stores inputs and outputs from calculations. Theoretically, all threads can access global memory, but access can take 200-400 cycles, which is over 10x slower than shared memory. This slowness is caused by extra physical distance, since global memory usually exists on separate chips connected to the GPU. L1/L2 Cache: This is the caching layer that is between the two different types of memory. Common values are automatically cached and stored for future use. When threads interact with global memory, the GPU stores copies in cache to speed up repeated accesses. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/26c76a5b-dd9d-40f4-9511-877eff2a42ba/GPU_optimization_ML__3_.png) The fundamental challenge in GPU programming is minimizing the number of global memory accesses by reusing data as much as possible in closer layers. ### Anatomy of a GPU Computation To understand how these pieces work together, let's trace a matrix multiplication C = A × B. Step 1: CPU Preparation The CPU allocates memory on the GPU and copies matrices A and B from system RAM to GPU global memory over PCIe, the mechanism that transfers data between the CPU and GPU. Step 2: Launching the Kernel The CPU launches a kernel, a space where the GPU can run functions and algorithms, with a specific grid (matrix) configuration. For two 1024×1024 matrices, you might launch 4×4 blocks, each with 16×16 threads, creating 1,048,576 threads working in parallel that are responsible for their own output matrix entry. We cover this in full action in our weekly PRO code tutorial. Subscribe to [AI, But Simple PRO](https://www.aibutsimple.com/upgrade?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=gpu-optimization-for-machine-learning-simply-explained) for industry knowledge and detailed code tutorials! ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/84240a6d-0c27-44cc-975d-e9bf2b606bc2/GPU_optimization_ML__5_.png) Step 3: Thread and Task Assignment The GPU scheduler, which is a piece of embedded software, distributes threads and blocks across available SMs. Threads are automatically grouped into warps of 32, but these can be modified using packages like [PyCUDA](https://pypi.org/project/pycuda/?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=gpu-optimization-for-machine-learning-simply-explained). Step 4: Loading Data from Global Memory Each thread loads its required data from global memory into registers for individual computation. For matrix multiplication, thread [i, j] needs row i from matrix A and column j from matrix B, which is sent to a register to compute. Step 5: Computation Threads perform calculations using data in their registers. Thread [i,j] computes a dot product, multiplying corresponding elements and summing results. All this arithmetic happens in registers using the thread's Arithmetic Logic Units (ALUs), and since these are parallelly computed, large matrix multiplications are extremely fast. Step 6: Writing Results Threads then send computed values back to global memory (the output). Thread [i,j] writes its result at position [i,j] in matrix C. Step 7: CPU Receives Product Once all blocks finish, the CPU copies the result matrix C back to system RAM. This pipeline highlights a critical principle: we should maximize compute time and minimize data movement time. ### Identifying Bottlenecks The primary bottleneck in GPU performance is global memory access. A technique called memory coalescing mitigates this issue. When a warp requests data, the GPU doesn't fetch 32 individual values; it fetches 128-byte caches. If the 32 threads in a warp access consecutive memory addresses, one memory transaction will be able to satisfy all requests. This is called coalesced access. Good pattern (coalesced): Thread 0 reads address 0 Thread 1 reads address 4 Thread 2 reads address 8 … Thread 31 reads address 124 Result: One 128-byte fetch serves all 32 threads! Alternatively, uncoalesced access occurs when the GPU tries to access data that would not exist along “nearby” memory addresses. Bad pattern (uncoalesced): Thread 0 reads address 0 Thread 1 reads address 1000 (Note that this address is outside of a 128-byte range) Thread 2 reads address 2000 ... Result: Up to 32 separate transactions. Matrix multiplication seems like a naturally coalesced example since elements are stored consecutively. However, reading columns requires jumping between rows, meaning that elements in a 1024x1024 matrix are 1024 positions apart in memory, creating uncoalesced access that significantly slows computation. In our [PRO Edition Code Tutorial](https://www.aibutsimple.com/upgrade?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=gpu-optimization-for-machine-learning-simply-explained), we explore a way to circumnavigate this issue. It’s for this reason that naively computing GPU matrix multiplication can be surprisingly slow despite previously mentioned parallelism. Let’s look at some optimization tactics we can use to approach structuring threads. ### Optimization Techniques The most powerful optimization for matrix operations is tiling, which exploits how shared memory works. Tiling essentially breaks computations into small chunks that fit in shared memory. Passing small chunks through less capable but faster shared memory (10x faster than global memory access) speeds up large computations that usually pull back and forth from global memory. Standard approach without tiling: 1. Thread [i,j] independently loads row i of A (each entry is from global memory) 2. Thread [i,j] independently loads column j of B 3. GPU performs 1024 multiply-add operations Total global memory accesses: 2048 per thread (1 for each matrix entry in A and B) Tiled approach (16×16 example): 1. Threads in a block cooperatively load a 16×16 tile of A into shared memory 2. Threads cooperatively load a 16×16 tile of B into shared memory 3. All threads compute using these shared tiles 4. Process next tiles until complete With tiling, we do not query global memory, saving us a lot of computation speed. Here is an example with a 4x4 matrix: ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/4c01060e-38fd-4bc9-a612-ce2858e26683/GPU_optimization_ML__6_.png) Depending on how optimized the matrix already is, for a 1024×1024 matrix, tiling can speed up this computation by approximately 2-3x by reducing the amount of global memory accesses—assuming that the matrix is coalesced and reasonably occupied. In real-world scenarios, global memory access is a bottleneck for larger matrix computations, so you can expect the efficiency boost to scale somewhat. Shared memory acts as a manually managed cache since you can explicitly tell the GPU to load data once, and multiple threads will reuse it many times. Another technique is maximizing occupancy on threads (or warps, which are groups of 32 threads). An SM can run multiple blocks simultaneously, but resources are limited: registers, shared memory, and thread slots. - Occupancy measures what percentage of an SM's maximum warps/threads are actually active at any given time. If only five are in use, we achieve 50% occupancy. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/ac6717a2-b283-49dd-a2ac-29432e2dbf39/GPU_optimization_ML__9_.png) Higher occupancy also helps to hide memory latency by increasing warp productivity. If Warp A needs to access global memory, this creates a period of 200-400 cycles where the SM would have to wait. During this time, the SM can switch to Warp B and continue working. The challenge that is presented here is that the number of blocks and number of warps per SM is limited by available resources. If you request too many resources per block, you’ll have fewer blocks on each SM. Here’s an example: An SM has 65536 registers total. If you launch blocks with 256 threads (8 warps) that each have 64 registers, that’s 256 x 64 = 16384 registers per block. This means you can have at most 4 blocks on the SM. This is great if the SM can only handle 4 blocks, but if the hardware is capable of more, you’re not using maximum occupancy. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/0de3fadc-a0d8-452f-9b75-13714e29492d/GPU_optimization_ML__7_.png) The good thing is that you can control occupancy through launch configuration. You can set the following: - Threads per block: More threads means more parallelism but higher resource consumption per block (more threads = more registers = fewer blocks). - Registers per thread: Complex kernel activations need more registers, but excessive register usage prevents the SM from fitting multiple blocks simultaneously, reducing occupancy. - Shared memory per block: Tiling strategies use shared memory. Requesting too much shared memory prevents multiple blocks from running on the same SM. A practical guideline: target at least 50% occupancy. Beyond that, the relationship between occupancy and performance becomes nonlinear: 100% occupancy sometimes performs worse than 75% if achieving it requires problematic tradeoffs elsewhere. ### Conclusion The core of what allows GPUs to achieve incredible performance necessary for today’s industries is thousands of threads working in parallel, but squeezing peak performance out of the hardware requires strategic optimization. As AI models continue scaling as aggressively as ever, efficient memory access and resource allocation will become increasingly critical for pushing the boundaries of what's computationally possible. --- Here’s a special thanks to our [biggest supporters](https://www.aibutsimple.com/upgrade?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=gpu-optimization-for-machine-learning-simply-explained): - Sushant Waidande - Sai Krishna Pashapu - Bob Boncella --- If you enjoy our content, consider supporting us so we can keep doing what we do. Please share this with a friend! Want to reach 50000+ ML engineers? Let’s work together: [WORK WITH US 📈](https://www.passionfroot.me/aibutsimple?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=gpu-optimization-for-machine-learning-simply-explained) Feedback, inquiries? Send us an email at [edwin@aibutsimple.com](mailto:edwin@aibutsimple.com). If you like, you can also donate to help our team push out better newsletters every week! [Support us ☕](https://ko-fi.com/aibutsimple?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=gpu-optimization-for-machine-learning-simply-explained) --- That’s it for this week’s issue of AI, but simple. See you next week! —AI, but simple team