# Simple Guide to Deep Q-Networks (DQNs) #56 Jun 23, 2025 RL https://www.aibutsimple.com/p/simple-guide-to-deep-q-networks-dqns The Deep Q-Network (DQN) is an innovative reinforcement learning (RL) algorithm that combines both deep learning and Q-Learning. Introduced by [DeepMind in 2015](https://storage.googleapis.com/deepmind-data/assets/papers/DeepMindNature14236Paper.pdf?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=simple-guide-to-deep-q-networks-dqns), the DQN shows how deep neural networks can be useful in many domains, even in complex RL tasks. DQNs can take in high-dimensional sensory inputs, beating traditional RL methods, which had difficulty scaling to large state spaces. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/53f8d937-63bc-4b2b-afcc-dadb71dd9e0c/Copy_of_Copy_of_Copy_of_Q-Learning__1080_x_600_px___1080_x_700_px___1080_x_750_px___1080_x_400_px___1_.png) As a quick brush-up, reinforcement learning is a subgroup of machine learning that allows an agent to learn in an environment by trial and error using feedback from its actions. The big idea is to allow computers to learn from their mistakes. Q-Learning is one of the fundamental algorithms of reinforcement learning, being closely related to the computer science concept of dynamic programming, often using a 2D array (which we call a Q-table) to store values. --- ### Join Murf AI’s Coding Challenge—Win Cash and 1 Million API Credits ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/27735c2a-61ed-4192-9027-0f2299591f2c/Newsletter_Banner.png) Know how to code? [Murf AI](https://lu.ma/0dzhcc01?utm_source=aibutsimple&utm_medium=newsletter&utm_campaign=codingchallenge_june) has a new coding challenge, which is open to all! Win from a prize pool of $1000 + 1 Million Murf API Credits. Build with the Murf API, a powerful voice solution for creating multilingual, hyper-realistic voiceovers. Get advanced control with their RESTful API and Python SDK. The new Streaming API generates lifelike speech in real time, perfect for interactive apps, assistants, or live voice experiences. Your first 100,000 characters are free, so start building something awesome and win. [Ready to Code? Join the Challenge Today.](https://lu.ma/0dzhcc01?utm_source=aibutsimple&utm_medium=newsletter&utm_campaign=codingchallenge_june) --- ### RL Definitions Before we move on, here are some useful definitions and a brief introduction to RL concepts needed for this issue. The first term to understand is the environment. The environment is essentially the physical world that houses the RL process, and it is where the agent operates. For instance, in chess, a chessboard could be the environment. The agent (A) is the entity that takes actions that affect the environment. For instance, a model learning to play chess is the agent. Actions (a) are the set of all possible operations the agent can perform. The agent makes a decision on which action to take. On the chessboard, these could be all the possible next moves on your turn. A state (s) is a specific situation that an agent finds itself in at a particular time. For example, the current position of all pieces on a chessboard could be the state. The environment takes the agent’s present state and action as information and returns the reward and a new state. The reward (R) is the feedback given by the environment used to determine the validity of the agent’s actions in each state. It is a critic of the model—a high reward means the model did good, and a low reward means the model needs some correction. For instance, in chess, the move made by the model might be risky or undesired, leading to a negative effect on the game. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/77bc091d-b773-4de0-a2a0-58e37c5601d4/image.png) The policy (π) is a strategy that the agent uses to decide what action to take in a certain state to maximize the reward. The model may strategize to control the middle of the chessboard, for instance. Something we call the value (V) measures how good a state is. It is the expected long-term return that an agent can expect in a given state following some policy. The discount factor (γ) is a number between 0 and 1 that determines the importance of future rewards. The idea is that the future is uncertain, so a certain reward now is weighted more than a future reward. - It is important in balancing exploration vs. exploitation—whether to gather new information (explore) or use existing knowledge for maximum current gains (exploit). Finally, the Q-value, or action-value, is a measure of the total expected reward if the agent (A) takes an action (a) in state (s) and then follows the optimal policy (π) afterwards. You can think of the Q-value as the “quality” of an action at a specific state. This value is central to the Q-learning RL algorithm. ### Q-Learning To understand how deep learning helps enhance Q-Learning in the DQN, we first need to further understand what Q-Learning is. Q-Learning is an algorithm we use to train our Q-Function, an action-value function that outputs our Q-value. The Q-function takes in the current state and an action, then spits out a Q-value using a 2D table called a Q-table. Each cell in the table corresponds to a state-action pair value. You can think of this table as the “memory” of the Q-function. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/2fcbe583-7446-44ac-bad4-13ffa140a05f/Q-Learning__1080_x_600_px_.png) The issue with Q-Learning is that it is a tabular method. This becomes a problem if the state and action spaces are too large to be represented efficiently by a table. Another issue with the table is that it scales very poorly. Q-Learning works well with small state space environments like playing tic-tac-toe (a couple thousand state spaces) or other discrete grid spaces. However, if we want to expand into, let’s say, playing a video game, Q-learning won’t cut it—we have to feed multiple large-sized frames as an input with each pixel storing different information about enemies, projectiles, and so on. For instance, the color version of Pong has 210×160×3 RGB image frames, and each pixel has a value ranging from 0 to 255, giving us 256210×160×3=256100,800 possible frames. This is astronomically huge, which makes using tabular methods like Q-learning infeasible. If the state space is huge, we need to find another way, since creating and updating a Q-table would be downright impossible. ### Deep Q-Learning Given a large state space, a good idea is to approximate the Q-values using a Q-function that is parameterized by a neural network. This neural network will approximate, given a state, the different Q-values for each possible action at that state—this will give us a vector of Q-values. The Deep Q-learning algorithm replaces the Q-table with a neural network that learns the Q-function instead. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/5dafe47a-e97b-4062-976c-2527e3d01c88/Q-Learning__1080_x_600_px___1080_x_700_px_.png) #### The Deep Q-Network (DQN) Continuing our example with Pong, a Deep Q-Network designed for the game has an architecture that resembles the following: ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/a4b6e2a0-0a5d-46eb-a1da-bdb0f3c3bbfb/Copy_of_Q-Learning__1080_x_600_px___1080_x_700_px___1080_x_750_px___2_.png) Here, we use multiple convolutional layers in addition to fully connected layers, as they capture spatial data well and are useful for image-related tasks. - If you want to dive deeper into convolutional layers, you can check out our previous issue: [Convolutional Neural Networks, Mathematically Explained.](https://www.aibutsimple.com/p/convolutional-neural-networks-explained-mathematically?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=simple-guide-to-deep-q-networks-dqns) You might notice that we pass multiple image frames as the input. Typically, when feeding frames to a DQN, we take a stack of frames and pass it through the network as a state. As the data passes through the convolutional layers, it gets flattened and then passed to FC layers. At the end, the network’s output layer is a vector with all the Q-values for all actions given the input state—in this case, we can either move the Pong wall up or down. During training, the agent takes the network’s Q-value vector output and uses the policy to choose the best action in the environment, which gives a reward and the next state. The next state is fed as input to the DQN (along with information about the reward), and the cycle continues until a stopping criterion is met. #### Preprocessing and Temporal Limitation For efficiency, we typically preprocess the input. It’s a crucial step to reducing the complexity of our state, decreasing the overall compute time needed for training. In Pong, the frames are kind of large, so we reduce the state space (image frame size) to 84x84 and grayscale it. Grayscaling the image allows us to reduce three color channels (RGB) to just one, and we can do this since color in Atari environments does not add important information. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/b521542b-7349-4a54-a900-86eeab9b7122/Copy_of_Q-Learning__1080_x_600_px___1080_x_700_px___1080_x_750_px___1080_x_400_px___1_.png) We can also crop a part of the screen in some games if it does not contain important information; then we stack four frames together. But why do we stack four frames? We stack frames together because it helps us deal with temporal limitation. Let’s take an example with the game of Pong. When you only see one frame, you have no idea where the ball is going. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/8ffb6d8f-ecab-4cf1-a18a-62a58f23c8a9/Copy_of_Copy_of_Q-Learning__1080_x_600_px___1080_x_700_px___1080_x_750_px___1080_x_400_px_.png) One frame is not enough to have a sense of motion. But what if you had three frames instead? Then, you can see that the ball is going to the right. Thus, to capture temporal information, we stack four frames together. #### The Deep Q-Learning Algorithm We know that Deep Q-Learning uses a deep neural network to approximate the Q-values for each possible action at a state. However, many more differences distinguish it from traditional Q-learning. One of the key differences is that during training, instead of updating the Q-value of a state-action pair directly like in Q-learning, in deep Q-learning, we use a loss function (Q-Loss) and a target network (Q-Target). - The target network is a separate network used to calculate the target value. - The loss function compares our Q-value prediction with the output from the target network and is used as a measure to perform gradient descent to update the parameters. ![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/1e275011-9223-4618-8268-314572b8d017/image.png) Here, note that both equations are parameterized by different neural networks (main and target networks), and the Q-value is obtained in the neural network’s output. Another difference is the use of a memory buffer—but why do we need one? In reinforcement learning, consecutive experiences (tuples of states, actions, etc.) are highly correlated. An agent moving through a maze will generate a sequence of very similar states. If we were to train on this data, the model would overfit. So instead of training on experiences in the order that they occur, the DQN stores them in a large buffer and samples a mini-batch of a variety of experiences. With this in mind, the Deep Q-Learning training algorithm has two main phases. The first is sampling, where we perform actions and store experience tuples in the buffer. Then, we select a small batch of these tuples and learn using gradient descent in an update step. We repeat the process iteratively until convergence or until a stopping criterion is met. --- Here’s a special thanks to our [biggest supporters](https://www.aibutsimple.com/upgrade?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=simple-guide-to-deep-q-networks-dqns): Sushant Waidande --- If you enjoy our content, consider supporting us so we can keep doing what we do. Please share this with a friend! Want to reach 50000+ ML engineers? Let’s work together: [WORK WITH US 📈](https://www.passionfroot.me/aibutsimple?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=simple-guide-to-deep-q-networks-dqns) Feedback, inquiries? Send us an email at [edwin@aibutsimple.com](mailto:edwin@aibutsimple.com). If you like, you can also donate to our team to push out better newsletters every week! [Support us ☕](https://ko-fi.com/aibutsimple?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=simple-guide-to-deep-q-networks-dqns) --- That’s it for this week’s issue of AI, but simple. See you next week! —AI, but simple team