# Fundamentals of Reinforcement Learning

#9 Jul 8, 2024 RL

https://www.aibutsimple.com/p/fundamentals-reinforcement-learning

Reinforcement Learning (RL) is a subset of machine learning that enables an agent to learn in an interactive environment by trial and error using feedback from its own actions and experiences.

- The fundamental idea behind RL is to allow computers to learn from their mistakes.
- This is very similar to human behavior, as we evolved through trial and error.

Some common applications of RL include playing games (AlphaGo, chess), robotics (BostonDynamics robots), autonomous vehicles (self-driving cars), finance-related activities (algorithmic trading), and other optimization problems.

In RL, the agent is the decision-maker who interacts with the environment to achieve a certain goal.

Unlike supervised learning, where the feedback provided to the agent (or model) are correct labels, reinforcement learning uses rewards and punishments as signals for positive and negative behavior.

- In RL, there is no direct indication of the correct action, but just a positive or negative signal

In reinforcement learning, the agent’s objective is to learn a policy (defines the agent’s behavior) to maximize the amount of reward

- A policy can be thought of as a strategy to determine the actions an agent will take at a specific period in time (state)

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/7c3f1f39-696d-4cc7-852c-ffe26f617890/image.png)

Some key terms that describe the basic elements of an RL problem are:

1. Environment — Physical world in which the agent operates
2. State — Current situation of the agent
3. Reward — Feedback from the environment
4. Policy — Method to map agent’s state to actions
5. Value — Future reward that an agent would receive by taking an action in a particular state

Let’s use a game example to model an RL problem. Let’s suppose we were playing the game Pac-Man, where the goal of the game is to eat the little circles without losing all of your lives.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/e822c5f5-854a-4a58-b921-0131eed740c7/dasdasdadasdas.gif)

In this case, the grid world is the interactive environment for the agent where it acts.

The agent receives a reward for eating circles and punishment if it gets killed by the ghost (loses the game).

- In most RL applications, a punishment takes the form of a low/negative reward value

The states are the locations of the agent in the grid world, and the total cumulative reward is the agent winning the game.

In order to build an optimal policy, the agent faces the dilemma of exploring new states while maximizing its overall reward at the same time. This is called the Exploration vs Exploitation trade-off.

- To balance both, the best overall strategy may involve short-term sacrifices. So, the agent should collect enough information to make the best overall decision.

### Markov Decision Process (MDP)

---

Markov Decision Processes (MDPs) are mathematical frameworks to describe an environment in RL (almost all RL problems can be formulated using MDPs).

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/7e4fabde-de2d-4abe-9326-056a7ad6a51d/image.png)

Reinforcement learning is the process of solving the MDP to find an optimal policy that maximizes cumulative rewards.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/957428ff-23f4-48c0-9ea0-7cf620d6568e/image.png)

### RL Algorithms

---

Reinforcement Learning algorithms branch off into two separate categories: Model-free RL and Model-based RL.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/f5655d60-049f-4d9a-be28-3df7c9983ee5/image.png)

Model-free RL algorithms do not require a model of the environment and learn policies directly from interactions with the environment.

In contrast, Model-based RL algorithms leverage a model of the environment to make predictions about future states and rewards.

- These models can be used to simulate experiences and plan actions, potentially leading to more efficient learning.

Specifically, Model-free RL relies on exploration strategies, and can be sample-inefficient.

Model-based RL uses simulations for efficient exploration and is generally more sample-efficient.

However, in a lot of cases, we usually don't have any prior knowledge about how the environment works. Model-free RL methods come in handy in such cases.

- Model-Free is simpler and less computationally intensive per iteration, whereas model-based methods are more computationally expensive due to the simulations and planning.

In regards to Model-free RL algorithms, Q-learning and SARSA (State-Action-Reward-State-Action) are two commonly used ones.

- These two methods are simple to implement, but they do not have the ability to estimate Q-values (which can vastly increase the ability to learn the optimal policy).
- This can be overcome by more advanced algorithms such as Deep Q-Networks (DQNs) which use Neural Networks to estimate Q-values. But DQNs can only handle low-dimensional spaces.

Here’s a rundown of what a sample Model-free RL Process would look like:

- The agent initializes its policy and value functions, if applicable.
- The agent observes the current state s of the environment.
- Based on its policy π, the agent selects an action a.
- The action is executed, leading to a transition to a new state s′ and a reward r from the environment.
- The agent updates its policy and/or value functions based on the reward received and the new state.

These steps are repeated, allowing the agent to learn over time.

### Q-Learning

---

Q-learning is a commonly used model-free approach that can be used for many RL applications.

It is one of the simplest and most popular RL algorithms due to its ease of implementation and theoretical foundation.

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/df7dec0d-c0cc-4a95-b306-ab904f9501fe/image.png)

Q-learning update rule

The core of Q-learning is the update rule, which adjusts the Q-value based on the observed reward and the estimated value of the next state:

- Learning Rate (α): Determines how much new information overrides old information. A value between 0 and 1.
- Discount Factor (γ): Determines the importance of future rewards. A value between 0 and 1, where 0 makes the agent short-sighted and 1 makes it consider long-term rewards.
- Reward (r): The immediate reward received after taking action a in state s.
- Max Q-value: The maximum estimated Q-value for the next state st+1, indicating the best possible reward.

Here’s some pseudocode for the full algorithm:

- Initialization:
  - Initialize the Q-values arbitrarily for all state-action pairs. Often, they are initialized to zero.
  - Set parameters: learning rate (α), discount factor (γ), and exploration rate (ϵ).
- Loop:
  - Repeat until convergence (or for a specified number of episodes):
    1. Initialize state s.
    2. Repeat until the episode ends:
       - Action Selection:
         - Choose an action a using the policy derived from Q (e.g., epsilon-greedy).
       - Take Action:
         - Execute action a, observe the reward r and next state s′.
       - Update Q-Value:
         - Update the Q-value for the state-action pair (s,a) using the Q-learning update rule (shown previously)
       - State Transition:
         - Set the current state to the next state

![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,width=1200,format=auto,onerror=redirect,quality=80/uploads/asset/file/cd130b62-e316-4223-a261-67b640960915/video_2.gif)

Trained Agent on Q-Learning Algorithm (navigating a maze)

If this issue inspired you to dig deeper into the inner workings of reinforcement learning, check out this well known book by Sutton and Barto: [Reinforcement Learning](http://incompleteideas.net/book/the-book-2nd.html?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=fundamentals-of-reinforcement-learning).

---

If you enjoy our content, consider supporting us so we can keep doing what we do. Please share this with a friend!

If you like, you can also donate to our team to push out better newsletters every week!

[Support us ☕](https://ko-fi.com/aibutsimple?utm_source=newsletter.aibutsimple.com&utm_medium=referral&utm_campaign=fundamentals-of-reinforcement-learning)

---

That’s it for this week’s issue of AI, but simple. See you next week!

—AI, but simple team