Reinforcement Learning
- 1
Foundations: MDPs and the Bellman Equations
Start with the mathematical skeleton of RL: Markov Decision Processes, returns, value functions, and the Bellman equations that tie them together.
incompleteideas.netbook
Reinforcement Learning: An Introduction (2nd ed.), Chapters 1–3The canonical text, free online; given your math background you can read the MDP formalism and Bellman equations directly from the source.
▶youtube.comvideo
RL Course by David Silver - Lecture 2: Markov Decision ProcessesStill the clearest lecture-form treatment of MDPs and Bellman equations, though from 2015.
- 2
Dynamic Programming, Monte Carlo, and TD Learning
Move from the model-based world (policy iteration, value iteration) to model-free learning: Monte Carlo methods and Temporal-Difference learning, including the TD(0)/TD(λ) family. This is the conceptual core that everything in deep RL builds on.
Reinforcement Learning: An Introduction (2nd ed.), Chapters 4–6Covers Dynamic Programming, Monte Carlo methods, and Temporal-Difference learning — the tabular core of RL.
▶RL Course by David Silver - Lectures 3–5 (Planning by DP, Model-Free Prediction, Model-Free Control)Walks through policy/value iteration, then TD prediction and control, matching the book chapters.
- 3
Hands-on: Q-Learning and SARSA from scratch
Implement tabular Q-learning and SARSA yourself on classic control tasks (FrozenLake, Taxi) using Gymnasium. This turns the equations from the last two stages into working code and builds the muscle memory you'll reuse for deep RL.
- 4
Function Approximation and Deep Q-Networks
Replace the Q-table with a neural network. Cover why naive function approximation destabilizes learning, and how DQN's tricks (experience replay, target networks) fix it.
nature.compaper
Human-level control through deep reinforcement learningThe original DQN paper — the paper that started modern deep RL, worth reading closely rather than only in summary.
huggingface.cocourse
Hugging Face Deep RL Course — Unit 3: Deep Q-Learning (hands-on with Atari)Trains a DQN agent on Space Invaders using RL Baselines3 Zoo — makes the paper concrete.
- 5
Policy Gradient and Actor-Critic Methods
Shift from value-based to policy-based methods: the policy gradient theorem, REINFORCE, and how actor-critic architectures reduce variance by combining both approaches.
spinningup.openai.comarticle
Part 2: Kinds of RL Algorithms & Part 3: Intro to Policy OptimizationThe clearest derivation of the policy gradient theorem aimed at practitioners, from Spinning Up's introductory material.
Reinforcement Learning: An Introduction (2nd ed.), Chapter 13: Policy Gradient MethodsThe formal treatment of REINFORCE and actor-critic to complement Spinning Up's practitioner framing.
- 6
Modern Deep RL Algorithms: PPO, TRPO, SAC
Study the algorithms that dominate applied deep RL today: trust-region methods (TRPO) and their practical simplification (PPO), plus soft actor-critic (SAC) for continuous control with entropy regularization.
arxiv.orgpaper
Proximal Policy Optimization AlgorithmsThe PPO paper itself — short, readable, and the algorithm most applied RL work still defaults to.
Spinning Up: Algorithms (TRPO, PPO, SAC, DDPG, TD3)A concise map of the algorithm family and how they relate, with pseudocode and links to each original paper.
- 7
Practical Toolkit: Gymnasium and Stable-Baselines3
Get fluent with the standard applied-RL stack: Gymnasium environments and Stable-Baselines3 implementations of DQN, A2C, and PPO, including training, evaluation, and sharing trained agents.
Hugging Face Deep RL Course — Unit 1: Train your first Deep RL AgentHands-on with Gymnasium and Stable-Baselines3, training a LunarLander agent and evaluating it properly.
stable-baselines3.readthedocs.ioarticle
Stable-Baselines3 DocumentationReference documentation for the algorithms (DQN, A2C, PPO, SAC) and their hyperparameters, useful once you're building your own experiments.
- 8
RL and LLMs: RLHF, DPO, and Modern Post-Training
How reinforcement learning is used to align and improve large language models: the classic RLHF pipeline (supervised fine-tuning, reward modeling, PPO), and the newer wave of alternatives like DPO and GRPO that show up in models like ChatGPT, Claude, and DeepSeek-R1. This ties everything from policy gradients and PPO directly into today's most visible application of RL.
rlhfbook.combook
RLHF Book: Reinforcement Learning from Human Feedback and LLM Post-TrainingA free, quantitatively rigorous book that walks through the entire RLHF pipeline — instruction tuning, reward modeling, PPO, and direct alignment methods like DPO — written for readers with an ML/math background.
Illustrating Reinforcement Learning from Human Feedback (RLHF)A clear, visual introduction to the three-stage RLHF pipeline (pretraining, reward modeling, RL fine-tuning) — a good on-ramp before the denser book.
cameronrwolfe.substack.comarticle
PPO for LLMs: A Guide for Normal PeopleBridges classical policy-gradient/PPO theory directly to how it's actually applied when the "environment" is a language model generating tokens.
- 9
Capstone Project
Pick a harder environment or a small original problem (a game, a control task, or a toy trading/scheduling problem) and build an agent end to end, applying what you've learned across the whole stack.