AI tutorialsMighty Professional
Build a Language Model · Making it useful

Reinforcement Learning from Scratch

Some models can't be taught by showing them the right answer. A chatbot gets a rating after it replies; a game agent finds out it won only at the end. Reinforcement learning trains from a score that arrives after the agent acts. This page builds the core ideas on a 4×4 grid small enough to check by hand, from working out how good each square is to training a policy directly, the method the next page applies to language models.

Time~45 minLevelIntermediatePrereqsNeural Networks for learned function approximation; the tabular sections stand alone.StackC++20 & Rust · browser demos
◂ Build a Language ModelPhase 4 · Making it usefulNext · Fine-tuning and alignment ▸
What you'll be able to do by the end

Compute how good every square of a small known world is by repeated one-step lookahead, and read off the best move. Write a Q-learning update and say what each term does. Add reward hints that speed up learning without changing which policy is best. Tell a real episode ending from a time limit in your training loop and build the right target for each. Explain why subtracting a baseline from a policy gradient makes it less noisy without changing its average.

01An agent learns from scores, not answers

Pretraining teaches a language model by showing it text and asking for the next word, so every example comes with the right answer. Many problems don't come with answers. A chess program finds out whether a move was good only when the game ends, many moves later. A chatbot gets a score after it replies; nobody hands it a correct reply to copy. And the move an agent makes decides which positions it sees next, so its own choices shape its training data. Reinforcement learning (RL) is the family of methods for this setting: an agent takes actions, gets back numeric rewards, and learns to pick actions that collect more reward over time. The agent maximizes the reward you wrote down, which isn't always the behavior you meant: it can raise its score by doing something you didn't intend.

To reason about this, RL describes the world as a (MDP). An MDP has states (where things stand right now), actions, and rules that say which next state and reward can follow each state and action.[1]

The agent's rule for choosing an action in each state is its policy. The "Markov" part is a demand on the state: it must hold everything needed to predict what comes next, so the earlier history adds nothing. A single screenshot of a ball game can fail that test, because it doesn't show which way the ball is moving.[1]

Every demo on this page uses one tiny world. It's a 4×4 grid with squares numbered 0 to 15, left to right along each row, starting at the top left. Square 5 is a wall. The agent can try to move up, right, down or left; walking into the edge or the wall leaves it where it was. Entering square 15, the goal, earns +1 and ends the episode (one complete run from start to finish). Every other move earns −0.04, a small cost that makes shorter routes better. The rules are fully known and nothing is random, which is what makes every number on this page checkable. Real games add hidden information and randomness that this grid leaves out.

The agent's goal is the total reward it collects from now on. Rewards further in the future usually count for less: each extra step multiplies a reward by a discount factor γ (gamma) between 0 and 1. That discounted total is the return.

Worked example

From square 6 (second row, third column) the shortest route to the goal is right to 7, down to 11, down to 15: three moves that earn −0.04, −0.04 and +1.

With γ = 0.9: return = −0.04 + 0.9 × (−0.04) + 0.9² × 1 = −0.04 − 0.036 + 0.81 = 0.734.

The general formula adds up every future reward with its discount, which lets an agent compare a route that pays off soon with one that pays off later.

G_t = reward_(t+1) + γ × reward_(t+2) + γ² × reward_(t+3) + …

In words: the return from time step t, Gt, is the next reward, plus γ times the reward after that, plus γ² times the one after that, and so on. Each step further away shrinks a reward by another factor of γ. In an episode, the sum stops at the move that ends it.

The return depends on which moves the agent makes, so a value is always tied to a policy. A state's value V(s) is the return you expect starting from state s and following that policy. An action value Q(s, a) is the same thing with the first action fixed to a. In the worked example, square 6's value under the shortest-route policy is 0.734.

A value isn't a probability of success. Its units come from the rewards, and it can be negative. Multiply every reward by 10 and every value grows tenfold, while the best policy stays the same.[1]

02Plan when you know the rules

Sometimes you know the rules completely: for any state and action you can look up the next state and the reward. A board game has fixed rules, and a simulator can tell you what any move does. Then the agent doesn't need to try anything. It can work out how good every state is before taking a single step. That's planning.

Planning rests on a one-step lookahead. A state is worth the best you can do with one move: that move's reward, plus the discounted value of the square it lands on. Applying that rule to one state is a . You don't know the values to start with, so set them all to zero and apply the backup to every square, again and again. Each pass carries news of the goal one more step outward.[1]

Worked example

γ = 0.9 and every value starts at 0.

Sweep 1: square 11 can step down into the goal for +1, and the episode ends there, so there's no future term: V(11) = 1. Square 10 can't reach the goal in one move. Its best option is −0.04 + 0.9 × 0 = −0.04.

Sweep 2: from square 10, moving right lands on square 11, which is now worth 1. V(10) = −0.04 + 0.9 × 1 = 0.86.

Written as a formula, the backup computes a state's new value from the current values of the squares next to it. It's how a planner turns known rules into values.

V_next(s) = max_a [reward(s,a) + γ × V(next(s,a))]

In words: try each of the four moves. Score each one as its reward plus γ times the current value of the square it lands on. The state's new value is the best of those scores. A move into the goal has no future term, and the goal's own value stays at zero. (With known rules in a world with random outcomes, each move's score is an average over the squares it might land on.)

Running the backup over every state, sweep after sweep, is value iteration. The widget's version is synchronous: each sweep reads only the previous sweep's values and writes a fresh array. Updating values in place as you go is a valid variant too, but its in-between numbers differ. A sweep isn't a move. Nobody walks the grid during planning, so don't read the sweep count or the changing arrows as a path the agent takes.[1]

Back up reward through the grid

Press Step to run one sweep over every square, or Play to keep sweeping. The left grid shows each square's value; the right grid shows the best move according to those values, with ties going to the first of up, right, down, left. For the first five sweeps the start square's value is slightly negative, because news of the goal hasn't reached it yet. At γ = 0.9 the residual in the readout, the largest change the next sweep would make, shrinks by a factor of 0.9 each sweep and then drops to zero at sweep 6, when the start square settles at 0.4267.

Why does this settle at all? With γ below 1, each sweep shrinks the biggest gap between the current values and the true ones by at least a factor of γ; the residual in the widget shrinks the same way. On any world with a finite number of states, repeated sweeps converge to one set of values.[1] That guarantee covers this exact update with a known model. It isn't a guarantee for a neural network trained on stored experience.

The precise statement

The update that takes the best move, applied to a whole table of values, is a contraction in the maximum norm: for any two value tables, one sweep brings them at least a factor γ closer, measured by their largest difference. A contraction has exactly one fixed point, and repeating it from any starting table converges to that point. The fixed point is the table of optimal values.[1]

The widget reports the residual instead of promising that some number of sweeps is always enough; a bigger or more tangled world needs more. Planning is also only as good as its model. A perfect solution to the wrong rules still picks the wrong move.

03Learn from trial and error

Usually you don't have the rules. A robot can't look up what its wheels will do on a wet floor, and a game-playing agent might see only the screen and never the game's code. What the agent can always do is act and watch. Each move gives it one record: it was in state s, took action a, got reward r, and landed in state s′. Q-learning learns action values from those records alone.[1]

It keeps a table with one number per square and action, Q(s, a): its current guess at the return for taking action a in square s. After each move it forms a better guess from what just happened: the reward it actually got, plus γ times the best Q value at the square it landed on. Then it moves the old guess part of the way toward the better one. The future part of that better guess comes from the table itself, so it's called the bootstrap term: the table uses its own estimates to improve its estimates.

Worked example

The agent is in square 10 and Q(10, right) = 0. It moves right to square 11 and gets −0.04. The best Q value at square 11 so far is 0.5. γ = 0.9 and the learning rate α (alpha) is 0.4.

target = −0.04 + 0.9 × 0.5 = 0.41. error = 0.41 − 0 = 0.41.

new Q(10, right) = 0 + 0.4 × 0.41 = 0.164. No other entry in the table changes.

The whole update is three lines, and it's how the agent turns one experienced move into a better value guess.

target = reward + γ × max_a Q(next_state,a)
error = target − Q(state,action)
Q(state,action) ← Q(state,action) + α × error

In words: the target is the reward you saw plus the discounted best guess for where you landed. The error is how far the old guess was from the target. The learning rate α says what fraction of that gap to close. When the move truly ends the episode, the future term is dropped and the target is just the reward.

That error is the temporal-difference error, or TD error, because it compares guesses made one time step apart. A supervised label stays fixed. This target is built from the table, so it moves as the table learns. Each update changes only the one entry for the state and action just taken.

Learn from one move at a time

Press Step to take one move and make one update, or Play to keep going. The left grid shows the best Q value in each square, with the agent's square in yellow; the bars on the right are the four Q values where the agent stands now. Values go negative at first, because every move costs −0.04 and the +1 only starts spreading back once the agent stumbles into the goal. Reaching the goal sends the agent back to square 0. Changing a slider restarts the run with the same random seed.

Q-learning is : the target always uses the best next action (the max), even when the agent will actually explore and take a different one. So the agent can behave one way, with random exploring moves, while learning the values of a different policy that always takes the best move. That doesn't let it skip experience it never had. A state and action the agent never tries never gets an update, so its value stays a guess.[1]

The widget uses a fixed learning rate of 0.4 so each change is big enough to see.

When is Q-learning guaranteed to converge?

The classical guarantee for table-based Q-learning needs two things: every state and action keeps getting tried, without end, and the learning rate shrinks over time, slowly enough that the rates add up to infinity but fast enough that their squares add up to a finite number.[1] The widget's fixed rate of 0.4 doesn't meet the second condition, so a run here demonstrates the update and proves nothing about convergence.

04Explore on purpose

An agent that always takes the move it currently rates best can get stuck. Suppose its first lucky trip to the goal went the long way round. That route now has the highest Q values, so the agent repeats it, and it never finds the shortcut because it never tries the moves it rates lower. To learn, it has to spend some moves trying things.

The simplest fix is an ε-greedy policy: with probability ε (epsilon), pick one of the four moves uniformly at random, the current best one included; otherwise take the move with the highest Q value.[1]

Worked example

ε = 0.2, and the Q values for up, right, down and left are 0, 1, 0 and 0, so right is best.

Right: 0.8 from the greedy branch, plus 0.2 / 4 = 0.05 from the random branch, = 0.85.

Each other move: 0.05. Check: 0.85 + 3 × 0.05 = 1.

In general, with four moves and one best move, the best gets 1 − ε + ε/4 and each other move gets ε/4. When several moves tie for best, this widget splits the greedy share, 1 − ε, equally among them. Code that breaks ties another way, say by always picking the first, behaves differently, even though the max in the Q-learning target is the same.

Work out the ε-greedy probabilities

Drag the right-action value to set how the move right compares with the other three, which stay at 0, and drag ε to set how often the agent explores. Above zero, right is the single best move; at exactly zero all four tie; below zero the other three tie for best. The right panel shows each move's exact probability, random branch included, and they always add up to one. At the defaults (ε = 0.2, right valued at 1) right gets 0.85 and each other move 0.05.

Raising ε gives the low-rated moves more chances. When all four tie, ε changes nothing: every move gets 0.25 either way. Keep ε in mind when you measure an agent, too. Its score while exploring with ε = 0.3 answers a different question from its score when it always takes its best move. With every result, record which policy was evaluated, the random seeds, the episode limit and the reward definition. One successful training episode tells you little about how reliable the agent is.

Random moves are a weak explorer when the reward needs a long, specific sequence of actions, because the chance of stumbling on that sequence falls quickly as it gets longer. How well any exploration works also depends on how the states and actions are represented. For rewards that are rare and far away, treat ε-greedy as the baseline a better method has to beat.

05Give hints without changing the task

In the grid, the only positive reward is at the goal, so a learner wanders for a long time before anything useful happens. It's tempting to add hints, like a small bonus for each move toward the goal. Careless bonuses change the task, though. Reward a soccer robot for touching the ball and it can learn to stand next to the ball tapping it over and over instead of scoring. Potential-based shaping is a way to add hints that provably doesn't change which policy is best.[2]

Give every state a fixed number, its potential Φ (phi), that says how promising it looks. Here the potential is minus the number of moves to the goal. The bonus for a move is γ times the potential where you land, minus the potential where you started. Moving toward the goal earns a bonus and backing away costs one. Over a complete episode the bonuses cancel down to one number that depends only on where you started, so every complete route gets the same extra amount.

Worked example

Take the three-move route from square 6 again (6 → 7 → 11 → 15) with γ = 0.9. The potentials along it are −3, −2, −1 and 0; the goal's potential is always 0.

Bonuses: 0.9 × (−2) − (−3) = 1.2, then 0.9 × (−1) − (−2) = 1.1, then 0.9 × 0 − (−1) = 1.

Discounted total: 1.2 + 0.9 × 1.1 + 0.9² × 1 = 1.2 + 0.99 + 0.81 = 3, which is exactly −Φ(6). Any other complete route from square 6 gets the same 3.

As a formula, the bonus is computed for each move and added to the real reward:

shaping(s,s′) = γ × Φ(s′) − Φ(s)
shaped_reward = original_reward + shaping(s,s′)

In words: the shaping bonus for moving from s to s′ is γ times the new state's potential minus the old state's potential, and the agent trains on the real reward plus that bonus. Use the same γ as the return, and give the goal a potential of zero.

Why the bonuses cancel: add them up along any route of T moves, with discounting, and every potential in the middle of the route appears once with a plus sign and once with a minus sign. All that's left is −Φ(start) + γTΦ(end). With the goal's potential at zero, the extra return is just −Φ(start), the same for every complete route from that start, so the ranking of routes from best to worst can't change.[2]

Tasks that never end

A continuing task has no terminal state. With γ below 1 and a potential that stays within fixed bounds, the leftover term γTΦ(end) shrinks toward zero as T grows, so the same conclusion holds.[2]

Check that shaping shifts every route equally

Drag γ or the potential scale and compare two complete routes from square 0 to the goal. Path A is the six-move shortest route; Path B takes eight moves, including a step back. Each panel shows a path's real return and its shaped return. At the defaults, A's returns are 0.4267 and 6.4267 and B's are 0.2696 and 6.2696: shaping adds exactly 6, which is −Φ(start), to both, so A stays ahead by the same margin. Changing γ changes the returns but both paths still get the same amount added. Set the scale to 0 to turn shaping off.

Path A visits squares 0 → 1 → 2 → 3 → 7 → 11 → 15. Path B visits 0 → 4 → 8 → 9 → 8 → 9 → 10 → 14 → 15. The potential is minus the Manhattan distance to the goal (rows apart plus columns apart), times the scale slider. It ignores the wall, and the cancellation still works, because the argument holds for any fixed potential; a poor potential just gives less useful hints. A potential that changes during training isn't covered by this argument.

So shaping can help a learner find a good policy faster without changing which policy is best. It can't fix a reward that measures the wrong thing, and it can't supply information the state is missing.

06The task ended, or you stopped watching

Training code cuts episodes short all the time. If a game agent hasn't reached the goal after 200 moves, the loop resets it to collect fresh data. That cutoff is a decision about data collection; the game itself would have gone on. If the code treats it the same as reaching the goal, the target for the last move drops the future term, as if the world had ended there.[3]

So there are two different ways an episode can stop. Termination means the task itself is over: the agent reached the goal, or lost. No more reward can come, so the target drops the future term. Truncation means you stopped watching: a time limit ran out, or a batch filled up. The task would have gone on, so the target keeps γ times the next state's value.

Worked example

The last move earned a reward of 1, the next state's estimated value is 5, and γ = 0.9.

If the task terminated: target = 1.

If a time limit truncated it: target = 1 + 0.9 × 5 = 5.5.

Treat the truncation as a termination and you train that move's value toward 1 instead of 5.5.

One formula covers both cases, with a flag that's 1 only when the task truly ended:

target = reward + γ × (1 − terminated) × next_value

In words: multiply the future term by zero only when the task really ended. A time limit or a batch boundary leaves terminated at 0, so the future term stays.

Gymnasium, a widely used Python interface for RL environments, returns separate terminated and truncated flags for this reason.[4] On truncation, compute the future term from the last observation of the interrupted episode. Vector environments add a trap. In Gymnasium's same-step autoreset mode, the observation returned alongside the truncation already belongs to the next episode, and the real final observation is stored separately. Using the reset observation in the target mixes two episodes together. Check which autoreset mode your environments use, and fetch the real final observation before updating.[5]

Keep or remove the future term

Drag the observed reward, the next-state value estimate and γ. The left bar is the target if the task truly ended: the reward alone. The right bar is the target if a time limit cut the episode short: the reward plus γ times the next value. At the defaults that's 1 versus 5.5. When the deadline is part of the task itself, the state needs to include the time left, as the next paragraph explains.

A deadline that's part of the task is a different case. If the job is "reach the exit within 50 moves", the time left changes which move is best, so it belongs in the state, and running out of time is a real termination. Pardo et al. separate the two cases, which is why "always keep the future term on a time limit" is too broad a rule.[3]

07Train the policy directly

Everything so far learned values and picked moves from them. Another family of methods skips the values and adjusts the policy itself. The policy becomes a function with weights that outputs a probability for each action, and training makes the actions that led to high return more likely. The next page uses this kind of method on language models, where the policy is the model and each action is the next token.

After trying an action and seeing its return, nudge the weights in the direction that raises that action's log probability, scaled by the return. That direction is the gradient of log π, where π is the policy's probability for the action. This recipe is a score-function estimator.[6]

It's noisy. If every action earns a positive return, every sampled action gets pushed up, just by different amounts, and the estimate swings a lot from one sample to the next. The standard fix is to subtract a from the return: a reference level, such as the return you'd expect in this state. Actions that beat it get pushed up and actions that fall short get pushed down.

The widget uses the smallest possible case, a two-action bandit: one decision, with no later states. Action 0 always pays 1 and action 1 always pays 4. The policy has a single weight, a logit, and picks action 1 with probability p. The gradient of log probability with respect to that logit is 1 − p for action 1 and −p for action 0.

Worked example

p = 0.35, so the gradients of log probability are 1 − 0.35 = 0.65 for action 1 and −0.35 for action 0.

No baseline: picking action 1 gives 4 × 0.65 = 2.6; picking action 0 gives 1 × (−0.35) = −0.35. Average: 0.35 × 2.6 + 0.65 × (−0.35) = 0.6825.

Baseline 2.95: action 1 gives (4 − 2.95) × 0.65 = 0.6825 and action 0 gives (1 − 2.95) × (−0.35) = 0.6825. The average is unchanged, and both samples are now equal, so the noise is gone.

The general estimator, for one sampled action, is:

estimate = (return − baseline) × ∇ log π(action | state)

In words: take how much the sampled action's return beat the baseline, and multiply by the direction that makes that action more likely. Averaged over many samples, this points in the direction that raises expected return. The baseline mustn't depend on which action was sampled, and it's held constant when you take the gradient.

Why doesn't the baseline change the average? Averaged over actions, its contribution is the baseline times Σ π(a|s) ∇log π(a|s). Since π ∇log π = ∇π, that equals the baseline times ∇ Σ π(a|s). The probabilities always add up to 1, and the gradient of a constant is zero. The variance can still go either way, and a poorly chosen baseline can make it worse. A learned estimate of the state's value is a common baseline, but it isn't guaranteed to give the lowest variance.[6]

Move the baseline: same average, different noise

Drag the baseline and compare the panels. The left bars are the gradient sample you'd get if action 0 or action 1 were picked; the right bars are their exact average and variance, computed from both outcomes instead of by sampling. At the default p = 0.35 the average stays at 0.6825 for every baseline, while the variance falls from 1.98 at baseline 0 to zero at 2.95. A baseline that removes all the variance exists here only because there are two actions with fixed rewards; in general no baseline does that.

A full multi-step policy gradient also has to decide how discounting enters and which states count in the objective. The one-decision bandit has none of that, so the widget shows the baseline's effect on its own.

08A Q-learning update in code

These complete programs implement the Q-learning update from section 03 for one move. The caller passes the next state's Q values and a flag that's true only when the task really ended. The tests use the numbers from section 06 (reward 1, best next value 5, γ = 0.9): they check that a truncated move keeps its future term and gets a target of 5.5, while a terminated one gets 1. They also check that the target is computed before the current row changes. That matters when a move bumps into a wall and the agent stays put, so the current and next rows are the same. The C++ test passes the same array twice; Rust's borrowing rules forbid that, so the Rust test passes a copy.

A checked Q-learning update, complete programs
#include <array>
#include <algorithm>
#include <cassert>
#include <cmath>
using Actions = std::array<double,4>;
double update(Actions& current, int action, double reward,
              const Actions& next, double discount, double learningRate,
              bool terminated) {
    assert(action >= 0 && action < 4);
    assert(discount >= 0 && discount <= 1);
    assert(learningRate > 0 && learningRate <= 1);
    // next refers to the final observation, not an auto-reset observation.
    const double bootstrap = terminated ? 0.0 : *std::max_element(next.begin(),next.end());
    const double target = reward + discount*bootstrap;
    const double error = target-current[action];
    current[action] += learningRate*error; // Read the whole target before modifying Q.
    return error;
}
int main() {
    Actions current{0,0,0,0};
    const Actions next{2,5,1,0};
    const double error = update(current,1,1,next,0.9,0.4,false);
    assert(std::abs(error-5.5) < 1e-12);
    assert(std::abs(current[1]-2.2) < 1e-12);
    Actions terminal{0,0,0,0};
    update(terminal,1,1,next,0.9,0.4,true);
    assert(std::abs(terminal[1]-0.4) < 1e-12);
    // An external time limit has terminated=false and retains bootstrapping.
    Actions truncated{0,0,0,0};
    update(truncated,1,1,next,0.9,0.4,false);
    assert(truncated == current);
    // Aliased current/next is safe because the target is evaluated first.
    Actions self{1,2,3,4};
    update(self,0,0,self,0.9,1.0,false);
    assert(std::abs(self[0]-3.6) < 1e-12);
}
type Actions = [f64;4];
fn update(current: &mut Actions, action: usize, reward: f64,
          next: &Actions, discount: f64, learning_rate: f64,
          terminated: bool) -> f64 {
    assert!(action < 4);
    assert!((0.0..=1.0).contains(&discount));
    assert!(learning_rate > 0.0 && learning_rate <= 1.0);
    // next refers to the final observation, not an auto-reset observation.
    let bootstrap = if terminated { 0.0 }
        else { next.iter().copied().fold(f64::NEG_INFINITY,f64::max) };
    let target = reward + discount*bootstrap;
    let error = target-current[action];
    current[action] += learning_rate*error; // Read the whole target before modifying Q.
    error
}
fn main() {
    let mut current = [0.0;4];
    let next = [2.0,5.0,1.0,0.0];
    let error = update(&mut current,1,1.0,&next,0.9,0.4,false);
    assert!((error-5.5).abs() < 1e-12);
    assert!((current[1]-2.2).abs() < 1e-12);
    let mut terminal = [0.0;4];
    update(&mut terminal,1,1.0,&next,0.9,0.4,true);
    assert!((terminal[1]-0.4).abs() < 1e-12);
    // An external time limit has terminated=false and retains bootstrapping.
    let mut truncated = [0.0;4];
    update(&mut truncated,1,1.0,&next,0.9,0.4,false);
    assert_eq!(truncated,current);
    let mut self_state = [1.0,2.0,3.0,4.0];
    let previous = self_state; // Rust separates the immutable target from the mutable row.
    update(&mut self_state,0,0.0,&previous,0.9,1.0,false);
    assert!((self_state[0]-3.6).abs() < 1e-12);
}

The listing leaves out the loop that runs the environment, storage for past experience, schedules for α and ε, and evaluation. It also relies on a table, which stops working once there are too many states to list. Bigger problems replace the table with a neural network that predicts Q values. The network shares weights across states, so an update for one state also shifts the predictions for others, which makes training less stable. DQN (deep Q-network), from Mnih et al., added two fixes. A replay buffer stores past moves and trains on random batches drawn from it, which breaks up runs of nearly identical consecutive moves. A separate target network, a copy of the Q network that's refreshed only every so often, computes the targets, so they don't shift with every update.[7]

Neither fix guarantees that deep Q-learning converges, and both add choices of their own: how to sample from the buffer, how old its data may get, how terminal moves are stored, and how often to refresh the target. Test the table version first. Then check the network on small worlds whose true values you can compute, before blaming a failure on the network being too small.

09Measure the agent you meant to build

A reward curve mixes together the learning algorithm, the exploration policy, the environment and the way you evaluated. Record each one separately. Compare against a simple fixed policy, run several random seeds, and watch what the agent actually does: a high return can come from exploiting a flaw in the reward instead of doing the task.

Count environment steps and gradient updates separately. Replay can train on the same experience many times, so "steps" on its own is ambiguous. The widgets here count sweeps or sampled moves, and neither one is elapsed game time.

Success on a toy grid doesn't make an agent safe to deploy

The grid shows the agent everything, and nothing in it is random. A real controller can face delayed actions, partial observations and states it never saw in training. Test those conditions directly before relying on a learned policy.

10What's next

Fine-tuning and Alignment applies the policy gradient from section 07 to a language model. It covers a reward model learned from people's choices between two replies, and RLHF, which maximizes that reward while a KL penalty keeps the model close to where it started (the penalty grows as the model's probabilities drift from the original's). It ends with DPO, which reaches the same optimum without a separate reward model.

11Sources

The cited equations define the algorithms. Widget readouts report the displayed toy arrays and settings; they are not hardware benchmarks.

  1. Richard S. Sutton and Andrew G. Barto, 2018, second edition. Reinforcement Learning: An Introduction. Chapters 2–6: returns, Markov decision processes, planning, and Q-learning; chapter 13: policy gradients.
  2. Andrew Y. Ng, Daishi Harada, Stuart Russell, 1999. Policy invariance under reward transformations: Theory and application to reward shaping. Potential-based shaping and the policy-invariance conditions.
  3. Fabio Pardo, Arash Tavakoli, Vitaly Levdik, Petar Kormushev, 2018. Time Limits in Reinforcement Learning. Task horizons, external truncation, and bootstrapping.
  4. Farama Foundation, Gymnasium documentation. Handling Time Limits. The separate terminated and truncated flags and their targets.
  5. Farama Foundation, 2025. Deep Dive: Autoreset Modes in Gymnasium v1.1 Vector Environments. Same-step reset observations and retrieving the actual final observation.
  6. Richard S. Sutton, David McAllester, Satinder Singh, Yishay Mansour, 1999. Policy Gradient Methods for Reinforcement Learning with Function Approximation. Policy-gradient expectations and action-independent baselines.
  7. Volodymyr Mnih et al., 2015. Human-level control through deep reinforcement learning. Replay sampling, lagged target networks, and learned Q functions.