1 min lesson
RL-for-LLMs from first principles
Use "RL-for-LLMs from first principles" to explain each part and the role it plays.
Step 1 of 2
Reinforcement learning on an LLM is supervised fine-tuning, or SFT, on the model's own samples, with the reward deciding how hard each sample counts. You sample answers, score them and push probability toward the good ones.
The Cursor screens and the research deep-dive are likely to probe whether you can build this up from the math rather than recite acronyms. Start by naming the object you're optimizing, because the whole stack falls out of one expectation.
Frame token generation as a Markov decision process. The model emits one token at a time and each token is a decision conditioned on everything written so far.
The MDP, mapped to a language model
- State
- The prompt plus every token generated so far: the full context window the model conditions on.
- Action
- The next token sampled from the vocabulary distribution.
- Policy π_θ
- The LLM itself, written
π_θ(token | context). The weights θ are what RL updates. - Reward
- A scalar on the finished sequence: did the code compile, pass tests, get preferred by a grader.
- Episode
- One full generation, from prompt to stop token. For an agent, it runs to the end of a multi-tool trajectory.