1 min lesson
RL-for-LLMs from first principles
Use "RL-for-LLMs from first principles" to explain each part and the role it plays.
Step 1 of 2
Reinforcement learning on an LLM is just supervised learning where you don't have the labels - so you sample answers, score them and push probability toward the good ones.
The Cursor screens and the research deep-dive both probe whether you can build this up from the math rather than recite acronyms. Start by naming the object you're optimizing, because the whole stack falls out of one expectation.
Frame token generation as a Markov decision process. The model emits one token at a time and each token is a decision conditioned on everything written so far.
The MDP, mapped to a language model
- State
- The prompt plus every token generated so far - the full context window the model conditions on.
- Action
- The next token sampled from the vocabulary distribution.
- Policy π_θ
- The LLM itself: π_θ(token | context). The weights θ are what RL updates.
- Reward
- A scalar on the finished sequence - did the code compile, pass tests, get preferred by a grader.
- Episode
- One full generation, from prompt to stop token (or, for an agent, to the end of a multi-tool trajectory).