2 min lesson
Q2 - PPO vs GRPO and their compute tradeoffs
Compare two rows from "Q2 - PPO vs GRPO and their compute tradeoffs", then say when each one fits.
Step 1 of 3
Q2 - PPO vs GRPO and their compute tradeoffs
- Baseline / advantage
- PPO
- Learned value network (critic).
- GRPO
- Group-relative: normalize rewards across K samples of the same prompt.
- Extra model in memory
- PPO
- Yes - the critic, trained alongside the policy.
- GRPO
- No critic; the group is the baseline.
- Compute cost
- PPO
- Higher - critic forward/backward and its own tuning.
- GRPO
- Lower memory, but K rollouts per prompt cost sampling.
- Stability lever
- PPO
- Clipped ratio bounds the policy step.
- GRPO
- Group normalization plus KL control.
| PPO | GRPO | |
|---|---|---|
| Baseline / advantage | Learned value network (critic). | Group-relative: normalize rewards across K samples of the same prompt. |
| Extra model in memory | Yes - the critic, trained alongside the policy. | No critic; the group is the baseline. |
| Compute cost | Higher - critic forward/backward and its own tuning. | Lower memory, but K rollouts per prompt cost sampling. |
| Stability lever | Clipped ratio bounds the policy step. | Group normalization plus KL control. |
GRPO trades a learned critic for more samples per prompt - attractive when sampling is cheaper than carrying a second large network.
Learn more
Full explanation
Q3 - RLHF vs RLVR and two reward hacks in coding
Q3 - RLHF vs RLVR and two reward hacks in coding
- RLHF
- Reward model trained on human preferences; scores fuzzy quality, but is gameable and can drift.
- RLVR
- Verifiable reward from tests, compilers or checkers; hard to fake, but only where a checker exists.
Model special-cases the test inputs or stubs the function to return expected values.
Mitigation: held-out hidden tests, property-based checks, reward only on unseen cases.
If the reward model likes long, comment-heavy code, the policy inflates it without improving correctness.
Mitigation: adversarial eval set, periodic reward-model retraining, KL leash to the reference.
Q4 - Attention from first principles
Walk Q/K/V, the scaled dot-product and why the scale is the square root of d_k.
Attention(Q, K, V) = softmax( Q Kᵀ / sqrt(d_k) ) V # Q·K grows with d_k; without /sqrt(d_k) the logits get large, # softmax saturates and gradients vanish. The scale keeps variance ~1. # Multi-head: run h of these on projected subspaces, concat, project out. # KV cache: at decode, cache past K and V so each new token is O(seq), # not O(seq²) - you only compute Q for the new token.
Learn more
Full explanation
Q5 - Chinchilla and training with less compute
Q5 - Chinchilla and training with less compute
State the compute-optimal intuition and tie it to the role's charter of training at lower compute.
- For a fixed compute budget, model size and training tokens should scale together - roughly in proportion - not pour everything into parameters.
- Earlier large models were under-trained: too many parameters, too few tokens. Chinchilla showed a smaller model on more data beats them at equal compute.
- Heuristic anchor: on the order of ~20 tokens per parameter at the compute-optimal point (an intuition, not a law).
- For Cursor's “less compute” charter, the lever shifts to data quality, difficulty calibration and sample efficiency - squeezing more signal per token rather than buying more FLOPs.
When you answer one of these, attach a Cursor hook in a sentence. After Chinchilla, add that their charter is training at lower compute, so the real lever is datapoint quality and sample efficiency, not raw FLOPs. The fact lands the recall; the hook lands the fit.
You can recite the PPO objective and still fail “why does the clip exist and what happens to the gradient when the ratio leaves the clip range.” Prep the second and third question deep on each item, because that's where a memorized line breaks and a real understanding holds.