The policy-gradient objective

The policy-gradient objectiveREINFORCE

You want to maximize expected reward over sequences the policy generates. You can't differentiate through sampling, so the log-derivative trick rewrites the gradient as something you can estimate from samples.

J(θ)   = E_{τ ~ π_θ} [ R(τ) ]            # expected reward over trajectories
∇J(θ) = E_{τ ~ π_θ} [ R(τ) · ∇ log π_θ(τ) ]

# In words: sample sequences, then for each one nudge the log-probability
# of every token UP if the reward was high, DOWN if it was low.
# No labels - the reward does the supervising.

Read that gradient out loud in an interview: it is a reward-weighted log-likelihood. High-reward samples get reinforced, low-reward samples get suppressed and the magnitude scales with reward. The whole field is variations on making this estimator less noisy.