The policy-gradient objective

The policy-gradient objectiveREINFORCE

You want to maximize expected reward over sequences the policy generates. You can't differentiate through sampling, so the log-derivative trick rewrites the gradient as something you can estimate from samples.

J(θ)   = E_{τ ~ π_θ} [ R(τ) ]            # expected reward over trajectories
∇J(θ) = E_{τ ~ π_θ} [ R(τ) · ∇ log π_θ(τ) ]

# In words: nudge each token's log-probability UP if the reward
# was high, DOWN if it was low. The reward does the supervising.

Read that gradient out loud in an interview: it is a reward-weighted log-likelihood. High-reward samples get reinforced, low-reward samples get suppressed and the magnitude scales with reward. The whole field is variations on making this estimator less noisy. Here is the whole loop in one picture; the next two subsections fill in the weighting and the leash.