What Is a Policy Gradient?

Policy Gradient Explainer

A policy-gradient method directly optimizes a parameterized policy. Instead of first learning action values and then choosing the best action, it adjusts the parameters in a direction estimated to increase expected cumulative reward.

The policy-gradient theorem expresses the objective gradient using sampled actions and returns. An action that led to a high return is made more likely in similar states, while lower-return behavior receives less support.

Direct policy optimization works naturally with stochastic policies and continuous action spaces. However, sampled returns can produce high-variance gradient estimates, making training unstable or inefficient.

Baselines reduce variance without changing the expected gradient when used correctly. Advantage estimates, larger batches, reward scaling, entropy bonuses, and trust-region constraints are common tools, but results still depend strongly on environment design and evaluation.

A policy-gradient method adjusts policy parameters in the direction estimated to increase expected cumulative reward.

Key terms include policy, policy parameter, expected return, policy-gradient theorem, trajectory, stochastic policy, baseline, advantage and variance.

The agent runs its policy, records actions and rewards, and uses those samples to estimate how parameters affected return.

Because the estimate comes from random trajectories, repeated samples are needed to control noise.

A baseline separates whether an outcome was better or worse than expected, producing a more focused learning signal.

Advantage estimation and constrained updates can improve stability, but they introduce settings that require careful validation.

Explore more "Explainers"

Discover additional explainers across politics, science, business, technology, and other fields. Each explainer breaks down a complex idea into clear, everyday language—helping you better understand how major concepts, systems, and debates shape the world around us.