abstract
The paper introduces Generalized Sequence Policy Optimization (GSPO), a method that addresses a fundamental mismatch in how Group Relative Policy Optimization (GRPO) applies off-policy corrections. GRPO computes a separate importance ratio for every token and clips each independently. But in typical RL setups, the reward is given at the sequence level — the entire response gets one score. GSPO argues that the importance weighting should match the reward granularity: one ratio per sequence, not per token.
sequence-level clipping
Sequence-level clipping means computing one importance ratio for an entire response and clipping that single value, rather than clipping a separate ratio for every token.
In GRPO, each token in a response gets its own likelihood ratio $w_t = \frac{\pi_\theta(a_t|s_t)}{\pi_{\theta_\text{old}}(a_t|s_t)}$, and each ratio is clipped independently to $[1-\epsilon, 1+\epsilon]$. GSPO instead aggregates all token ratios into a single sequence-level ratio, clips once, and applies that one clipping decision to the entire response.
a concrete example: 17 × 23
A simple example:
Response has 100 tokens. 99 token ratios are near $1$, but one token's ratio is unusually large.
Token-level clipping: that one token may be clipped independently, while the other 99 are treated differently.
Sequence-level clipping: the ratios are aggregated into one response-level ratio; that ratio determines whether the response's contribution is clipped, and the tokens are weighted uniformly.
The "length-normalized" part is important: GSPO uses the geometric mean of per-token likelihood ratios, rather than their raw product. Without normalization, longer responses would naturally produce much larger or smaller ratios and would require different clipping ranges.
Normalization helps by making the sequence ratio scale-invariant to response length. The geometric mean $( \prod_{t=1}^T w_t )^{1/T}$ keeps the ratio in a consistent range regardless of whether the response has 10 tokens or 1,000 tokens. This means the same clipping threshold $\epsilon$ works across all response lengths.
where token-level correction creates a mismatch
They are saying that if the reward is given to the whole sequence, applying off-policy correction at token level seems problematic. In GRPO, each query produces a group of rollouts, each rollout receives one sequence-level reward, and that reward-derived advantage is copied to every token in that rollout. The problem is that GRPO then applies a different off-policy ratio to each token, even though the reward does not distinguish between those tokens.
Suppose the query is:
"Solve $17 \times 23$ and explain your reasoning."
The old policy generates three rollouts:
Rollout Response Reward y^1 "17 × 23 = 391. First, 17 × 20=340, then 17 × 3=51, so 340+51=391." 1 y^2 "17 × 23 = 381." 0 y^3 "17 × 23 = 392." 0
Using group-relative normalization, the correct rollout gets a positive advantage and the incorrect rollouts get negative advantages. For simplicity, let us write them as:
$$ \hat A^1=+1,\qquad \hat A^2=-0.5,\qquad \hat A^3=-0.5. $$
The important point is that every token in $y^1$ gets $\hat A^1=+1$, including:
"17" "×" "23" "=" "391" the explanation tokens
Likewise, every token in $y^2$ gets $\hat A^2=-0.5$. This is sensible if the reward evaluates the response as a whole: the rollout is correct or incorrect, rather than only one particular token being correct.
where token-level correction creates a mismatch
Now suppose that, for the correct rollout $y^1$, the new policy's token-level likelihood ratios are:
Token Token ratio w_t "17" 1.05 "×" 0.98 "23" 1.02 "=" 1.40 "391" 0.95
Because $\hat A^1=+1$ for every token, the response is being rewarded overall. But the individual token ratios make its contribution uneven:
- the "=" token gets a large weight, $1.40$;
- "391" gets a lower weight, $0.95$;
- the other tokens receive still other weights.
With clipping, the differences remain: some tokens may be clipped while others are not. Thus, the same positive sequence-level reward can produce a gradient that strongly favors or suppresses particular tokens based on their local likelihood ratios — even though the reward provided no token-level information saying that "=" deserves more credit than "391".
That is the mismatch the paper is pointing to: sequence-level credit is being combined with token-level importance weighting. The authors argue that these unequal token weights can accumulate across long responses and introduce high-variance noise.
gspo-token: a middle ground
GSPO-token is a middle ground: it still uses a sequence-level importance ratio, but permits different token-level advantages when the task genuinely provides finer-grained or multi-turn credit information.
For ordinary single-turn GRPO-style training: if the only reward is for the completed rollout, then the natural advantage is the same for every token. GSPO-token is an optional variant.
multi-turn and finer-grained credit
The reason for introducing GSPO-token is that some RL setups may have more detailed credit information than one reward for the whole final answer — for example, multi-turn interactions where each turn receives a separate reward, or a method that estimates which parts of a response contributed more.
In GSPO-token, step 1's tokens can have advantage $0.7$, step 2's tokens can have advantage $-0.3$, while the clipping decision is still made once for the entire rollout.
how gspo counts clipped tokens differently
The important GSPO distinction is that one clipped sequence can make every token in that response count as clipped. Suppose a rollout has 1,000 tokens and its sequence-level ratio is outside the clipping range. GSPO may count all 1,000 tokens in that rollout toward the clipping fraction, even though it made only one clipping decision. This is why Section 5.2 can report a much larger fraction of clipped tokens for GSPO than for GRPO. That is also why "clipping more tokens" does not necessarily mean GSPO is using less data. It means more token positions are in saturated/clipped regions, yet GSPO can still train better because its sequence-level weighting is less noisy than GRPO's independent token-level weighting. The paper explicitly describes this counterintuitive result: GSPO clips a much larger fraction of tokens but still achieves higher training efficiency.
empirical observation
The paper shows that despite clipping a larger fraction of tokens, GSPO achieves better training efficiency and stability compared to GRPO. The sequence-level approach reduces gradient variance without sacrificing learning signal.
routing replay: a workaround for grpo with moe
Routing Replay is a workaround for GRPO's instability when the policy is a Mixture-of-Experts (MoE) model: it forces the new policy to use the same experts that the old policy used when generating each rollout.
what routing replay does
Imagine that the old policy generated a response using the following routing:
Token Experts used by old policy "The" E₂, E₇ "answer" E₁, E₅ "is" E₃, E₈
During the GRPO update, the new policy might naturally choose:
Token Experts naturally chosen by new policy "The" E₂, E₆ "answer" E₄, E₅ "is" E₃, E₉
With Routing Replay, the training system caches the old policy's routing decisions and forces the new policy to reuse them:
Token Experts used for both policies during ratio calculation "The" E₂, E₇ "answer" E₁, E₅ "is" E₃, E₈
The paper's Figure 3 shows that GRPO with Routing Replay converges normally, while GRPO without it fails to converge properly in their MoE experiment.
why gspo doesn't need routing replay
The key difference is that GSPO cares about the likelihood of the whole response, not the individual likelihood ratio of every token. Individual routing changes can cause some token probabilities to fluctuate, but these fluctuations are less damaging when summarized into a length-normalized sequence likelihood.
Suppose a response has four tokens. The new policy's token-level ratios are:
$$ 1.20,\quad 0.80,\quad 1.15,\quad 0.90. $$
GRPO uses these separately. Each token can receive a different weight, and the effects can interact unpredictably with the token-level advantage.
GSPO combines them into one response-level value:
$$ s_i = (1.20 \times 0.80 \times 1.15 \times 0.90)^{1/4}. $$
The exact value is less important than the principle: GSPO makes one sequence-level clipping and weighting decision instead of allowing each routing-sensitive token ratio to independently affect the update.
numerical precision and engine discrepancies
GRPO uses a separate ratio for every token. Suppose the true old-policy probability of a token is $0.200$, but the inference and training engines produce slightly different values:
$$ 0.2000 \quad\text{versus}\quad 0.2010. $$
That looks like a small absolute difference, but the ratio changes from, for example,
$$ \frac{0.2100}{0.2000}=1.05 $$
to
$$ \frac{0.2100}{0.2010}\approx1.045. $$
For one token, this may not matter much. But GRPO makes a separate clipping and weighting decision for every token. Across a long response, many small discrepancies can cause individual token ratios to cross clipping boundaries or produce noisy token-level gradients.
GSPO aggregates the token likelihoods into one length-normalized sequence ratio. The result is still sensitive to likelihood errors — "tolerant" does not mean "immune" — but it is less dependent on every individual token ratio being perfectly reproduced across engines. The paper describes sequence likelihood as "much more tolerant of precision discrepancies" for this reason.
inference engine likelihoods vs training engine
The inference engine already generates each token and can usually return its log-probability under the policy used for generation. According to the paper, this makes it possible to use likelihoods returned directly by the inference engine, rather than sending the rollout back through the training engine just to recompute them.
GRPO:
inference likelihoods → individual token ratios → token clipping
GSPO:
inference likelihoods → aggregate sequence ratio → sequence clipping
In principle, the inference engine could return the old-policy log-probabilities when it generates the response. That is often the natural thing to do. However, this paper says that in practice, people often use the training engine to recompute the old-policy likelihoods. Why? The inference and training engines may differ in numerical precision, kernels, parallelization, or implementation. If the old-policy likelihood comes from the inference engine but the current-policy likelihood comes from the training engine, the resulting ratio may contain a discrepancy caused by the engines rather than by the policy update. Engine Discrepancies
Thus the more conservative GRPO workflow is:
Inference engine: generate response Training engine: evaluate old policy on response evaluate current policy on response compute token-level ratios compute GRPO loss
The old policy may be stored as a checkpoint or otherwise available to the training engine for this recomputation.
The paper's claim should be understood carefully. It does not mean that the inference engine alone performs the entire training update or that the training engine is unnecessary. The training engine still computes the current-policy forward pass and gradients.
Rather, GSPO makes it possible to reuse the likelihood statistics returned by the inference engine — especially the old-policy likelihoods — instead of recomputing them with the training engine:
GSPO: Inference engine: generate response return old-policy token log-probabilities Training engine: compute current-policy log-probabilities aggregate both sets of log-probabilities form one sequence ratio compute loss and gradients
The inference engine still may return token-level values, because summing token log-probabilities is the usual way to obtain a sequence likelihood. But those values are immediately aggregated.