This repository contains a compact experimental study comparing Group Relative Policy Optimization (GRPO) and Group reward-Decoupled / Dominance-Based Policy Optimization (GDPO) for multi-objective reinforcement learning in language models.
The project is inspired by the GDPO research paper and explores whether dominance-aware optimization leads to more stable and balanced learning than scalarized reward optimization, particularly in small models trained under noisy reward signals.
We train a small instruction-tuned language model on a multi-objective arithmetic task where each output is evaluated across three competing reward dimensions:
- Correctness — whether the predicted number is accurate
- Brevity — whether the output remains short
- Formatting — whether the output is a clean numeric string
GRPO optimizes a weighted scalar reward, collapsing multiple objectives into a single value. GDPO instead compares rollouts using Pareto dominance, avoiding scalarization and preserving tradeoff structure across objectives.
The experiments analyze how these different optimization strategies affect:
- Training stability
- Reward oscillation and regressions
- Multi-objective tradeoff preservation
- Pareto frontier structure in policy behavior
GRPO frequently oscillates between objectives, over-optimizing whichever reward dominates the scalar signal at a given time. GDPO reduces destructive tradeoff collapse by favoring updates that improve one objective without harming others, aligning more closely with Pareto-optimal learning.
While benefits depend on reward quality and objective conflict, GDPO shows more stable multi-objective behavior in noisy, low-capacity settings.
GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-Reward RL https://arxiv.org/abs/2601.05242