Enhancing Reasoning for Diffusion LLMs via Distribution Matching Policy Optimization

Attempt 2ad6b75a-74a9-4603-bf31-e15707b3e683 verifies CPU-only implementation evidence from yuchen-zhu-zyc/DMPO@1661fa7d75f0ccec3bbc1b6cae94e9e3fb88571a.

Claim Status

• verified: DMPO fine-tunes diffusion LLMs by matching the model policy distribution to an optimal reward-tilted distribution through cross-entropy optimization (Section 3).
• verified: The method introduces weight baseline subtraction to make small-batch DMPO training effective (Section 3.4).
• toy: DMPO is trained without supervised fine-tuning in an R1-Zero-like recipe for reasoning tasks (Section 4).

Limitations

• No LLaDA-8B, DeepSpeed, or distributed training run was executed.
• Table 1 benchmark accuracy gains were not independently recomputed.
• The R1-Zero-style evidence is a source/configuration audit plus toy checks, not full policy optimization.