This is a nice write up. I will be referencing it later.
For people who are less familiar, RL is powerful especially for long horizon tasks with verifiable rewards but it uses more memory because you need to actually calculate a bunch of “roll outs” with your model.
For example in GRPO which is what Deepseek R1 used, the G stands for “group” and the stability of this update method increases as group size increases. But each member of the group is one roll out. So you’re trading quality for speed.
One idea is to run the roll outs at a lower precision, but the problem is that it means your predictions are less accurate and the model updates can diverge.
So a solution to that problem is to run the forward pass in low precision and the backwards pass in high precision, plus adding some guardrails to make sure we don’t lose too much information.
For people who are less familiar, RL is powerful especially for long horizon tasks with verifiable rewards but it uses more memory because you need to actually calculate a bunch of “roll outs” with your model.
For example in GRPO which is what Deepseek R1 used, the G stands for “group” and the stability of this update method increases as group size increases. But each member of the group is one roll out. So you’re trading quality for speed.
One idea is to run the roll outs at a lower precision, but the problem is that it means your predictions are less accurate and the model updates can diverge.
So a solution to that problem is to run the forward pass in low precision and the backwards pass in high precision, plus adding some guardrails to make sure we don’t lose too much information.
https://news.ycombinator.com/newsguidelines.html#generated