Matt

I’m not going to comment on the future of PPO vs GRPO, but I find the specific line of reasoning “critics stabilize long-horizon learning” a bit unrigorous. Often it’s justified with some informal language like “critics perform credit assignment, which means only the tokens that matter get trained on”. But if you actually try to formalize this, the statement is much closer to “TD learning increases tractability of long-horizon learning” which is both obvious and orthogonal. The more relevant question then is not “did they estimate advantage using MC or function approximation”, but rather “what did they set lambda to in GAE”, which is a much more interesting question. If you set lambda = 1, it’s not clear that the critic is a groundbreaking improvement. You’re exposed to reward-to-go noise either way. This seems more like a property of policy gradient than of MC vs function approximation. On the other hand, lambda < 1 is a bet against scale, because bias doesn’t wash out with scale the way that variance does. @MillionInt talks about this. This is why, for example, GRPO does in fact work across 1000s of turns and 10s-100s of compactions. If variance scaling with horizon is your biggest concern (not obvious that it should be), there are algorithms that are actually unaffected by horizon. Evolutionary Strategies (ES) comes to mind, which has variance scale instead with parameter subspace dimension (e.g. low-rank mutations have lower dimension).