This is misleading from the lens of practical model architecture research. Some clear limitations that weaken the conclusion: - While SWA can inherit weights learned by full attention, you don't assume the same for linear attention and train them on top of the QKV proj matrices learned by full attn with zero ablations on whether this is close to optimal (in fact this training method is far from optimal). - Apart from how you train it, the linear attn variants tested here are also years behind the current sota. No frontier model uses pure linear attn. It has repeatedly been shown that the optimal config is a hybrid of linear and full, which beats both pure linear and pure full - e.g. Kimi Linear's architecture with 3 kimi delta attention to 1 full attn is, to date, still by far the best-performing arch among the ones I've tried. And yes, SWA hybrids were included in the comparison. - "Training linear attention models is expensive and most software/hardware is made for Softmax attention" This is untrue. There are many fast open-source kernels written for linear attn models whose speeds are even faster than training full attn models under fair context length. Speaking of which, I don't see a training sequence length specified in the paper. - I understand the limited compute. You don't have to do from-scratch or large scale training, but even under this switching setting a principled work still need to properly compare the variants. Overall, I think it is important to have a full understanding of the current state of method X before claiming its false inferiority. If one lacks the compute/infrastructure/data/etc., open research is still very appreciated, but overly strong and unprincipled claims could be misleading.