These very helpful data/assumptions are from this presentation: youtube.com/watch?v=ZY9MUhfu…
46
Putting out to Twittersphere (maybe @SemiAnalysis_ can shed light 🙏). How did they get to a cost of ownership of $1.66 per GPU-hour? Many variables. I can get close assuming useful life of 3.5 yrs, but not sure that's the right assumption.
1
1
75
These guys seem to know their semi sh*t, but I can’t verify that because information like this isn’t that accessible to a layman. (I do get the key takeaway about CUDA)
TogetherAI's Chief Scientist @tri_dao announced Flash Attention v4 at HotChips Conference which is up to 22% faster than the attention kernel implementation from NVIDIA's cuDNN library. Tri Dao was able to achieve this 2 key algorithmic changes. Firstly, it uses a new online softmax algo to skip 90% of output rescaling. Secondly, in order to better overlap the softmax computation with the tensor core computation, it uses software emulation of exponential (MUFU.EX2) to boost throughput.
This is the key advantage of the CUDA moat, is developers like Tri Dao implement algorithmic breakthroughs on CUDA. Tri Dao or any of his researchers does not use ROCm AMD GPUs or Trainium chips. Furthermore, Flash Attention v4 uses CUTLASS CuTe Python DSL which is 10x harder to port to ROCm HIP, verus CUDA C++ is easilier to port ROCm HIP.
133







