Jay (@incrementalROI) header
Jay (@incrementalROI) profile picture

"The money money makes, makes more money" - Ben Franklin. Striving for equanimity and clear headedness | Not investment advice

Joined May 2018
this is you guys
1
60
Replying to @WaterworldCapi1
Three. More. Years.
5
163
L/S momentum basket. You mfers can't help yourselves.
1
3
1,222
Putting out to Twittersphere (maybe @SemiAnalysis_ can shed light 🙏). How did they get to a cost of ownership of $1.66 per GPU-hour? Many variables. I can get close assuming useful life of 3.5 yrs, but not sure that's the right assumption.
1
1
75
These guys seem to know their semi sh*t, but I can’t verify that because information like this isn’t that accessible to a layman. (I do get the key takeaway about CUDA)
TogetherAI's Chief Scientist @tri_dao announced Flash Attention v4 at HotChips Conference which is up to 22% faster than the attention kernel implementation from NVIDIA's cuDNN library. Tri Dao was able to achieve this 2 key algorithmic changes. Firstly, it uses a new online softmax algo to skip 90% of output rescaling. Secondly, in order to better overlap the softmax computation with the tensor core computation, it uses software emulation of exponential (MUFU.EX2) to boost throughput. This is the key advantage of the CUDA moat, is developers like Tri Dao implement algorithmic breakthroughs on CUDA. Tri Dao or any of his researchers does not use ROCm AMD GPUs or Trainium chips. Furthermore, Flash Attention v4 uses CUTLASS CuTe Python DSL which is 10x harder to port to ROCm HIP, verus CUDA C++ is easilier to port ROCm HIP.
133
The perfect gif for TSL@ skeptics
1
45
Replying to @WaterworldCapi1
you right now
2
237