open source inference || prev @MSFTResearch
Joined May 2026
- Tweets378
- Following228
- Followers31
- Likes1.7K
Pinned Tweet
new account here :) im an inference engineer and researcher and will be posting about high performance inference systems. if that interests you, do consider a follow :)
posting about underappreciated / unconvential techniques with CUDA.
one genuinely underappreciated technique is branchless top-k sorting using PTX predicate registers to avoid loops and branches
Suppose you have sorted values in registers, a conventional CUDA implementation inserts the new value into the sorted array by comparing individual elements. with setp.gtu.f32 you can create a predicate based on a comparison and select between two values based on that predicate using selp.f32
this paper is lowkey an absolute hidden gem
Not sure whether u have seen our paper lol arxiv.org/abs/2611.02028
pythongiant retweeted
Not sure whether u have seen our paper lol arxiv.org/abs/2611.02028
deep learning inference in libraries like tensort is so unoptimized that it virtually guarantees that if you want to host any model outside of an LLM you would have to so some sorts of kernel engineering to get to real production standards
