Sahil Thaker (@_sahilt) header
Sahil Thaker (@_sahilt) profile picture

♥️ Machines, Math, Sports, Music. Co-Founder @hyperline_xyz Past: @Glean, @Facebook News Feed AI, Video AI, @Uber marketplace optimization

San Francisco, CA
Joined January 2014
Photo by Sahil Thaker (@_sahilt)Photo by Sahil Thaker (@_sahilt)Photo by Sahil Thaker (@_sahilt)Photo by Sahil Thaker (@_sahilt)Photo by Sahil Thaker (@_sahilt)Photo by Sahil Thaker (@_sahilt)Photo by Sahil Thaker (@_sahilt)Photo by Sahil Thaker (@_sahilt)
Sahil Thaker retweeted
IMO, just as today we think of codegen AI as a 10x productivity multiplier for CS degrees (or equivalent), in a year or two we will be thinking of CS degrees as 10x productivity multipliers over codegen AI.
Replying to @mlevchin
do you think AI can teach people to code or they should still go through books, courses and school?
22
8
7
261
47,100
Sahil Thaker retweeted
I took 100g of creatine and now i’m doing LLM inference in my head
119
363
45
8,894
389,571
Sahil Thaker retweeted
TL;DR: We built a transformer-based payments foundation model. It works. For years, Stripe has been using machine learning models trained on discrete features (BIN, zip, payment method, etc.) to improve our products for users. And these feature-by-feature efforts have worked well: +15% conversion, -30% fraud. But these models have limitations. We have to select (and therefore constrain) the features considered by the model. And each model requires task-specific training: for authorization, for fraud, for disputes, and so on. Given the learning power of generalized transformer architectures, we wondered whether an LLM-style approach could work here. It wasn’t obvious that it would—payments is like language in some ways (structural patterns similar to syntax and semantics, temporally sequential) and extremely unlike language in others (fewer distinct ‘tokens’, contextual sparsity, fewer organizing principles akin to grammatical rules). So we built a payments foundation model—a self-supervised network that learns dense, general-purpose vectors for every transaction, much like a language model embeds words. Trained on tens of billions of transactions, it distills each charge’s key signals into a single, versatile embedding. You can think of the result as a vast distribution of payments in a high-dimensional vector space. The location of each embedding captures rich data, including how different elements relate to each other. Payments that share similarities naturally cluster together: transactions from the same card issuer are positioned closer together, those from the same bank even closer, and those sharing the same email address are nearly identical. These rich embeddings make it significantly easier to spot nuanced, adversarial patterns of transactions; and to build more accurate classifiers based on both the features of an individual payment and its relationship to other payments in the sequence. Take card-testing. Over the past couple of years traditional ML approaches (engineering new features, labeling emerging attack patterns, rapidly retraining our models) have reduced card testing for users on Stripe by 80%. But the most sophisticated card testers hide novel attack patterns in the volumes of the largest companies, so they’re hard to spot with these methods. We built a classifier that ingests sequences of embeddings from the foundation model, and predicts if the traffic slice is under an attack. It leverages transformer architecture to detect subtle patterns across transaction sequences. And it does this all in real time so we can block attacks before they hit businesses. This approach improved our detection rate for card-testing attacks on large users from 59% to 97% overnight. This has an instant impact for our large users. But the real power of the foundation model is that these same embeddings can be applied across other tasks, like disputes or authorizations. Perhaps even more fundamentally, it suggests that payments have semantic meaning. Just like words in a sentence, transactions possess complex sequential dependencies and latent feature interactions that simply can’t be captured by manual feature engineering. Turns out attention was all payments needed!
175
544
236
5,189
1,304,170
Sahil Thaker retweeted
FROM MAGA TO CHINA Here are four things MAGA is getting wrong, and why it's handing over the world to China. (1) First, MAGA correctly understands that America’s economic position is in decline but thinks this is due to economic competition itself, rather than lack of competitiveness. (2) Second, MAGA also understands that the US has wasted trillions abroad in foreign wars, but thinks the problem is global leadership itself rather than poor leadership. (3) Third, MAGA knows that their Blue American enemies have allies abroad, but has incorrectly overreacted to this by treating every non-Red-American as an enemy. (4) Fourth, MAGA sees the billions of dollars flowing from the US to foreign recipients, but isn't grasping that the US can only print those dollars in the first place so long as it's the hub of a global empire. When you put these together you can both understand MAGA's actions and understand why they will not lead to the intended result. Basically: MAGA is hyperfocused on cutting off any apparent flow of funds from Red Americans to Blue Americans and non-Americans. And they only have ~500 days in power. So they're trying to quickly shut off imports, close down institutions, and exit all wars. OK. Except the reason the imports exist in the first place is because US products aren't competitive relative to Chinese products (or Fed printing). The reason those institutions exist is because the US set them up to run the world. And the reason those wars are happening is not because of American leadership per se, but because of the absence of good leadership. If you shut all of that down at once — if you abandon global competition and global leadership — you shut down American Empire, and with it the ability to print money. And then everyone in that empire has a very bad time.
803
2,068
511
13,578
2,975,335
one way to think about ai: its humanity's collective brainpower over time i think we'll get experts and fragments to this central brain, but directionally its "collective intelligence"
I just asked OpenAI Deep Research to analyze some fresh & highly complex unpublished experimental data that we couldn’t decipher, related to an immune cell. When I read its report, I was absolutely stunned!😳 It solved the issue in the most profound way possible🤯Speechless🥹
1
527
Came across this neat tool with viz to dig into llm internals: bbycroft.net/llm
3
202
Great to see meaningful usecases of reasoning models
Something amazing happened with the @deepseek_ai model! DeepSeek R1 just generated a highly innovative idea for a cancer project I’m working on😮 This is the first time an LLM, other than o1 models, has produced such creative ideas! I’m now using it side by side with o1-pro!
2
376
Sahil Thaker retweeted
The (true) story of development and inspiration behind the "attention" operator, the one in "Attention is All you Need" that introduced the Transformer. From personal email correspondence with the author @DBahdanau ~2 years ago, published here and now (with permission) following some fake news about how it was developed that circulated here over the last few days. Attention is a brilliant (data-dependent) weighted average operation. It is a form of global pooling, a reduction, communication. It is a way to aggregate relevant information from multiple nodes (tokens, image patches, or etc.). It is expressive, powerful, has plenty of parallelism, and is efficiently optimizable. Even the Multilayer Perceptron (MLP) can actually be almost re-written as Attention over data-indepedent weights (1st layer weights are the queries, 2nd layer weights are the values, the keys are just input, and softmax becomes elementwise, deleting the normalization). TLDR Attention is awesome and a *major* unlock in neural network architecture design. It's always been a little surprising to me that the paper "Attention is All You Need" gets ~100X more err ... attention... than the paper that actually introduced Attention ~3 years earlier, by Dzmitry Bahdanau, Kyunghyun Cho, Yoshua Bengio: "Neural Machine Translation by Jointly Learning to Align and Translate". As the name suggests, the core contribution of the Attention is All You Need paper that introduced the Transformer neural net is deleting everything *except* Attention, and basically just stacking it in a ResNet with MLPs (which can also be seen as ~attention per the above). But I do think the Transformer paper stands on its own because it adds many additional amazing ideas bundled up all together at once - positional encodings, scaled attention, multi-headed attention, the isotropic simple design, etc. And the Transformer has imo stuck around basically in its 2017 form to this day ~7 years later, with relatively few and minor modifications, maybe with the exception better positional encoding schemes (RoPE and friends). Anyway, pasting the full email below, which also hints at why this operation is called "attention" in the first place - it comes from attending to words of a source sentence while emitting the words of the translation in a sequential manner, and was introduced as a term late in the process by Yoshua Bengio in place of RNNSearch (thank god? :D). It's also interesting that the design was inspired by a human cognitive process/strategy, of attending back and forth over some data sequentially. Lastly the story is quite interesting from the perspective of nature of progress, with similar ideas and formulations "in the air", with a particular mentions to the work of Alex Graves (NMT) and Jason Weston (Memory Networks) around that time. Thank you for the story @DBahdanau !
132
974
145
6,641
866,442
Deepmind chess paper showed that deterministic software can be mapped to neural nets arxiv.org/abs/2402.04494 This allows us to start from existing knowledge, but design the system to self-improve it through experience
Replying to @sytelus
Now the time for that bigger insight: Stockfish engine is littered with all kind of complexities built with traditional computational primitives of conditionals, branching, data movements and so on. What DeepMind paper points to is that all of these can potentially be distilled down to mere matrix multiplications! There is something so utterly bewildering here. I think we are on the precipice of entire computer science being rewritten from its origin of Turing machines.
6
980
The benefits of caloric restriction for primate lifespans (including our own!) are probably overrated. We have two rhesus monkey studies, and they do not support big benefits🧵 First, take a look at this diagram:
23
37
8
416
104,868
Sahil Thaker retweeted
What’s the difference between reason, intelligence and understanding? Reason is all methods of criticism that improve ideas. Intelligence is how fast we create ideas that solve problems. Understanding is the ability to address criticisms of an idea. Rationality is seeking to improve ideas and using the best ones. Judgment is anticipating the long-term consequences of actions. Conjecture is the unexplained process of creating ideas. Creativity is making ideas that are not predetermined by the evolution of a system. Science is ideas that can be tested through repeatable experiments. Knowledge is ideas that are useful. Did I miss anything?
32
19
2
138
10,025
Sahil Thaker retweeted
“Experience is what you get when you didn't get what you wanted.” “When you're screwing up and nobody's saying anything to you anymore, that means they gave up.” Video: youtube.com/watch?v=ji5_Mqic… Book: amazon.com/Last-Lecture-Rand… 2/4
1
8
1
174
12,566
If you set reasonable milestones, you will only make reasonable progress. unreasonable people drive to make unreasonable progress.
The first Starships to Mars will launch in 2 years when the next Earth-Mars transfer window opens. These will be uncrewed to test the reliability of landing intact on Mars. If those landings go well, then the first crewed flights to Mars will be in 4 years. Flight rate will grow exponentially from there, with the goal of building a self-sustaining city in about 20 years. Being multiplanetary will vastly increase the probable lifespan of consciousness, as we will no longer have all our eggs, literally and metabolically, on one planet.
2
1
9
448
Sahil Thaker retweeted
Elon Musk open-sourced all of Tesla's patents for other car companies to copy. Wall Street called it 'dumb' and thought Tesla will be crushed. Elon was right. Everyone was wrong. Every entrepreneur needs to understand why he did it and why it worked:🧵
1,474
4,197
445
35,941
7,238,632
I'm shocked. Just shocked that continually averaging a data corpus without exogenous inputs results in degraded information quality. Doomer ouroboros arguments were always silly. But they get demonstrably sillier as the research continues. nature.com/articles/s41586-0…
13
12
3
179
22,778
My google exp reinforced a few learnings for me: (1) consumers buy products; enterprises buy platforms. (2) distribution advantages overtake product / tech advantages and (3) companies that reach PMF & then under-invest in S&M risk staying niche players or worse: get taken down.
1
29
3
218
18,366
200x compression is mathematically impossible for this data. Given the specs of Neuralink's chip, you cannot do better than 5.3x compression. Here's why 🧵 (1/10)
Neuralink started a compression challenge. They're asking for people to find methods to losslessly (!!) compress files to 1/200th (!) the size but the files are extremely noisy so thats certainly not possible 🙁 (spectrogram of one randomly chosen file for illustration)
39
51
12
799
519,079
This MIT research as well as anthropics monosemantic paper discovered that LLMs converge to same concept representations across models and modalities. wow! reminds us of word2vec analogies. I believe this is the root of observed intelligence in LLMs
What makes up the abstract concept of an apple? We read the word "apple" as a string, see 2D pictures online, 3D shape in real life, and moving apples in videos. We touch the apple, feel its geometry in our palms and texture through the rich tactile sensation on our fingers. Do all these different modalities converge to the same representation space, given sufficient learning capacity? After all, they are all shadows of one "true reality" projected onto our different senses. I really like this study paper from MIT, "The Platonic Representation Hypothesis". The authors show that highly capable LLMs and vision models learn very similar representations, even though the modalities are never explicitly co-trained. Concretely, the experiments compare the similarity between strings "apple" and "orange" with the similarity between a picture of an apple vs orange. These two turn out to agree with each other in a wide selection of off-the-shelf models. Reminds me a bit of the famous word vector experiment: f("king") - f("queen") ~= f("man") - f("woman"). But this time, the equation holds across modality, datasets, and completely separate training pipelines. I would love to see this study extended to more modalities: how about 3D point clouds? Speech? Sound of a lion vs a bird? Tactile signals on a robot hand? Can we train meta-models that transfer to *unseen* modalities at test time, since it's just another projection of reality from a different angle?
1
1
4
241
'values are only interesting to the extent that the opposite of your value is also viable'
3
97
'every exponential (system) is a sigmoid in disguise' is such a profound observation
1
87