james (@http_413) header
james (@http_413) profile picture

develooper / bullish on a post-scarcity, post-suffering world via ai / building @thereabouts_app so you never lose a new place to try

Joined August 2017
Photo by james (@http_413)Photo by james (@http_413)Photo by james (@http_413)Photo by james (@http_413)Photo by james (@http_413)Photo by james (@http_413)Photo by james (@http_413)Photo by james (@http_413)Photo by james (@http_413)Photo by james (@http_413)
james retweeted
what if a model ran some of its steps twice before writing the next bit of text? that's the idea behind looped inference. take sensenova-loop-9b as an example: it has 32 layers but runs 56 layer steps. here's why that matters if you're running it at home.
1
1
27
Anthropic $ANT just rang the opening bell for the stock market today at the NASDAQ exchange
1
149
when someone says “it fits on my gpu,” a pretty good follow-up is “with how much text?” i put a little demo in my ml guide where you can add tokens and watch the cache grow. ml.webfor.fun/transformers#k…
1
4
this is where the hy4 flash model card gets interesting. tencent says its kv cache is 1/24 the size of hy3's and 1/6 the size of hy4's. that leaves more room for a long chat or a second request. you still have to fit the model itself.
2
14
say the prompt is “the sky is” and the model adds “blue.” when it processes “blue,” it reuses the keys and values it already saved for “the sky is” and adds an entry for “blue.” which saves it from doing the same math on the prompt again.
1
11
a model writes one little piece of text (a token) at a time. as it reads each token, it calculates two sets of numbers. the key helps it decide how relevant that token is later. the value carries information it can use from that token. kv = key and value.
1
7
ever load a model, see that it fits, then paste a long document and run out of memory? the model didn't get bigger. the conversation did. one reason is something called the kv cache.
1
14
Replying to @ecombeckett
then i could only stick one peanut up there
18
wait til you see the 15% tax on software and ai subscriptions
14
Replying to @gabrielste1n
thank your mayor and alderpeople
1
71
Replying to @staysaasy
oh no only 5 mil? how will they survive??
101
Replying to @calebfoundry
$100,000,000,000.000000
16
Replying to @25YearsAgoLive
time to liecht en der stein
3
344
at home, you only have to store one copy of the middle layers. but your machine runs them twice for each new token (a piece of text). that usually means fewer tokens per second than running those layers once on the same setup.
1
6
why does doing the same thing twice help? think kneading dough. same hands, same motion, but the dough's already changed. the same layers can build on the work they just did. sensenova's team found that more passes helped on the tasks they tested, up to a point.
1
2
sensenova runs the first 4 layers, then the middle 24, then those same 24 again, then the last 4. that's 56 steps. the arrow between the two middle boxes is the whole trick. whatever comes out of pass one goes straight into pass two.
1
10