Clayton Kim retweeted
We are hiring a Senior Clinical Specialist, AI Systems at @function - jobs.gem.com/function-health…
Clayton Kim retweeted
SITUATION EXPLAINED: The limits of the transformer architecture.
We asked @alex_whedon and Justin Dangel, co-founders of @subquadratic
"Over the last nine years, most of the innovation has come from spending massive amounts of capital, compute, and data on the same algorithm, the transformer algorithm, and it has scaling laws of big O of N squared, which if you've been through software engineering, that's a failing grade."
"This inefficiency is baked into the algorithm. The more data you send to the model, the less efficient it gets. It limits the model's ability to reason over large amounts of data, limits the amount of training we can do on a fixed amount of compute, and creates skyrocketing costs."
"The opportunity is to transition from dense attention, the need to compare every token to every other token, to sparse attention, a more intelligent system that only looks at the tokens that are necessary."
Clayton Kim retweeted
Rick Rubin already wrote the playbook on tech roles
Experimenters
1. Slop cannon prototypers
2. Product builders
Finishers
3. Sweepers
4. Growers
5. Maintainers
As engineering, product, design, DS, etc. melt into a new kind of role, I was reflecting on what roles might look like in the future. For example, when I look at the Claude Code team I see what I think is five archetypes:
1. Prototyper: comes up with brand new ideas; churns out many ideas, most of which don't ship
2. Builder: quickly turns a prototype/idea into production-grade product/infra
3. Sweeper: cleans up the UI, simplifies the code and system, unships, optimizes performance
4. Grower: takes a product that has been built and iterates on it to improve Product-Market Fit
5. Maintainer: owns a mature system to make it secure, reliable, fast, and efficient as it scales
Many people span across 2 roles, and sometimes 3 roles. I also notice that these roles are not really tied to job function -- eg. across Anthropic, some designers match category 1, some 2, some 3; same for engineers, PM, DS.
A healthy team needs a mix of these, depending on the product:
- A product that is new and pre-PMF needs people that are strong at 1+2+3
- A product that is growing and has found PMF needs 2+3+4 and some 5
- A product that has strong PMF needs 3+4+5 and some 2
Maybe product roles of the future will look more like this, and less like the domain-specific roles of today?
Clayton Kim retweeted
ngl it’s kinda wild that China is the land of hypercapitalist competition fueled by open-weight models anyone can use, and America is the land where the executive branch of government must personally approve you to have the privilege of giving a private company your money
Clayton Kim retweeted
Join Subquadratic in SF for a casual gathering this Saturday (link in comments)!
We are a foundation model company building the most compute-, memory-, and sample-efficient foundation model architectures for the next era of AI computing! We love to chat about challenging base assumptions of the industry.
We are hiring folks to work on large-scale pre- and post-training, long-context modeling, model architectures beyond attention, world models, efficient inference and training, and more.
Clayton Kim retweeted
I am hoarding these for the upcoming GPT-5.6 release like health potions before a boss fight.
ALT A colorful interface prompts to reset usage limits, featuring programming code in the background and a button for confirmation.
Clayton Kim retweeted
@AppenResearch independently evaluated @subquadratic's SubQ 1.1 Small Preview on NIAH (1M–12M tokens) and LiveCodeBench (1,055 problems).
Results: 100% NIAH retrieval at 1M–2M, 98% at 12M, 89.7% pass@4 on LiveCodeBench.
Public brief: appen.com/whitepapers/subqua…
Clayton Kim retweeted
Here is the technical report on SubQ 1.1 Small.
subq.ai/subq-1-1-small-techn…
This is the second iteration on our Subquadratic Sparse Attention (SSA) model, and the first to be deployed with design partners in the coming weeks.
The results are compelling and verified by @AppenResearch.
- Near-perfect long-context retrieval up to 12M tokens on the needle-in-a-haystack test, with up to nearly 1,000x attention compute reduction.
- A balance of long-context optimization and general reasoning ability, with strong performance retained across knowledge, coding, and non-coding enterprise agent benchmarks.
- At 1M tokens, SubQ 1.1 Small requires 64.5x less compute than dense attention and runs 56x faster than FlashAttention-2.
These results highlight a significant scaling advantage thanks to the efficiency gains from the SSA architecture.
We included some details and learnings from the development process which may be helpful to the community.
Comment with questions, I’ll try to respond!
Clayton Kim retweeted
the amount of name changes for different way of using the same tech reminds me of 2021 frontend twitter
Clayton Kim retweeted
any lack of polish now gets equated to ai slop
all fields: software bugs, bad movie, bad video game
before the audience would make a judgement about your skills, which was tolerable because you can get better
now they make a judgement about your character (lazy/fraud/etc)
Clayton Kim retweeted
We’ve partnered with @layerlens_ai to continuously evaluate SubQ across nearly 100 benchmarks and 200+ frontier models on Stratix.
The goal is to continuously improve performance while reinforcing a shared commitment to transparency, auditability, and responsible model assessment.
Results and future evaluations will be published publicly.
Link below to learn more.
Clayton Kim retweeted
Hey, folks! We have been blown away by the response to SubQ and the SSA breakthrough over the last 48 hours.
It is awesome to see how many people are responding to our mission of creating more efficient algorithms to create better models.
We are working hard to firm up our release timeline and will share more very soon.
We will also share additional data and third-party validation in our model card next week.
If you have questions, please post them in the thread, and I'll do my best to respond!
Above all, THANK YOU! The support, feedback, and discussion from this community have been inspiring.
