CS PhD'ing @UofT. ex-@GoogleDeepMind. Official Narrative Enjoyer. Overton Window Technician.
Joined September 2020
- Tweets304
- Following1.4K
- Followers1.2K
- Likes37.9K
the paradox of RSI: the closer we get to RSI, the longer hours each human has to work to engineer RSI loops
if math is solved, then presumably the models a few generations from now might be constructed by math that humans have no hope of understanding
presumably once you solve math you also solve theories of deep learning / computation, which lets autoresearch move past deep learning
an interesting possibility is that preventing those chinese llm startups from distillation will have the same effect on their llm industry as the export control chip ban had on their chip industry
namely, forcing them to become "self-sufficient" in capability advancement
there is only Learning, there's no "meta learning", "in-context learning", "lifelong learning", "continual learning", etc; there is only Learning, everything else is just an emergent property when the core Learning process itself becomes sufficiently strong
knowledge transfer within the same empire is allowed; knowledge transfer across competing empires is forbidden
Maybe an alternative to UBI (presumably used to pay rent to housing and AGI providers) is that everyone owns a piece of Land, Guns, and local AGI
A future of self-sufficiency rather than permanent dependency
I am bearish on World Action Model; i think it will work on static scenes (like a tabletop setup) but won’t work on dynamic scenes (like self-driving)
If the scene has many possible futures, WAM might really struggle
Video pretraining is great; inverse dynamics maybe not
Darwin basically argued for RL from Scratch while Intelligent Design argued for Pretraining from the source of providence
“A well regulated ASI (Militia), being necessary to the security of a free State, the right of the people to keep and bear AGI (Arms), shall not be infringed.”
One possible future is that Kimi open sourcing their best models actually *increases* their revenue because the AI industry is kind of like the Entertainment industry where it’s mostly about the fame and the brand
whenever I watch shows and movies these days I recognize their subtle psyops and cultural messaging and systematic programming of unconscious mind, and they immediately remind me of that 2017 quote: “the fundamental question of our time is whether the West has the will to survive … we can have the largest economies and the most lethal weapons anywhere on Earth, but if we do not have strong families and strong values, then we will be weak and we will not survive”
"distillation laundering": model K is accused of distilling model A, model I wants to avoid distilling model A, so model I distills from model K instead, then everyone agrees that model I got to frontier without distillation
Lunjun Zhang retweeted
Many people are asking how did Kimi K3 catch up so fast to Western models? Simply, a frontier model really only requires compute, data and people. There is no magic secret.
A few answers to this question
1. Kimi with help from the Chinese government has thousands if not tens of thousands of experts (lawyers, scientists, doctors, programmers.. etc.) making RL env data every day. A frontier model is RLed on “tasks”. Each one of these tasks needs to be created by either a human or an LLM. Claude did not wake up one day knowing how to use say the Github CLI. He learned how to use the Github CLI in an RL env.
Meta is pursuing this exact same strategy with its Applied AI org and IMO it appears to be working.
2. I have said this before regarding GLM 5.2 - Kimi obviously distilled from GPT 5.5 and Claude Opus. This only eliminated their cold start problem in RL, meaning they skipped say some X number of months in cold start RL. The mass number of RL envs created by their experts is still the most crucial part here, and you cannot attribute Kimi’s success to distillation. Distillation only saved them some time.
3. Agentic coding and frontier LLMs significantly accelerated their research. My hunch is they use proxies to access Claude API and GPT API for their own model development. Claude most likely wrote all their training code.
This release leads to some very interesting questions. What happens now in a world in which an almost Fable class model is open sourced and free on July 27th? My view is intelligence / software will become very soon close to free. Chip makers and inference providers are big winners from Kimi’s success.