Ali Sezer

10 million context windows incoming

Memory and state baked into weights will be the last step before the first signs of consciousness

The university story @amasad shared at the very end of the interview was pretty cool.. Another quote that caught my attention: “Even if model development stopped right now, Replit would continue to get better for the next 5 years”. Our systems still have a long way to go..

This is pure gold and true

Imagine living in a city where ad boards look like this. Only in SF..

It’s probably in the best interest of the US for the AI houses to collaborate and share research.. which means eventually, it might become a mandate depending on how the race goes

China import ban on GPUs feel like a political move rather than a logical one, aimed to create pressure more than anything else. Chinese chips are not that advanced, yet. Not sure if they would risk falling behind the race by handicapping themselves.

I saw a Cursor CLI post in my timeline today, and it got me thinking about how they might be following a sub optimal strategy by trying to cater for all the models in their products.. In their recent podcasts, @OpenAI and @AnthropicAI provided updates and vision on their products, Codex and Claude Code. They refer to these products as the "Harness" and the "Scaffolding" of their models, in that order. Something that is very clear, and we are seeing this already with gpt-5-codex, is that they will begin optimizing their models for their specific products. Which means, tools like Cursor CLI will begin to operate at a disadvantage. Its almost like they are building a car which can swap engines, but the engines they will be using are going to be geared to run better on other cars.. Tells us something about how important it is to own the weights or have a strong partnership in place with the owners of the weights, because they can always come up with something better than what you have, given that the model, is the most important component. IMO, a good strategy Cursor could follow here is to partner with a company who haven't made huge progress in this field, like @xai and look to optimize their offerings together..

This provides a good hint towards the AI ROI story. The other points that were made in the last months, with regards to how each model release should be treated almost like a seperate project/company is very important.

This is what Field Engineering is partly about

At this point, if I could choose between having smarter models or having longer context windows like 10-100 million tokens, I would go for the context window. Memory feels like the bottleneck

Lets say they make $200 million in the next twelve months. Thats a 35x multiplier.. Median of SaaS is around 15x right now

Funny enough this is from Sonnet's system prompt: "Claude never starts its response by saying a question or idea or observation was good, great, fascinating, profound, excellent, or any other positive adjective. It skips the flattery and responds directly."

The "you are absolutely right!" paradigm to me sounds like a system prompt problem mixed with some psychophancy. The chat version of Claude doesn't really do that, so it is probably easily fixable. Codex has similar patterns, for example, GPT5 always tries to put __all__ at the end of every Python module..

Finally got the chance to try out codex, few observations / thoughts: > The system overall seems very capable. I used it for coding in specific frameworks such as Spark, Ray, and some other utilities where models usually perform worse as opposed to pure language programming (lack of training examples). Even then, it had great performance and haven't ran into the usual hallucination problems yet. > UI can get better, Claude Code offering on that side is better, however the tool has been only around for so long and I'm sure they will improve. If you check out the changelog on their github, you'll see the speed of their delivery > Being able to switch to the Pro model mid conversation then switching back to the regular one could be a very nice feature. For example, a certain task gets super hard, and you want pro to have a look, you switch using /model, and then switch back to the regular model to resume the conversation, with the context pro has generated. I think that would be super powerful. > Resuming conversation features could also be another one that can be implemented, but I think that is probably coming very soon.. > Something that really surprised me, I was doing some EDA work which required querying the DB quite frequently to find the necessary data points. I asked GPT to create a CLI based utility to run queries against the DB, and then stated that this can be used as a tool in the future as we do the work. Didn't register the tool in any programmatic way, however it was able to remember how to use it perfectly and more importantly where to use when I prompted for it randomly further down in the conversation. I think thats a great show of intelligence levels > Context length is very impressive too, the conversation goes for much longer as opposed to Claude Code. Not sure if that is because the system is more efficient with its tokens or the model supports a longer context length. > Currently using side by side with Claude Code, where I prompt Codex to do the application level programming and Claude to take care of things like environment setup, CLI commands, infra, tests, etc.. I share context between the two somewhat succesfully through a lot of markdown files. > Codex can do a little bit better when it comes keeping a track of file updates - if a file gets added or updated outside of its workflow (for example, if I update something or if Claude does) it's not able to detect it because I think it assumes that its the only system changing the code. I prompt it to check everything first at the moment, which is not too bad. > All in all, final observation, AI usage through Cursor for me has become minimal. I sometimes go for it for testing another model like Grok Code, however there is a clear pattern here where agentic systems offered by the model companies themselves are getting ahead. I don't think it would matter that much for me right now if I used vscode over Cursor.. But you never know, they might release a CLI solution that might put them on par with the rest of the tools

I was watching @bcherny latest interview about Claude Code where he made an interesting observation with regards to how coding behaviours are slowly changing as they are shifting away from “manipulating text directly in an IDE”. This is partially true for me too, especially for side projects - I would rather have an agent do the coding and just review results or interject if it goes totally of the rails, which makes you think about how the programming languages are going to begin to look like in the future. Do you need all the syntax we put in there for readability like tabs ?Or whether programming languages will actually exist at all ? Maybe it really is just going to be all nueral net based operating systems. Which raises the question, will we ever trust them since they wont be deterministic, but when you think about, we already do, for example, we begun to trust them to drive us around already..

Interesting takes on this topic, and whether or not Databricks “fumbled” the investment here. Databricks aquired Mosaic right after chatgpt 3.5 came up, which was the first moment where everyone started realising the capabilities of the technology. At the time, it seemed like organisations and companies were each going to begin training their own language models. In that scenario, it made sense for a infra/platform company to have a horse in the race: hey customer, if you want to train an LLM, we got your back… A lot of open source LLMs from different orgs were being released everyday at the time. However, soon enough, it became clear that training the best models were going to require a ton of GPUs, and a ton of investment. This is when semianalysis released the “gpu rich” vs “gpu poor” terminology and it was right. And, it doesnt make sense to train one if better versions are going to exist, why bother spending the money when Meta or Deepseek is going to make what you have obsolete in a few months? You can just use what they produce, or the closed source alternatives, especially if you are an enterprise and not an AI Native company.. Fine tuning didn’t take off as expected too, at least at a large scale to justify and investment as such. So what do you do when the winds change ? Pivot, and begin building low barrier to entry tools, such as Agent Bricks, for the enterprises..

It almost feels like Anthropic doesn't have enough GPUs