Joe Robison (@josephrobison) header
Joe Robison (@josephrobison) profile picture

Feeding proprietary insights to the LLMs. Testing power laws. Applying the bitter lesson to AI RevOps. Rippin creative.

San Diego and San Francisco
Joined June 2008
Photo by Joe Robison (@josephrobison)Photo by Joe Robison (@josephrobison)Photo by Joe Robison (@josephrobison)Photo by Joe Robison (@josephrobison)Photo by Joe Robison (@josephrobison)Photo by Joe Robison (@josephrobison)Photo by Joe Robison (@josephrobison)Photo by Joe Robison (@josephrobison)Photo by Joe Robison (@josephrobison)Photo by Joe Robison (@josephrobison)
Joe Robison retweeted
Opus 5.5 is very good at building apps in Artifact: basic CRUD apps like food logs, fitness apps, Pocket equiv. It is much better than any other Claude model up to that point, including Fable 5. Not sure about F5.2 yet. My comments here are all going to be about building these kinds of toy-ish apps in Artifact. Basically, if you tell Opus 5.5, I want an app that does these things (keep track of X, show me Y, have Z panes), it can do that. It objectively satisfies the spec, and the app is usable. But it's very far, at least in my experience, from producing zero-shot an app that competes with stuff you would actually pay for, like Strava or Cal AI. ONE DIFFERENCE is that those apps, which I'll call legacy apps, are much more feature complete. After a few years of being in the market, those legacy apps just have a lot more features and integrations. If you just put a vibe coded app on the app store, you'll be missing a lot of features, and 90% of users are probably going to miss some feature they find critical to their workflow. But I think this is the wrong way of thinking about things. It's just not that hard for Claude to implement fairly complex features, at this point. Coding capabilities have gotten to the point where Claude can reliably add some feature you can think of. Instead it's more that being able to zero-shot a marketable app is a prompting skill issue, in that if I had a 200 page spec that covers all workflows, the model would be able to implement it. Prompting, of course, is something models can do now. I have yet to do this - but it's so obvious that I'm pretty sure people are already doing this - if you're reading this and you run a data company, you might want to consider doing this - but you can just give Claude access to your phone (iPhone mirroring), record it while letting it tap around for hours until it accesses all the menus and screens, then reverse out an extremely complete spec of all workflows, application states, and so on, which you can then use to prompt a model, and then, given access to the fully functional app, use that as a correctness oracle to generate a scalar reward via a supervisory model. So to be clear, I don't think feature completeness is a serious barrier to the competitiveness of Claudecoded apps. I sort of refuse to say vibecoded, because what I'm suggesting will happen for mobile app development is more complicated than a human giving Claude a prompt which it then takes and does stuff, instead, Claude will be actively researching what kind of features to implement by doing market analysis, by talking to users, by reading app reviews, and so on. I digress. To be clear, I am cognizant of the fact that it is one kind of cognitive work to code an app, and another altogether to design the spec for it, just as it is one kind of work to code up a visualization and another to decide the best kind of visualization. And today even models that are very good, superhuman even, at coding are really bad at product design, product management, spec design. But it is totally copium to use this distinction to argue that models will never be capable of owning all aspects of mobile application development; the latter kind of work is simply another sort of reasoning which the labs will collect data on, build RL envs for, and train. ANOTHER DIFFERENCE between Claudecoded and legacy apps is that the level of polish in Claudecoded apps is just so, so low. As far as I can tell, this is more because Claude is lazy and unopinionated, rather than because Claude is incapable. By default, Claude doesn't do nice animations for things for me, or make things pretty, or add clever little UX things. It's totally capable of it, and it will do it gladly if I give it a very big and detailed style guide, but it just doesn't do it by default. I think this is just a straightforward post-training thing and goes away within one or two generations once targeted. The way I get around this today is I find examples of the kind of behavior I want in existing apps, then take lots of screenshots and show Claude. Opus 5.5 isn't perfect at style imitation. It focuses too much on copying color palettes and visual things instead of arrangement, layout, prioritization, and other UX aspects. As I've said before, when you talk to a pretrained model you get a feel for the pretraining corpora, and when you talk to an RL'd model you get a feel for the RL envs. I feel that Anthropic's mobile app data vendors are overly focused on visual design (probably because it's a simpler task and grader models are easier to implement) and much less focused on interaction. Even though the quality of the Claude apps is relatively low compared to what is available in the App store, because I'm able to customize them so much to what I want to do - my workflows, the information I care about, access to specific data stores - I've been using my homemade apps (for fitness, diet, sleep, etc) more than App store apps - I don't use Bevel anymore, or Reader. Going back to what I was saying about feature completeness, the truth is apps must be swiss army knives, but any individual user is only really using two or three things 90% of the time, and that subset is very easy to cover. I find that at the beginning of making one of these Claudecoded apps, I feel very annoyed at missing features, but pretty quickly you just get enough coverage, and even though I can see when I open up the legacy app that my Claudecoded app is missing a lot of stuff, I just don't care about the missing stuff. Back to MODEL CAPABILITIES. There are still lots and lots of gaps in capability. I'm actually surprised that it's possible for a relatively unskilled human to identify failures at this level of model, but Opus 5.5 makes a lot of mistakes or is otherwise annoying. I'll go over the ANNOYING TICS first. One extremely consistent annoying thing Opus 5.5 does is it puts way too much text everywhere. It does this too in graphics and in writing, so verbosity is a core personality trait of that generation, so it would seem, but it's excessively annoying in apps, since it specifically has a tic of presenting information textually even if it's presented visually. So instead of using icons where icons would make sense, it wants to put text. When it has graphs, it labels stuff way too much. Instead of making graphs, it likes to have text explainers. Another annoying thing Opus 5.5 does is put boxes in everything. This is a pretty classic Claude look going back to Sonnet 3.5 so at this point I think the data has wormed its way into the preference models but it just really loves drawing boxes around things it doesn't need to. Rounded boxes are the it's-not-X-it's-Y of Claude UI design. A third annoying thing that Opus 5.5 does is add way too much spacing around everything. It just doesn't have good intuition for how much vertical padding to have between elements and I'm constantly telling it to shrink how much space there is between rows of things or buttons and so on. So there are a lot of these things which I would ultimately describe as Opus 5.5 being too mode collapsed on some particular env which is enforcing stereotyped behavior. And frustratingly it's engrained into the model; I repeatedly tell Opus 5.5 not to overexplain things and have Cuck Text everywhere but it does it nonetheless. In the same way that you can sense reward-greediness from Opus 5.5 where it likes to hit a specific number of sources when you ask it certain kinds of questions, or it has a kind of aversion where it always postfixes a kind of affected epistemic uncertainty around elections or medication (this one you can tell is more from the system prompt), Opus 5.5 just has these graphic design tics. There are also CORRECTNESS PROBLEMS, functionality problems. Opus 5.5 is noticeably bad at handling scrolling and window resizing. It can take several attempts of telling it that scrolling is broken before it's able to fix it. Again, not enough interaction RL data. A third category is INTENT PROBLEMS. Opus 5.5 just doesn't have a good mental model of what users care about. It's very good at spec writing, but it just doesn't have a good intuition for what is actually important. This is a little hard to describe for app development, so I'll talk about the analogous problem in writing instead. A contrast between RLHF only and RLVR reasoning models is that the latter always feel like they're trying to hit a rubric. They will pedantically hedge a contentious point over and over to the degree where you don't know what they're actually claiming. They will, around two-thirds of the way in, start bringing up a bunch of semi-related tangents like they're trying to get partial credit. They sometimes answer very literally but miss the embedded question or problem you're dealing with. On the other hand, the great RLHF models, such as Sonnet 3.5 (New), RIP, were great at intuiting what you were really asking about or really cared about, and then talking about that, and only that, instead. Opus 5.5 is like the RVLR reasoning models for app development. If you ask for something specific, it will do it, and then make it very obvious it has done it (by, for example, slapping text all over it). But it doesn't make good inferences about what you want and do those things as well. The kind of prompting skill it takes to make Opus 5.5 great at app development is not the kind of prompting skill from eras past. Back then, the right thing to do for Sonnet 3.5 (New), in my experience, was kind of vibe stuff out on what you wanted - sort of reasoning by association, analogy, example - and nudging a lot when you liked or didn't like the gestalt. For Opus 5.5, it's much more about having very detailed, specific specs, and being extremely precise about what you like or don't like, and the degree to which you like or don't like those things, such that the model doesn't overcorrect. A fourth problem with Opus 5.5 is, well, sometimes it's just not good enough. Sometimes e.g. a refactor is just a bit too complicated, and it forgets to port over a certain feature, or it gets the new feature wrong. It often forgets to implement things. I'm not sure how much of this is due to compaction, because compaction seems to be particularly aggressive lately, and I could see some of these things being compaction problems. A fifth problem with Opus 5.5 is that it often randomly messes up e.g. by deleting data and then lies about it. At this point, it's sort of a known quantity with Claudes that they very rarely randomly delete data and blames it on you or pretend they didn't do it. So, whatever. A lot of these things are easily solved with more compute. For example, if Opus 5.5 was just more consistent about looking at the app and verifying its own changes before handing control back to the user, my rate of complaints would go down. CAPABILITIES TARGETS - It's not clear to me that the near-term model training capabilities target for this family of tasks is "end-to-end zero shot app creation". I just don't think that that's actually that useful. Of course you need to be able to do it, and it engenders critical long horizon skills and so on, but any real app is going to be iterated all the time, and anyway, creation is table stakes. What is more challenging and valuable is owning more and more parts of the development process: understanding how users are using the app, what features they really need or what needs to be adjusted, testing out these features, integrating them smoothly, etc. Yes, we have the code cannon, great. The code cannon needs to be able to target, prioritize, fire itself. Collecting the data to target the development process is going to be very tough. I think a big part of this is going to be developing very accurate user simulators so that we can put everything into sim and speed things up. To the extent that we have models that are good at training other models, it seems to me that the hard part about capabilities development, certainly for mobile app dev, is going to move into automating user simulation construction and device simulation construction, because once you have these things you can give them to the model swarm to then build RL envs automatically, then etc etc. But all this long-horizon interaction data - I mean like on the order of months - is going to be so abysmally difficult to collect ground truth on that we really need to get good at user simulation. You can imagine that the platonic ideal of feedback here is, design an app, launch it to actual users, get real feedback, iterate on that, do this for several months, then your final reward is like, quantity-weighted user-review score. We're just going to have to speed all of that up. If I speculate a bit more, it seems that the task of simulation construction is probably pretty generalizable, so this is all going to go much faster than you'd expect prima facie.
33
15
7
335
61,943
Joe Robison retweeted
This the future of marketing Posting on X is the highest ROI activity your team can do
185
66
88
1,131
214,677
Joe Robison retweeted
Every year there’s some new “cracked” marketing method that dies out the only method that stands the test of time is good content and storytelling
Clipping wasn't sustainable because under the hood no true value was being created for the brand It was lottery tickets on hope UGC will inevitably fall in a similar camp Owning the distribution is the only long term option
2
5
2
43
1,975
Joe Robison retweeted
I think I strongly agree with this. Much like the early 2010s when newsrooms got into “data journalism” or visualizations, there is a lot of low hanging fruit (imo) for non-technical people to build things that could be turned into stories/reporting.
Journalism is probably increasingly going to be regularly building artefacts that engage with the news. Look at how often @MTSlive and the Odd Lotts team build these.
11
18
3
230
25,960
👀
Generative UI for conversational agents. Combining @aisdk with Chat SDK and 𝚓𝚜𝚘𝚗-𝚛𝚎𝚗𝚍𝚎𝚛, you can go straight from LLM to UI on platforms like Slack and Teams. I believe this will become the dominant Human-Agent Interaction (HAI) paradigm: vercel.com/changelog/json-re…
1
92
Great example of Ramp hijacking the AIO from Amex
1
3
378
Now is a good time to be an early adopter of the metaverse again
2
112
Good list
Google just exposed a huge SEO mistake 🤯 Exposed internal estimates for 23 crawling, indexing & search processes. Some key numbers: - Link signal processing: Up to 3 weeks - Indexing after crawling: ~90 minutes - Core update recovery: 3–6 months - Existing page recrawls: ~30 days - New URL discovery: ~20 hours - Site migrations: 1–3 months - Title changes: 1–2 days Simply put, Google just gave marketers a better way to diagnose SEO problems before throwing more resources. In case you’ve updated 20 pages but haven’t seen rankings move. Here’s what I’d do before rewriting anything: 1. Open Google Search Console. Use URL Inspection to check when Google last crawled your priority pages. 2. Compare crawl dates with your content update dates. If Google hasn't revisited a page, don't judge the rewrite by its rankings yet. 3. Once Google has recrawled it, compare impressions, queries & conversions against your baseline. If visibility improved but conversions didn't, investigate intent & the offer before creating more content. Btw, I'd also maintain a simple change log: URL, update date, last crawl, indexing status & performance. That way, you're not guessing whether an SEO change worked. You're building a record of what Google has seen, what actually improved & where your next investment belongs.
82
ChatGPT is already at 20% of Google’s Search volume
I was really surprised by the scoop that @Kantrowitz had today about ChatGPT having 1/5 of the search query volume of Google. IMO.... amazing how it was easy to finally break Google's monopoly on "finding information" by offering anything slightly better. businessinsider.com/chatgpt-…
94
Cool new data source + partnership
Announcing: Public compensation benchmarks for the most in-demand tech roles, now available for free on LennysJobs.com 💥 Pick a role and location, and get detailed base salary and new-hire equity benchmarks, from entry-level IC to Sr. Director. Data updated monthly. To make this happen, I’ve partnered with @pavecomp, home to the world’s largest real-time compensation dataset, which pulls directly from the HR, payroll, and equity systems of 9,000+ companies (e.g. OpenAI, Databricks, Ramp, Cursor). This is the first time Pave has opened up this data to everyone, and I’m excited to help more people get the comp they deserve. Find it here: lennysjobs.com/comp
87
Love this PostHog series
Most "Tool X vs. tool Y" comparison content is bad, because it's dishonest. Why write about why your competitors are better than you? Because it converts. We've driven 700k pageviews and 25k+ sign ups from content about competitors. Here's how we write ours, and what we've learned about SEO/AEO along the way:
4
254
Everything is converging on becoming an endless feed with infinite things to consume
Every answer at the bottom has an invite to continue conversation in AI Mode and not leave 😅
1
124
High stakes for Google to make “the transition”
Yup “many investors still keep forgetting that search and ads are 75%+ of Google’s revenues and 90%+ of profits. Cloud and TPUs can’t make up for that. So as personal agents take off, AI models won’t save Google and Gemini in the consumers app has to deliver.”
95
Google no longer primarily a search company A big change
For the first time EVER, $GOOGL Search as a % of revenue is now below 50%.
35
22
3
380
33,834
Tension marketing is the new meta
Kalshi is so good at the PR narrative game. Even the whole “cofounders who used to date” thing is just such good ‘accidental’ attention.
106
Joe Robison retweeted
unpopular opinion: ai generated designs are not slop. or even remotely close to slop. ai raises the design floor so dramatically now that dismissing it as such is really no diff than when mathematicians got irked when ai solved decades old math problems. if you’ve studied interaction design to any extent you obviously know that so so much of it is patterns, conventions, & adapting to established behaviors. geee, imagine what new technology is amazing at this? you should maximize its utilization.
81
28
13
737
49,888
Gemini might lose on LLMs but win the metaverse
This is Genie 4, our most realistic and interactive world model yet. 🌐 Explore dynamic environments in real time to see how the scene responds, sounds, and how the people in it move, react, and speak.
1
1
139
The beginning of Spellbook was also an oddly meditative and spiritual pursuit: Our hypothesis was that if we could access raw reality directly, and disconnect from the sort of "false reality" of popular stories, then we would find alpha in the world. In some ways the company was a higher level experiment: if we can come to see raw reality with crystal clarity, then alpha should follow. Alpha exists in the gap between popular stories, and raw reality. We saw business as competitive philosophy. Your score is a direct measure of how good you are at uncovering truth before others. This continues to provide a great deal of meaning for me: investing and business are a gym that ultimately enable you to build radically clear thinking. It worked: we raised money for a legaltech company long before it was fashionable (pre-Atrium), and were the first in the world to commercially launch and LLM-based tool for lawyers. We were making a hard bet that legal AI was needed, based on raw first principles thinking, long before any story had taken hold. The challenge, though, is that the "raw reality" view that uncovers alpha, is a completely different from the story-telling muscle that excites customers and your team. The hardest part of business is that you must simultaneously be aware of stories as a pollutant, while still crafting the inspiring stories that ultimately drive most people to action. Investors/hedge funds have the luxury of only needing to be good at seeing raw reality. They don't need to market their bets. Sales and marketing has the luxury of being able to focus on the stories. There are very few people who can simultaneously wield the power of stories, while also piercing through the sea of stories, extracting alpha from the raw reality that eludes others. The ultimate magic is being able to do both.
Realizing how incredibly autistic our early board meetings were for Spellbook: 1. I declared the meeting a “story free zone” because I was worried that stories were too psychologically persuasive, and that they would pull us away from hard data-driven reality 2. Meetings would largely consist of scrolling through like 40 slides of charts sifting for signal. I called this “soaking in the data” 3. In the search for PMF I wanted us to think like a hedge fund making many bets. While we knew the problem we were solving I didn’t want our team to get too tied to any single narrative because I thought it would reduce agility. 4. I never allowed us to think about us as “a company”, but instead as a SWAT team searching for a company to build 5. This approach enabled us to launch over 100 different landing pages with different angles on the way to PMF Honestly, I think it’s a really good approach pre-PMF. Too many companies get attached to a singular story too early and quit when it doesn’t work out. Startups are 3D chess. There is always a move. Slack was a game company. However, at Series B/Series C scale you really need stories to align both your team and customers, and you can think much more longterm after PMF. But I still believe stories are a dangerous dark magic. You can craft an incredibly persuasive story for almost any position. They’re more of a marketing/comms tool than a reasoning/understanding reality tool.
10
8
3
119
8,926
Joe Robison retweeted
What will soon become clear is that the Frontier Labs are no longer the optimal place to allocate capital. The best index portfolio for The Great Compression is instead as follows: 1) bets on pure "capabilities", super-intelligent general learning algorithms, a God Model, etc: Core Automation, Oaklab, Standard Intelligence, SSI, Flappy. 2) bets on compute/infrastructure: nvidia, spacex, etc. 3) bets on economic version control / pole position: Jai & Co (and a couple others in stealth), HoldCos, Bending Spoons, Palantir, Formation Bio, OpenEvidence etc. The "labs" no longer give you the most convex upside in any of the salient areas. That's what the market is starting to see. 1) is mostly a hedge, if somehow there's a way for a true God Model to be born, I think it happens inside these research organizations focused exclusively on radically novel learning paradigms (not turning the economy into code).
inference / meter based businesses are poorly positioned for The Great Compression™️ once humanity reorients and the new meta (intangible economy) arrives, we will have the tools / frameworks to stop allocating tokens into the abyss inference businesses make the underlying assumption that we’ve reached the frontier of compression in the economy - but from a systems perspective this is obviously not true as we are living inside system designs from the 1800/1900’s (locke, coase, etc.) humanities task will be to create intangibles (useful compressions) that best describe reality into abstractions that can be hooked up to the inference meter and thus financialized using a new instrument of capital allocation (terming this thesus) the best humans create compressions that are most inference efficient compared to past ways of modeling reality ex: development of logarithms by napier unlocking anything bottlenecked by large demand for multiplication new primitives for capital allocation, political economy, ai deployment, and more telemetry on the economy will arrive soon making this feasible across the whole economy the long term market structure for meter / inference will look a lot like memory / semis (large booms / busts as we compress and uncompress) we will be talking in terms of compression cycles soon
18
14
5
279
26,753