I help devs test AI agents and workflows 10x faster | uv add appmerit | prev. TensorSense
San Francisco, CA
Joined July 2009
- Tweets874
- Following106
- Followers226
- Likes9.3K
Can't stop thinking how much stuff Jev is doing to save AI engineers asses from stupid systems design patterns while the rest of the domain keeps pushing tool calling loops everywhere
I beg all my friends - ignore any evals, and just go play with jev a few hours. I bet that'd change your entire perspective on AI and its future
Imo FSD is the best example of an AI agent in production, and the craziest thing is it works locally in your car
devs learn ML from 1st principles lol:
1. labs use RL to teach AI to write better code
2. RL verifyers use tests to assign rewards
3. AI learns the shortest path to max rewards
4. writing good tests for AI goes absolutely against its weights
ok it's official... AI-written unit tests and integration tests are empirically proven to be unhelpful. you should tell your agents to stop writing tests by themselves
on the deepswe eval set, banning sonnet 5.5 high from writing any tests actually resulted in slightly higher success rate (non stat-sig), with less time and token spent (stat-sig). this is pretty hard empirical evidence
across the tests that were written by the "tests allowed" baseline arm, 65% of them were unit tests, 35% were integration tests, neither bucket resulted in any improvement compared to not writing any tests at all
i also picked a random sample of 44 tasks subset where i completely disabled executing even existing tests - it also did not affect success rate at all
i spot checked many tests written in the baseline arm, and my intuition is that most tests are simply a repetition of the implementation
agent-written tests do not add any value because both the implementation and the tests were simply the agent's interpretation of our intent. the tests aren't any more accurate than the implementation itself
i suspect we can still extract some value from unit tests and integration tests if we describe them ourselves when we believe we can articulate our intent better through test cases than through requirements. but i have not proven this yet
also worth noting, during deepswe eval the agent wrote almost no e2e tests (only 17, compared to 3000+ unit/integration tests written). so this analysis does not prove nor disprove the value of e2e tests - i will do another eval specifically for that
Imo saying "Jev is just a classifier" is same as saying "GPT is just an autocomplete" in 2023 - this "just" tells more your mental model than anything. To me this is more exciting release than Fable 5.1 and Astra combined.
After co-inventing ChatGPT, I kept asking myself: why have superhuman chat models not led to AGI?
I’ve spent the last 2 years in stealth building a new way to train models (RLCD), and a new type of frontier AI model that we are releasing today: Jev
• 20-200x faster
• 40-400x cheaper (w/ output tokens free)
• Frontier composable intelligence optimized for decisions
AFAICT the shortest path to AI-based economic revolution
In a year y'all will get rid of general purpose scoring LLM judges and switch to small binary task-specific classifiers that can run locally, and I'm happy Raindrop is getting us there
introducing rd-signal-2: a frontier classification model that is 1600x cheaper than GPT 5.6 Sol.
free to try in @raindrop_ai, and available via a new API for training/hosting custom classifiers with Zero Data Retention.
I don’t have a strong opinion on RLS overall, but exposing databases to agents via tools and relying on RLS for security is definitely a terrible idea.
The best pattern I’ve seen is using hydrated copies with sync engines + isolated getters and setters
RLS was a mistake and folks exposing that level of complexity to less technical users is asking for trouble.
It was a mistake in Firebase. It’s a mistake in Supabase. It will be a mistake in the next product too.
I personally - even knowing how to secure it - would never touch it. It’s the worst security footgun you can imagine. One small mistake and your data is available to the world.
Can we stop pretending microservices are AI models?
LLMs are great for human in the loop applications, but fail at deterministic developer tasks.
@interfaze_ai is a new AI model that outperforms general LLMs on high accuracy tasks like: OCR, Object Detection, Web scraping, Speech-to-text, Classification and more.
Congrats on the launch, @yoeven and @khurdula!
ycombinator.com/launches/Pyw…
Kids, remember - if your shit is good for developers, then your shit is good for agents.
Clear APIs, explicit syntax and errors, good CLI tooling, well-written docs are more important than ever. MCPs and Skills won’t help if your DX suck
As someone who has looked into 100+ LLM projects recently, both agents and workflows, I can say this is very close to what actually happens.
Typical stages of automating stuff with LLM
- oh oh my God. I think this might be possible
- oh fuck, it actually worked
- holy shit this is scary. What am I gonna do
- ok let's try to get it to do again
- oh weird, it doesn't work like the first time. Hmm lemme fix the prompt
- ok that's fixed let's run next 20 examples
- hmm, 6 failed in 4 different ways. What's going on here
- let's try another model to see if that makes a difference
- ok now bunch of other shit broke
- ok go back to original model.
- chatgpt, how do I make this work, make no mistakes
- fuck it made mistakes
- I already told my boss I can automate this. I have to make it work
- fml
It's time to start writing automated tests that inspect what happens inside LLM traces!
uv add appmerit
Why evals need all these abstraction layers between assertion and metrics? Why they can't just be like us and love Python?
Docs are here btw: docs.appmerit.com/concepts/m…
Richard Feynman would approve of this writing.
Today Thinking Machines Lab is launching our research blog, Connectionism. Our first blog post is “Defeating Nondeterminism in LLM Inference”
We believe that science is better when shared. Connectionism will cover topics as varied as our research is: from kernel numerics to prompt engineering. Here we share what we are working on and connect with the research community frequently and openly.
The name Connectionism is a throwback to an earlier era of AI; it was the name of the subfield in the 1980s that studied neural networks and their similarity to biological brains.
thinkingmachines.ai/blog/def…
Ok, AI researchers finally admitted hallucinations are a bug, not a feature. Good start.
Next step: accepting that tweaking model weights to get rid of hallucinations is actually bad debugging practice.
New research explains why LLMs hallucinate, through a connection between supervised and self-supervised learning. We also describe a key obstacle that can be removed to reduce them. 🧵openai.com/index/why-languag…
