Take Control Of Your Data. Join our Data Science Briefing newsletter for the best in #DataScience and #MachineLearning
Manhattan, NY
Joined February 2015
- Tweets7.6K
- Following462
- Followers1.5K
- Likes351
Pinned Tweet
17 days until the LLM Eval Harness workshop.
Build a production eval harness live in 4 hours. Calibrated judges, error bars, CI gates.
Engineers who measure ship faster.
maven.com/bruno-goncalves/ev…
Tomorrow, 12pm ET: Jev in One Call. 30 min, free, live on Zoom.
Meet Choice, Score, Noul on a real support ticket — a dozen questions in one pass.
Register: maven.com/p/1d62f8/jev-in-on…
If you need a framework to make your AI agent "safe," your agent isn't safe.
Safety is engineered into the system. Not imported from a package.
Curious what others think — am I wrong?
(Building this live on Oct 31 → luma.com/qwhit9op)
Free lesson — Wed Oct 21, 12pm ET.
Meet the 3 primitives of Jev on a real ticket. One ~100ms call, $0.042/1M tokens, output free.
Register: maven.com/p/1d62f8/jev-in-on…
1/ Most AI agents fail in production not because the model is bad — but because nobody engineered the system around it.
Here's what actually breaks (and how to fix it):
🧵
3/ Problem 2: The context window fills up and the agent starts forgetting — silently.
Token budgets, memory tiers, and conversation compaction aren't optional. They're how you keep the agent from going off the rails 45 minutes into a session.
4/ Problem 3: There's no retry logic, no cost ceiling, no audit trail.
One malformed query. One infinite loop. One $400 API bill later — you wish you'd built run supervision from the start.
I'm teaching this on Oct 31. 4 hours, hands-on, Python only.
→ luma.com/qwhit9op
I'm running a live 4-hour workshop on Oct 31: Build an AI Data Analyst you can actually trust — MCP tools, guardrails, context management, fault tolerance. Laptop-only, framework-free Python. 🎃
Details + tickets: luma.com/qwhit9op
1/
Deeper on the one failure that actually ships:
fabricated numbers.
How an agent reports a 12% revenue drop that never exists, and the exact guardrail that stops it -
3/
Then it verifies:
- value matches the query output
- unit is right (not $12M reported as 12)
- reporting period is complete
LLM judging is reserved for clarity - not for 'is this number real.'
4/
The rest of the harness does the same job for the other failure modes:
permissioned tools, context budgets, injection defense, retries, traces.
It's a 4-hour hands-on build, framework-free, on a laptop. Sat Oct 31 - 19 days out:
Register: luma.com/qwhit9op
19 days.
Sat Oct 31 - 4 hours, live - we build an AI data analyst you can actually trust, in plain Python. No GPU, no cloud, no API key.
$149.99, seats are limited.
Register: luma.com/qwhit9op
Last call — free LLM eval gate session is TOMORROW (Oct 7, 2 PM EDT).
Regression thresholds + GitHub Actions wiring + merge blocking, all in 30 min.
Register: maven.com/p/07c4b3/stop-bad-…
Hot take: most of your AI agent's reliability problem isn't the model.
It's that you're letting the model pick its own tools and trust its own numbers.
The model should SUGGEST. A harness should DECIDE.
Am I wrong?
Hot take: if your LLM app doesn’t have a CI eval gate, you don’t actually know if it works.
You know if it worked last time someone manually checked. That’s not the same thing.