Omev AI

Evals are the most underrated skill in AI engineering. 1. Start with error analysis, not metrics Sample 100+ production traces. Annotate each failure in free text (open coding), then cluster them into failure categories (axial coding). Count frequency per category. Your top 3 categories are your first metrics. 2. Build a versioned golden dataset Store inputs, expected outputs or rubrics, and metadata (intent, difficulty, user segment) in a versioned dataset. Stratify it so rare but critical cases aren't drowned out by easy ones. Version it alongside your prompts so every score is reproducible. 3. Generate synthetic data along explicit dimensions Don't just ask an LLM for "test questions." Define dimensions (persona × intent × complexity × edge case), generate combinations as tuples, then turn each tuple into a natural input. You get coverage instead of 200 near-identical examples. 4. Deterministic assertions first JSON schema validation, regex checks, required fields, length limits, code execution, SQL that actually runs against a test DB. These are cheap, fast, and have zero variance. Reserve model-based grading for what code genuinely can't check. 5. Evaluate retrieval separately from generation For RAG, measure retrieval with recall@k, MRR, or nDCG against labeled relevant chunks. Measure generation with faithfulness (is every claim supported by retrieved context?). If you only score the final answer, you can't tell which half is broken. 6. Trajectory evals for agents Log every step: tool name, arguments, result, tokens. Score tool-call precision and recall against a reference path, argument correctness, step count, and error recovery. An agent that reaches the right answer in 14 calls when 3 would do is a cost and latency bug. 7. pass@k vs pass^k pass@k = succeeded at least once in k runs. pass^k = succeeded in all k runs. At 80% per-run success, pass^5 drops to roughly 33%. Production reliability lives in pass^k, so run each case multiple times at your real temperature. 8. Simulated users for multi-turn Give an LLM a persona, a hidden goal, and constraints, then let it converse with your system until the goal is met or a turn limit hits. Grade goal completion, turns-to-resolution, and policy violations. Single-turn evals miss most state and memory bugs. 9. Align your LLM judge like a classifier Hand-label 100+ examples as pass/fail. Split into train/dev/test. Iterate on the judge prompt using dev, then report TPR and TNR on test. Use binary outputs with a written rubric, require a short critique before the verdict, and pin the judge model version. 10. Control judge bias in pairwise comparisons LLM judges have position bias and often prefer longer answers. Run every A/B comparison twice with swapped order and only count consistent verdicts. Normalize or cap length if verbosity is skewing results. 11. Respect statistics With 100 examples at 80% accuracy, your 95% confidence interval is about ±8 points. A jump from 80% to 83% is probably noise. Use bootstrapped confidence intervals, or a paired test on the same examples, before declaring a win. 12. Gate deploys in CI Run the eval suite on every PR that touches prompts, models, tools, or retrieval config. Set per-category thresholds instead of one global score, so a big win in one area can't hide a regression in another. Fail the build on a regression. 13. Online evals in production Sample a percentage of live traffic, run your judges asynchronously, and alert on drift per failure category. Use shadow deployments to run a new prompt or model on real traffic without showing users the output. 14. Make cost and latency first-class metrics Track tokens per request, cost per successful task (not per call), and p50/p95 latency next to quality. A change that keeps quality flat while cutting cost 40% is a real win and deserves to ship. 15. Close the loop Every production failure becomes a new labeled test case. Every new model release gets run against the full suite before anyone debates switching. Over time your eval set becomes the most valuable asset in the codebase. Prompts are disposable. Models get replaced every few months. Your eval infrastructure is what compounds.

Everyone’s excited about decision models. When should you buy one, or train your own? We fine-tuned a 2B model for a production classifier: frontier-level quality, 35ms, ~1% of the cost. We’re sharing when we think it’s worth it, and when it isn’t: airbnb.tech/ai-ml/frontier-q…

I think one of the next million-dollar AI businesses could come from solving a problem that barely gets discussed on X. Companies are delivering services they forget to bill for. And some AI companies are losing money on customers who look profitable in their dashboards. Imagine building a system that catches both. I'd call it REVENUE GHOST. Built with: Claude Code + Docling + DuckDB + Stripe Billing + Langfuse + Temporal. Six technologies. One contract-to-cash intelligence engine. Here's how I'd actually build it. 1. DOCLING — READ WHAT THE CUSTOMER AGREED TO Start with the signed customer contract. Docling extracts text and tables from PDFs and other supported documents. Install: pip install docling Then: from docling.document_converter import DocumentConverter doc = DocumentConverter().convert("contract.pdf") text = doc.document.export_to_markdown() Now use Claude to extract the commercially important terms: Base subscription price. Included usage. Overage rates. Volume discounts. Billing period. Contract expiration. Special pricing exceptions. But here's the rule. Claude proposes the structured terms. A human verifies them against the signed agreement before they become executable billing rules. 2. DUCKDB — FIND THE BILLING DISCREPANCIES Connect DuckDB to your company's authorized usage data. Suppose a customer has: 100,000 API calls included monthly. 118,000 actual billable calls. An agreed overage rate of $0.04. That's 18,000 additional calls. Potential overage charge: $720. Now compare that against the actual billing records. If the charge is missing, the system flags it for investigation. Scale this across hundreds of customers and thousands of contracts. The important part? DuckDB performs the calculations. Not an LLM guessing what the invoice should contain. 3. STRIPE BILLING — VERIFY WHAT WAS ACTUALLY CHARGED Use Stripe's API to retrieve subscriptions, invoices, invoice line items, and relevant billing information. Stripe also supports invoice previews. Compare three things: What the signed agreement permits. What the application actually delivered. What the billing system recorded. Account for discounts, credits, negotiated pricing, and billing-period boundaries before declaring any discrepancy. A difference is an investigation lead, not automatic permission to charge a customer. 4. LANGFUSE — DETECT CUSTOMERS DESTROYING YOUR MARGINS Here's where the system gets even more interesting. Connect Langfuse to your AI application. It can track model calls, token consumption, and estimated inference costs. Now join those costs with customer revenue. Imagine a customer paying $299/month while generating $420 in model costs alone. Your dashboard might call them an active subscriber. Your financial model should flag them as potentially unprofitable. And the solution isn't necessarily increasing their price. You might first reduce repeated model calls, improve caching, or route simple requests to cheaper models. That's why I'd include both revenue recovery and cost optimization. 5. TEMPORAL — MAKE THE AUDIT RUN RELIABLY Use Temporal to execute the process on a schedule. Contract extraction. Usage reconciliation. Invoice comparison. AI cost analysis. Exception reporting. Human approval. Temporal preserves workflow state and supports retries. Use stable task identifiers and idempotent activities so a retry doesn't create duplicate work. Most importantly, keep the initial system read-only. No automatic customer charges. 6. TURN IT INTO AN ENTERPRISE PRODUCT This is the commercial opportunity. Sell it to: AI SaaS companies. API providers. Cloud software businesses. Enterprise subscription platforms. Companies managing negotiated usage-based contracts. Charge for onboarding, recurring monitoring, and potentially verified recovery services where contractually appropriate. Here's the hypothetical business model: 20 enterprise customers. $5,000 monthly subscription each. That's $1.2 million in annual recurring revenue. Not a revenue prediction. Just the scale of the opportunity if the product proves valuable enough to win and retain those customers. THE COMPLETE ARCHITECTURE Signed Contracts → Docling Extraction → Claude-Assisted Term Review → DuckDB Usage Reconciliation → Stripe Invoice Comparison → Langfuse Cost Analysis → Temporal Exception Workflow → Human-Approved Remediation → Revenue and Margin Dashboard The critical engineering decision? Never let the AI calculate bills from memory or automatically change customer charges. Use verified contract terms, deterministic pricing rules, source-linked evidence, and independent reconciliation. I'd also measure recovered revenue separately from detected discrepancies. Because finding a missing charge doesn't mean the customer legally owes it, or that the company successfully collects it. What excites me about this fusion is that it doesn't depend on generating more content, finding viral trends, or predicting markets. It solves a measurable financial problem for businesses that already have customers. A system that finds billing mistakes, exposes negative-margin accounts, and helps finance teams recover legitimate revenue could become extremely valuable. Most AI agents are trying to help companies make more money. I'd build one that first discovers where the money they're already entitled to is slipping through the cracks. That's the kind of AI infrastructure I'd seriously consider turning into a company.

Why Your LLM Integration Breaks at Scale Most LLM integrations work well in the pilot. They develop problems at scale. The root cause is usually not the model. It is the architecture assumptions baked in during the prototype phase that nobody revisited before production. Three patterns show up consistently. Synchronous by default. The pilot was built with synchronous API calls because it was simpler. Production traffic at scale cannot sustain synchronous patterns. Latency stacks, timeouts cascade, and the user experience degrades in ways that are hard to diagnose without distributed tracing. No circuit breaker logic. When the LLM API slows down or returns errors at volume, systems without circuit breakers keep hammering the endpoint. This creates cost spikes, queuing failures, and rate limit violations simultaneously. Context management at the application layer. Teams that did not design a clear boundary for context window management end up with business logic embedded inside prompts. As the application evolves, prompt complexity grows, costs increase, and behavior becomes harder to test and govern. None of these are LLM problems. They are integration architecture problems that surface when traffic increases. The pilot proved the concept. The architecture review proves the system. #LLMIntegration #EnterpriseAI #AIArchitecture #SystemDesign #EnterpriseArchitecture #SolutionArchitecture #AIAdoption

0.01 cents in tokens. Sold to you for $2. That's "credit" pricing: the biggest scam in SaaS right now. I mean the tools built on top of LLMs: enrichment, video, contract review, reports. Call the API, add some sauce, sell it as credits. People aren't stupid. They see: - no cost transparency - credits that feel arbitrary - bills nobody can predict - vendors changing what a credit buys - token prices are public, so anyone can do the math And the first cracks are showing: - vendors going back to per-seat and passing AI costs through 1 to 1 - LLM providers shipping the features you used to buy a tool for AI has a real variable cost. Sure, but charge for it honestly: - pass it through at cost - bake it into the license when it's tiny - show your markup - make the bill predictable My bet: within 24 months most credit pricing in GTM tools is gone. Once a few vendors go transparent, everyone else has to follow. I know from the founder's perspective selling credits is attractive. I advised various companies to do so in pricing projects. But honestly, the sentiment is switching. So alternative or at least transparent credit pricings will outperform black boxes. How do you use credit pricing in your project?

AI coding could kill my startup… But it might also save it. Both for reasons most people don’t understand. Here’s what I’m seeing 👇 We run a lot of our AI models at around 30% gross margin. The GPU prices we’re getting right now are doubling. That can pretty quickly wipe out any margin we have. Finding capacity was already hard. I know founders of $10bn+ companies where securing GPUs has basically become the CEO’s main job. And the capacity we do find is scraps. Bits and pieces that other large companies aren’t using. We’re going around trying to stitch together enough compute to run a business. My worry is that the success of AI coding makes this much worse. If you can make more money from a GPU serving coding customers than someone else can make serving video customers, you can afford to pay more for it. You get more compute. You grow faster. You can afford even more compute. Then there’s your team. If an AI coding company can offer your best people much more money than you can, that massively changes the dynamics too. You’re competing for the same GPUs and the same people, with completely different economics. And once those companies have the revenue, the compute and the talent, they can move into other markets. Including yours. More supply should help. But one year in AI feels like five years in the previous tech world. A lot of startups won’t have time to wait. I think Jensen should care about this. Surely Nvidia wants thousands of successful AI companies buying its chips, rather than a handful of customers with the money and incentive to build their own. So does that mean we should all go and build AI coding companies? Maybe. But here’s the weird part. As these coding models get better, they become useful for things we didn’t necessarily expect. We’re seeing models like Opus 5.5 enable video editing agents. The progress in coding is opening up a whole new way for us to build our product. So the same market that could price us out of compute and take our team is also enabling our next big opportunity. I think both things can be true. The opportunity is enormous. So is the pressure on the companies trying to build it.

This is the strangest property of LLM inference: Your LLM's output depends on requests that aren't yours. The same prompt can produce different logits depending on how many other requests share its batch on the GPU. Here's what happens inside the GPU. Most of an LLM's forward pass is spent accumulating sums. Every matmul, RMSNorm, and attention step reduces thousands of floating-point values into a single output, so the order of those additions directly affects the result. The catch is that floating-point addition isn't associative. Every addition rounds the result to the nearest representable value, so (a + b) + c and a + (b + c) can produce different low-order bits. fp16 has an 11-bit significand, so between 1024 and 2048 the spacing between representable values is 1.0. If you compute 1024 + 0.4 + 0.4 left to right, each addition rounds back to 1024. If the two small values are added first, 0.4 + 0.4 gives 0.8, and 1024 + 0.8 rounds to 1025. The order of accumulation depends on batch size, because kernels choose their parallelization strategy to maximize GPU utilization. A GPU like the H100 has 132 compute units, and a small batch doesn't give each of them enough work. So the kernel splits each long sum into chunks, adds the chunks on different units at the same time, and then adds those partial results together. So a different batch size gives a different partitioning, a different accumulation order, and slightly different logits. Usually the difference is negligible, except when two tokens are nearly tied. In a batch of 8: - "Queens" might score 12.3417 - "New" might score 12.3415 In a batch of 32, the rounding might shift them to 12.3414 and 12.3416. If temperature is 0, argmax will pick "Queens" in the first case and "New" in the second. Every later token is conditioned on a different prefix, so the whole completion diverges. So the nondeterminism isn't coming from sampling, but rather from the fact that your request's numerics depend on concurrent traffic you don't control. This is pretty evident at Temperature 0, but the nondeterminism comes from the forward pass, not the sampler. You can fix this by making every reduction batch invariant, meaning each row is reduced in the same fixed order regardless of how many rows share the batch. This costs throughput, because the kernel can no longer select the optimal split for each batch size. It uses one fixed strategy even when that leaves compute units idle. The major inference engines now expose this as a flag: → vLLM has VLLM_BATCH_INVARIANT=1, still in beta. → SGLang has --enable-deterministic-inference, at a 25-45% slowdown. → DeepSeek's DeepInfer has DEEPINFER_BATCH_INVARIANT=1. If you run evals, RL rollouts, or anything that needs reproducible outputs, enable it and treat batch composition as part of your experimental setup. And in production, the batch around your request never stays fixed. With continuous batching, the server adds and removes requests at every decoding step. I wrote an article that explains how it works and why every major inference engine uses it. Read it below.