man in ✨ fínãnce ✨🥴 love to hate nyc 🐀🥀 @sportfoliokings @bettorgpt
d*mes sq**re
Joined February 2021
- Tweets6.1K
- Following319
- Followers3.1K
- Likes34.1K
deepvaluebettor retweeted
Since developers aren’t coding anymore, their new job is to rotate secrets every single fucking day
some model outputs looked sus , dug into the code , turns out astra 6 was conflating calendar year as season year and was using jan 2025 week 18 as week 18 for 2025 season 😭
this is why there is so much slop ... bc u need *some* domain knowledge to even detect something is off ... && then have at least *some* coding literacy to actually be able to read the code
deepvaluebettor retweeted
both of these are true:
1. frontier models are at the cusp of fully automating knowledge work
2. we are nowhere near the reliability and cost-efficiency needed for a lot of enterprise workflows
it's a crazy time and I have a lot of cognitive dissonance about this.
we hope this work shows just how early we are, even on a task as "simple" as document extraction.
super proud of the team for pushing the frontier on synthetic data and evals on this one.
Introducing ExtractBenchPro: the most challenging benchmark for real-world document extraction.
We see a massive long tail of document complexity in terms of use cases brought on from our customers, that are completely unsolved from a price-performance standpoint. These include trial balances, well logs, canvass binders, site plan reviews, duty reconciliations, loss runs, and more.
Most existing extraction benchmarks are saturated by the frontier models (our cheapest modes now score 93%+ on ExtractBench). So there is enormous economic value in building a complex extraction benchmark that reflects this long-tail of knowledge work.
We spent a long time curating the most complex elements that we saw from real world extraction use cases:
✅ Document conventions: containing legends or rules where the rest of the document should be interpreted
✅ Density of internal jargon: specific definitions that must be cross-referenced within a doc
✅ Aggregation and reasoning: beyond returning exact values on a page, infer values derived from aggregating multiple sources of information
✅ Drawn relations: complex diagrams like procedures and circuits
✅ Charts inside documents: read precise points within a chart
✅ Long chart sheets: some data series like well logs cram all this data into an extremely long page
Within the industries where document extraction is relevant, even small mistakes can have enormous negative impacts. Therefore our headline evaluation metric is based on perfect documents: the proportion of docs with perfect (100%) value f1 score.
(Besides this we also measure value f1 and grounded f1)
It’s an extremely hard task for even the latest frontier models to solve, topping out at ~58% or so in Value F1 score for Codex with Astra - which is quite good but extremely expensive (almost $1 per page!) Our LlamaParse agentic/agentic plus modes are 20-40% and cover the left side of the Pareto frontier, at price ranges that are feasible for full production use.
ExtractBenchPro is available today as a website/technical blog, and also on Github and HuggingFace. Check it out!
Blog: llamaindex.ai/blog/introduci…
Website: extractbenchpro.ai/
Hugging Face: huggingface.co/datasets/llam…
Github (is ExtractBench): github.com/run-llama/Extract…
srs there's so many of these clips made w ai that i honestly can't tell what is actually real anymore ( && w caleb it's esp difficult since he's so corny + memetic by nature ) ... it's not like i'm gonna due diligence every damn clip to confirm its authenticity ... 😑
deepvaluebettor retweeted
This is the least surprising graph in the world if you understand how gradient descent works.
this is the super bowl for the low-tier normie reddit school shooters of gambling twitter
unavailable
i miss the days when u needed to have at least a semblance of ball knowledge & modeling skill to make proper data viz . the dataslop is overwhelming . it's everywhere .
this is what ur median normie npc thinks ai is capable of ... just a total lack of understanding of how the models work && what their strengths + weaknesses are ...
if i ran a major league baseball franchise i’d feed every stat imaginable into frontier ai models & basically delegate the roster construction + entire game management layer to them. no traditional manager & no gm making baseball decisions by gut ever. the manager & gm are both ai agents.
for every game every pitch would update the model including exit velocity, pitch velocity, spin, fatigue, matchups, positioning, bullpen availability / usage, etc. & it would continuously simulate thousands of scenarios & make the highest expected value decision in real time.
then over a 162 game season it watches every game afterward through precision cameras, grades every decision, learns from every outcome, & gets better.
baseball might be the perfect sport for this. it has discrete events, absurd amounts of data, repeated situations, & a gigantic sample size.
billy beane found mispriced players but this finds mispriced decisions too on every layer.
can some billionaire please buy an mlb franchise & let me run this experiment pretty please?
this would be the houston astros on steroids. imagine ai winning a ring?
cc @JoshuaKushner
ETR chuds join survivor markets this season smugpoasting like they own the place only to galaxybrain themselves into getting taken to the woodshed week after week ⚰️
deepvaluebettor retweeted
Obama is perfectly displaying the concept of hallucination with LLMs here.
He sounds so darn good and confident, while having absolutely no clue wtf he sayin.
🚨 OBAMA ON AI: "If we are thinking about AI just in terms of how do we cure cancer or get better energy, you can do that without having agentic AI and having it just roaming free in the internet. The reason you are doing that is because you have to market a product that people will pay money for.
That’s a misalignment between what our society needs and the commercial imperatives that these companies are facing, not because necessarily they’re trying to do bad things, but because they’ve got to justify these valuations.
So, that’s one more reason why it is really important for us to have a competent government and a serious bipartisan conversation around this issue, and we have to do it fast. And I would encourage voters to pay attention to this. If somebody does not have a serious plan for how to deal with this, then they’re not meeting the moment, and you should probably look for somebody else."
rewatch an entire NFL game in under 3 min , underpinned by @sportfoliokings interactive volatility surface && in-game trading analytics
why this is mostly more bs misdirection ::
1 . dario , sam , && elon have been alarmist for yrs ( dario thought gpt-2 was too dangerous to release publicly )
2 . the labs are running out of non-synthetic data ( the internet has been fully scraped )
3 . the labs are facing major infrastructural constraints
4 . the labs need more time to find solutions to protect their IP fr the chinese , who continue to find clever ways of stealing it && using US frontier models to ramp up their own development
5 . the labs realize the normie public is already super antagonistic to data centers && ai in general , && the " slow down " ai narrative resonates well from an image perspective
6 . the labs also realize that using ai to train ai ( recursive self-improvement ) accelerates development but also causes them to lose control && visibility into what's going on && why . they all ack that to reduce pressure to do this they need coordinated commitment .
end .
