Stephen Lee

LLMs Don't Know What They Know: Benchmarking Temporal Self Knowledge for Financial Backtesting A new paper exposes a serious flaw in AI investment backtesting: models may already know the outcomes they're supposed to predict. The researchers tested 2,226 dated financial and economic facts. GPT-5 correctly answered 76% of the controlled questions but claimed to know only 38%. Even among facts it said it didn't know, it answered 61% correctly. Asking the model to answer as of a historical date made it much less likely to admit prior knowledge without materially changing its accuracy. This creates a problem for AI investment backtesting. Suppose you give a model financial statements from 2020 and ask it to predict a company's next earnings report. You can restrict its document searches to information available at that time. But what if the model already encountered the outcome during training? A great backtest might simply be a great memory. That doesn't mean every correct answer was memorized. It means the model's own assurances cannot establish that the test was clean. For an AI investment system, the real test should be prospective. Record the forecast before the event. Record the evidence and the decision. Compare outcomes against a fixed baseline. Measure whether the resulting positions add value after costs and at comparable risk. Then use those results to decide which research methods deserve more resources. The goal is to build a system that gets better at predicting the future, not one that gets rewarded for remembering the past. arxiv.org/abs/2611.06808

dynamic pricing is going to get ridiculous in the AI era. give it another 10 years. your coffee machine knows you slept 3 hours and have a meeting in 8 minutes. you ask for an espresso. “of course. that’ll be $49.” you reach for the plug. your smartwatch immediately tells it you’re price sensitive. “good news. you qualify for a personalized discount.” $48.99. you remind it that you bought the machine and the beans. “absolutely. that’s why there’s no delivery fee.” you decline the tip. “no problem. your next coffee will be $59.”

An investment firm’s research budget could build two assets. The answer to today’s question. The material that helps improve tomorrow’s work. Palantir’s approach to open models makes that possibility more concrete. Customers can adapt models such as NVIDIA Nemotron using their own operating data. For investment teams, useful examples could come from everyday research. An analyst finds evidence that weakens a thesis. A review reveals an assumption that was never tested. New information changes a forecast. Capture what was known at the time. Keep the correction and the reason for it. Then ask whether those examples help a model catch similar problems in later cases. That requires clear labels and a test the model has not already seen. The improvement still has to justify its cost. If it works, research spending could produce something that stays useful after the original question is answered. The next analyst starts with more context. The next model has better examples to learn from. A firm could get more value from the same work by making what it learns easier to reuse. That is the kind of compounding a research budget should aim for. nvidianews.nvidia.com/news/n…

A profitable trade can teach an AI the wrong lesson. Say a model buys a stock because it expects an earnings beat. Earnings miss. The company receives a takeover offer. The stock jumps. The trade made money. The earnings thesis did not play out. For a system that learns from outcomes, both facts matter. Keep the original forecast. Keep the evidence behind the decision. Keep the risk taken alongside the return. Then examine which assumptions held up and which ones failed before deciding what to reinforce. Profit matters. So does knowing what deserves repeating. An investing system should learn from its wins as carefully as its losses.

An AI forecaster became better calibrated while making fewer searches. An October study trained a model on 2,113 resolved Polymarket questions. The model chose what to search and read before producing a probability forecast. On 265 separate test questions, the researchers reported: Average search attempts fell from 3.8 to 2.25. Calibration error fell from 0.185 to 0.128. Its stated probabilities matched observed outcomes more closely. It also repeated fewer searches after empty or filtered results. That points to a useful goal for investment AI: learning where to spend the research budget. Which missing fact could materially change a valuation? Which evidence would change the confidence behind a position? When is another search unlikely to improve the decision? Those questions become increasingly valuable as we give agents more freedom to research. The study tests historical forecasts. Whether this translates into investment returns after costs remains a separate question. Research budgets deserve the same discipline as investment capital. arxiv.org/abs/2610.01955

What happens when AI investment research can finance its own improvement? The key relationship is between additional research spending and additional investment value. If the next dollar of research produces more than a dollar of value after trading costs and at comparable risk, part of those gains can fund the next round. Measured outcomes can then help identify which methods deserve more resources. The system can learn where deeper research pays and where it should stop. That creates a path toward a financial institution that compounds both its capital and its decision making capability. The difficult part is attribution. A profitable trade alone does not show that additional research helped. The process needs a credible baseline and a record of what was known before the decision. What evidence changed the forecast? How did that change the position? Did the resulting decision justify the additional cost? That is why preserving investment reasoning matters. It gives an institution a way to test whether its intelligence is becoming a productive asset.

Giving an AI forecaster memory made its research cheaper. It did not consistently make its forecasts better. Forecast Dojo tested 12 models on 230 held out prediction market events. The agents revisited each event at multiple historical dates. Researchers compared starting fresh at each date with carrying a notebook of earlier findings and forecasts. Both versions had the same research tools. Across five commercial models with available pricing the notebook reduced estimated API cost per forecast by a median 24%. Six of the 12 models improved their Brier scores. None of those improvements was statistically significant. Two models had significantly worse scores. The useful distinction is between reusing information and improving judgment. An agent can avoid repeating its research without becoming better at assigning probabilities. The result is specific to this notebook design and historical evaluation. For AI investing the value of memory should be measured in both forecast quality and the cost of producing it. arxiv.org/abs/2609.28876v2

imagine every physical thing has AI. your fridge, your shower, your toilet. everything can talk. and of course, everything has ads. you’re dropping a deuce and the toilet goes: “while you’re here, let’s talk about refinancing your mortgage.” you escape to the shower: “today’s shower thoughts are brought to you by our sponsor.” even the chair has something to say. “you’ve been sitting a lot lately. have you considered a gym membership?” you ask your house for five minutes of silence. “absolutely. would you prefer monthly or annual billing?” we’re going to make everything intelligent and then pay it to shut up.

Changing the reward changed how the AI forecasted, even with the same starting model. Researchers compared five probability scoring rules while holding model initialization, training data and other training settings fixed. The experiment used 7,076 training examples and 965 questions reserved for evaluation. All five trained variants produced lower observed Brier scores than the base model. In the main comparison, Brier training achieved the lowest Brier score. Log score training achieved the lowest calibration error, meaning its stated probabilities aligned more closely with observed outcome frequencies. The rewards all theoretically encourage honest probability estimates. In practice, they produced different calibration and error patterns, even when overall performance was similar. Some rankings changed across repeated training runs. The study used one base model and limited repetitions, so it does not establish a universally best reward. The training objective affected both forecast accuracy and the kinds of mistakes the model made. arxiv.org/abs/2608.28482

Two individually weaker AI models could outperform the strongest model. A study of 70 forecasters examined 1,121 pairs. Weaker pairs matched or beat the strongest individual in 11 of 16 comparison groups. Adding more models did not consistently improve performance. The combination weights were fitted on separate training data. However, the strongest individual and successful pairs were identified using their test scores. The study demonstrated that effective combinations existed, without establishing a method for selecting the winning pairs before outcomes were known. The cost result was less favorable. When the researchers required lower combined input prices, pairs matched or beat the strongest individual in only 2 of 16 groups. A weaker pair could produce a stronger forecast without necessarily producing a cheaper one. Two Heads Are Better Than One: Aggregating Weaker LLMs for Better Forecasts arxiv.org/abs/2609.33257

The next frontier in AI investing is improving the full decision process: how evidence becomes a forecast, how a forecast informs capital allocation, and how outcomes improve future judgment. The research is giving us a clearer picture of where value can be created. AI Augmented Predictions studied 991 participants answering six forecasting questions. Frontier LLM assistance reduced forecasting error by roughly 24% to 28% compared with a weaker assistant. ForecastBench’s July 2026 update reported several AI systems statistically indistinguishable from superforecaster accuracy. That comparison relies on extrapolating human forecasts collected in 2024, which limits what we can conclude about current human performance. Together, these results establish a serious basis for developing AI assisted judgment. The next challenge is connecting that judgment to economically useful decisions. Prediction Arena gave six agents $10,000 each to trade live on Kalshi for 57 days. All six finished down, with reported losses ranging from 16.0% to 30.8%. When Do Prophets Profit in Prediction Markets? addresses the connection more directly. It develops betting rules linking forecasting accuracy relative to market prices with positive expected profit under specified conditions, including sufficient liquidity. Its live pilot started with $200 and ran for 26 days. The reported trading return was 80.33%, falling to 24.2% after excluding the three best and three worst trades. These studies use different systems and evaluation conditions. They identify separate problems that need separate measurement: forecast quality, trade selection, position sizing and execution. A probability estimate only becomes economically useful in relation to the available decision. If an event has an estimated 70% chance of paying $1, buying the contract at 80¢ implies an expected loss of 10¢ before fees. The Kalshi research adds another dimension. Its historical sample shows a bias toward longshots and higher returns for makers than takers. Market structure and the price obtained influence the value captured from a forecast. There is also an opportunity to improve the underlying models. Bridgewater’s SQL research combines data verified by experts with training rewards designed around specific failure modes. It reports 91.37% accuracy at $0.035 per task with one sample. Using 16 samples increased accuracy to 92.97%, at $0.56 per task. The result is specific to SQL. For investment research, it raises a concrete question about how expert knowledge, verified evidence and better training objectives can improve judgment per unit of compute. My view is that this is where the research becomes especially valuable. An informative evaluation separates errors in evidence, probability estimates, trade selection, sizing and execution. Each failure calls for a different correction. That turns a return number into a diagnosis of what should improve. It also creates a more useful feedback loop for model development. We can identify recurring reasoning errors, improve calibration and determine which additional research changes a decision enough to justify its cost. The standard should be explicit: Forecasts scored against outcomes. Decisions evaluated against alternatives. Economic value measured after fees, slippage, data and model costs. Advancing the frontier means improving these capabilities together, with enough transparency to explain where the improvement comes from and whether it persists. Research links: AI Augmented Predictions arxiv.org/abs/2402.07862 ForecastBench July update forecastingresearch.substack… Prediction Arena arxiv.org/abs/2604.07355 When Do Prophets Profit in Prediction Markets? arxiv.org/html/2607.06166v3 Makers and Takers karlwhelan.com/Papers/Kalshi… Bridgewater SQL research bridgewater.com/aia-labs/put…

AI shouldn’t be our end goal. AI should be omnipresent, omniscient, and omnipotent. Hence, ASI

Google must be cooking some serious filet mignon

we labeled reality for ai. the interesting part starts when ai labels reality back.

“We’re early” has done generational work for bad investments.

AI may have just created the biggest paradox in software economics: The cheaper it becomes to produce code, the more expensive it becomes to verify that the code should exist. We are approaching a world where generating 1M lines of software is trivial. Understanding 1M lines of software is still human scale. The bottleneck isn't coding anymore. It's entropy.

Your investment memo is a photograph. Your thesis is a living thing.