Pinned Tweet
Lately, I have been trying to beat Google.
Ok not really xD
I have been building a search engine over niche web data and it's been one of the most fun projects I've ever worked on.
The idea of querying the internet exactly the way I want, at scale, has always fascinated me. Especially what I could do with it: look at 1,000 companies' job postings and find hiring trends using natural language, map competitive landscapes including the lesser known players, find alternative data for investing. Or just a personalized corner of the web, curated for me.
It's been an interesting journey.
Started out with 200 workers on localhost, rewrote the crawler from Python to Rust to get up to 1,600 workers in parallel. Moved it to the cloud. Hit challenges that made the whole Rust rewrite useless xD
Then I woke up to an abuse report from a French website with the words "European Commission", "appeal", "court" in it. For a second I genuinely thought I was about to have legal proceedings against me. Phew. Nevertheless it's been the most fun project I've worked on. It's surprising how you can cover a lot of information starting from a modest number of seed URLs.
I'll be sharing everything I learned in the open. Code will be open sourced soon and the search engine will be live too.
If you're curious how to build a distributed web crawler that covers 10M+ pages under $200/month, do read my blogpost.
Blogpost Link: lnkd.in/g7MsaXYA
hrishikesh kamath retweeted
None of these security incidents are a “they deserved it / had it coming” situation. We’re all next.
And if you’re a competitor using this moment to poach customers, remember this is going to happen to you too. Sooner than you'd imagine. And then everyone’s going to remember the shade you're throwing at others right now.
Have some empathy.
hrishikesh kamath retweeted
Good reminder that Cloudflare is a cool mission driven company and actually we need our infrastructure providers to be competent and forward looking
(Not a shareholder)
PHP changes are live. Start patching. php.net/archive/2026.php#202…
Huge thanks to coordinated work by @ThePHPF!
hrishikesh kamath retweeted
"any old projects you've long forgotten about will be found and used against you"
This is great advice for everyone - companies and individuals.
Ask your agents to look through your old projects, scan your github, domains etc and see what needs updates, fixes, or just shutting
i'm sure you're all better than me at this but just some of what we've seen if it helps - they're not going to stop. it's going to get increasingly sophisticated and they're going to go after every surface of your system and all of your employees
any old projects you've long forgotten about will be found and used against you
the only reason we're surviving is because we have agents going 24/7. we're still barely sleeping
good luck, sorry you're dealing with this, but really it's a collective problem we're all facing together
What’s funny is that Muse and Instinct became popular because people were tired of Claude’s verboseness and jargon maxing
hrishikesh kamath retweeted
Going to start adding more tutorials on how things work and this one for @papertrade_xyz is pretty good.
Looking for more folks to test the papertrade features in the mobile app!
hrishikesh kamath retweeted
It can be simultaneously true that we’re going to need a shit ton of GPUs and that the equity return of owning GPUs isn’t going to be super high. I’m biased but I think most value will accrue in the software layer – applications and infrastructure.
hrishikesh kamath retweeted
Progress of AI in finance, from someone using it every day in an investment seat
March 2026: “AI is kind of like an intern. It takes some tedious tasks off my plate but always need to check its work. Things are often pretty wrong”
June 2026: “Wow, it is more like an analyst now. It can do things with decent accuracy, but needs a lot of detailed instructions. It still hallucinates every now and then”
October 2026: “AI is doing almost all of my excel modeling and grunt work. I feed it assumptions and instructions, but it can get 80% of the way there by itself without needing any special prompts. I can ask it to be creative with the analysis it puts together, and it does a pretty good job”
The change of pace has been incredible to watch
I don’t get restaurants that give you free fries or chips with takeout but not when you dine in. Is it a consolation prize for missing out on the dining experience?
Nobody is talking about the search for agents infra goldrush, I will talk about it soon.
🔍 Search & Fetch APIs are now open to everyone!
Powered by our in-house search engine, serving hundreds of millions of requests daily worldwide.
💰 Each endpoint: $0.75 per 1,000 calls.
📖 Docs: api-docs.deepseek.com/guides…
📌 Search & Fetch calls via server-side tools in DeepSeek Harness, Claude Code, and Codex are currently free for DeepSeek API users. Billing starts Oct 31, 2026, at 16:00 UTC.
3/4
hrishikesh kamath retweeted
And interesting frontier lab strategy would be to encourage distillation, set price that mass distillation is unit economic positive, and then just focus on building the best models. This results is a downstream ecosystem addicted to you. And with no capability to surpass.
hrishikesh kamath retweeted
I wonder if we'll get to a point where being distilled becomes profitable to the labs. If they spend X to create a model. And more than 20 companies pay 1/20th of that to distill it. They're in the money. It's a new form of infra where downstream handles distribution.
hrishikesh kamath retweeted
Exclusive: Instinct is using Exa for web search
We gave Instinct URLs for nonexistent pages on a website we administer. Since the requests were for fake pages and the timing matched our conversation with Instinct, the requests couldn’t plausibly have come from anything else.
In total, it made about 12 requests over an hour and a half. We checked Cloudflare analytics and the server’s raw request logs to get the full picture.
A few things stood out:
- Instinct used Exa to fetch pages. Every direct fetch arrived with the user agent Mozilla/5.0 (compatible; ExaUser/1.0; +crawler.exa.ai/exauser), from a different IP each time.
- Exa signs its requests (Web Bot Auth, Signature-Agent: "crawler.exa.ai"). But nothing in the requests identifies Instinct. They are indistinguishable from those of any other product built on Exa.
- In one test, after the fetch returned a 404, Instinct loaded the page in its browser without being asked.
- Instinct’s browser had no self-identifying features. It looked like standard Chrome on desktop Linux from a residential IP address, with no bot user agent or signature.
We directly asked it to use the browser in two other tests. All three browser visits came from the same residential IP over about 80 minutes. The browser kept its cache between visits, consistent with a persistent browser behind a residential proxy. We can’t say whether that IP is stable over longer periods or shared with other users, so it’s not a reliable way to identify Instinct.
This presents unique challenges for marketers hoping to track Instinct traffic to their site. Unlike ChatGPT, Claude, or Meta, Instinct doesn’t send its own user agent or signature. Its fetches look like generic Exa traffic, and its browser visits look like a normal user’s.
All of this means you can’t track, allow, or block Instinct specifically. The only reason we could link these requests to Instinct was that we controlled the URLs and the timing. For normal traffic, you don’t have those luxuries.
Link to it on sunbeam: sunbeam.ai/thoughts/instinct…
Inference this inference that, I am sure @thinkymachines is crushing it silently. They are heads down building without them having to post on Twitter. I joined @cosign to only rate them as highest aura company.
hrishikesh kamath retweeted
Coming soon to Tailor by @buttonxyz (dm for TestFlight)
$LLY seems a great hedge till Anthropic publishes their S1, no more forcing bad trades