Never write another web scraper. Diffbot structures information from the web, so you don't have to.
The Future
Joined September 2009
- Tweets2.1K
- Following7.6K
- Followers8K
- Likes10.2K
Step 1: Build a free DNS provider
Step 2: Collect tolls for access to sites behind their DNS
Step 3: Build a crawler that doesn't need to pay their own toll
Step 4: Profit
Diffbot 🤖 retweeted
Ever wondered what your white name should have been? Introducing: whatismywhitename.com
Upload a picture of you, and let the puppy guess your name! Let's test out nominative determinism 🫡
(Immigrants who named themselves will correlate more highly. Give us feedback plz)
Our thanks to:
- @modal for their generous credits toward training this meme model
- @diffbot for the clean, diverse dataset!
- @leannch86920 for the training research!
- Everyone NOT named David (biggest & noisiest dataset ever)
Sorry! It’s not ideal but it’s the easiest way to weed out 99% of abusers. When the product makes it easy to crawl the web, you get a lot of bad actors.
Still thinking of a better way to solve this!
Sorry for the late reply (and happy new years!)
It's not on the immediate horizon, but implementing a credit balance model with a low minimum is something we've discussed. I personally prefer it.
Would you mind emailing me at jerome[@]diffbot?
Diffbot 🤖 retweeted
State of E-commerce Data Providers - Q4 2025
E-commerce runs on constant measurement: prices, promos, availability, seller changes, and "what the shelf actually looks like" across retailers and marketplaces.
The challenge is stable collection at scale, retries when sites break, anti-bot evasion, clean geo signals, and then turning messy HTML into usable structured data.
In preparation for the holiday season, we mapped the landscape of e-commerce data providers:
Competitive intel + digital shelf: @dataweavein, @Price2Spy, @bigdataNODE, @Profitero, @WiserInc
Marketplace intelligence + data: @junglescout, @H10Software, @datahawkco, @SellerSprite_EN
Trade, Supply Chain, Imports / Exports: @Trademo1, @ImportYeti, @datamyne
Scraper APIs & Extraction Platforms: @zytedata, @diffbot, @Stratalis, (AutoScraping handle?), @serpapi
Managed Data Extraction & Services: @groupBWT, @Data_Ox, @epctex, @MrScraper_
Retail Media & Ad Platforms: @Pacvue, @PerpetuaLabs, @Teikametrics
Network & runtime infra for e-com scraping: @playwrightweb, Puppeteer, @browserless
Diffbot 🤖 retweeted
YouTube, TikTok, Mastodon, & Threads are mostly there but need optimizing.
Diffbot goes incredibly far with articles & that’s also moving along well.
Reddit & Bluesky are readily available but I haven’t spent the time.
X is finished by the endpoint gets rate limited 😞
Diffbot 🤖 retweeted
San Diego developers, join us and our technical partners @neo4j, @Intuit, Eyepop.ai, @Replit , and @diffbot at our HackNight next week!
We're excited to join @neo4j , @Intuit, EyePop.ai, @Replit and @diffbot as technical partners for the upcoming Startup San Diego - FirstWave Innovator HackNight happening Wednesday, February 19th at the Intuit San Diego Campus.
This is going to be an epic night where 100+ developers will come together to create innovative solutions for five select startups.
🚀 Join as a developer for free or get tickets: lu.ma/4brcg3lz
89,886 developers are building their own Perplexity on-prem with Diffbot LLM — huggingface.co/diffbot/Llama…
The model isn't the moat. Perplexity can be recreated as a side project. #DeepSeek proved this. We proved this.
Download Diffbot LLM. Run it off your own GPU. Congrats, your on-prem #AI is smarter than #Perplexity.
Let's be frank — The score difference is insignificant. And we'll probably play SimpleQA tag for awhile.
What IS significant is how we got here vs. Perplexity.
1. Diffbot LLM is a side project. Sonar is Perplexity's entire business.
... so I set it up to run the 4000 question eval on Diffbot LLM overnight and went to bed.
The next morning, we beat Sonar Pro.
While working on my talk last week, Perplexity released Sonar Pro API with a special emphasis on its factuality benchmark F1 score of 0.858, handily beating other internet connected LLMs like Gemini-2.0-flash.
The SimpleQA benchmark they used is open source and LLM judged...
#Perplexity Sonar Pro API launched last week as the best performing model on factuality.
24 hours later, it's the 2nd best performing model (and it's not because of #DeepSeek).
Why? 👇
Diffbot 🤖 retweeted
Diffbot launches open-source AI model that achieves 81% accuracy by querying a trillion-fact Knowledge Graph in real-time instead of relying on static training data 🧠📊
Read more: venturebeat.com/ai/diffbots-… #ArtificialIntelligence #Enterprise #MachineLearning @diffbot
