Tokenization is the first step in every LLM pipeline — and we barely look at it.
I built TokEntropyDrift to fix that.
It's a tokenizer drift + entropy explorer for researchers, prompt engineers, and LLM nerds like me.
2
1
103
🔧 Written in Go
📦 CLI + plugin support
📊 Designed for research pipelines, prompt debugging, or CI-based tokenizer analysis
🧪 Built for scientific reproducibility
1
89
Check it out here:
👉 github.com/RevBooyah/TokEntr…
Docs + screenshots in the README.
If you work on LLM infrastructure, tokenizer internals, or just like to see under the hood — I’d love your feedback.
72
Different tokenizers (BPE, WordPiece, SentencePiece, OpenAI’s, etc.) break down the same text differently.
This affects:
• Compression
• Prompt consistency
• Model performance
• Debuggability
And yet… nobody’s instrumenting this.
1
65
Steve Cook retweeted
i vibe coded a little game called Coldplay Canoodlers
you're the camera operator and you have to find the CEO and HR lady canoodling
10 points every time you find them
👇link
Steve Cook retweeted
After 15 years at the Tour de France, the Shimano mechanics who saved Tadej Pogacar after his crash are happy every time they help a rider, whether they are in first or last. ❤️ #TDF2025
Can You Do It All? Lessons from Johnny Walker youtube.com/shorts/_hO0qQPRZ… via @YouTube
36
Steve Cook retweeted
Every airplane on Earth:
Emirates Flight 129: LEEEEEEEROYYYYY JENKINNNNNNNNNSSSS
Steve Cook retweeted
all the cyber security contractors watching everyone deploying vibe coded SaaS apps
When do we get the Grok app in the Tesla? I'd love to have conversations while driving. Thanks @elonmusk
59
Steve Cook retweeted
This from @ThePrimeagen is spot on
Steve Cook retweeted
Instead of “how much time a task will take,” asking “what task can I complete in the available time?” has been a good productivity boost.
Pleased that I was able to finish a feature and refactor some code in the 15 minutes of free time this morning before heading to airport.














