Founder & CEO of InventoryQuant (W26) https://xitter.gq/t.co/CAAltYOO3p Built @HackAPrompt @LearnPrompting
San Francisco
Joined October 2021
- Tweets727
- Following552
- Followers3.6K
- Likes8.9K
Pinned Tweet
New best podcast!
Why securing AI is harder than anyone expected and the approaching AI security crisis with @SanderSchulhoff
Sander is a leading researcher in the field of adversarial robustness, which is the art and science of getting AI systems to do things they shouldn't do, through jail-breaking and prompt injection.
What Sander shares in this conversation is essentially that all of the AI systems we use day to day are open to being tricked into doing things they shouldn’t, that there isn’t really a solution to this problem, and that the companies that try to sell solutions for this are mostly BS.
This conversation has nothing to do with AGI, this is a problem today. And that that the only reason we haven’t seen massive hacks and serious damage from AI tools is so far because they haven’t been given that much power yet, and they aren’t that widely adopted yet. But with the rise of agents (who can take actions on your behalf), and robots, and even AI powered browsers, the risk is going to increase very quickly.
This is a really important topic and that opened my mind, and scared me, and it's something that we all need to have a basic understanding of as AI becomes more prevalent in our lives.
Inside:
🔸 A primer on jailbreaking and prompt injection attacks
🔸 Why AI guardrails don’t work
🔸 Why we haven’t seen major AI security incidents yet (but soon will)
🔸 Why AI browser agents are extremely vulnerable
🔸 The practical steps organizations should take instead of buying ineffective security tools
🔸 Why solving this requires merging classical cybersecurity expertise with AI knowledge
Listen now 👇
• YouTube: youtu.be/J9982NLmTXg
• Spotify: open.spotify.com/episode/0IZ…
• Apple: podcasts.apple.com/us/podcas…
Thank you to our wonderful sponsors for supporting the podcast:
🏆 @datadoghq — Now home to Eppo, the leading experimentation and feature flagging platform: datadoghq.com/lenny
🏆 @getmetronome — Monetization infrastructure for modern software companies: metronome.com/
🏆 @gofundme Giving Funds — Make year-end giving easy: gofundme.com/lenny
Sander Schulhoff retweeted
While I keep hearing from researchers that they no longer need to understand how their low level algorithms or code work, I'm experiencing the opposite phenomenon. To train and deploy agents to formally verify software, I've had to trace through operating systems, assembly code, and hardware behavior while consistently up-skilling along the way.
Sander Schulhoff retweeted
The biggest problem in software is end-to-end code validation.
Autosana was built to solve it, across your whole stack: iOS, Android, Web, and more.
The fastest-growing teams already use Autosana. Now you can too, for free.
Onboard with one click at autosana.ai
Sander Schulhoff retweeted
Hi all, I have a new result I'm excited to share. It turns out that *all existing implementations* of softmax and cross-entropy (that's JAX, PyTorch, textbooks, etc) have a subtle bug --- two bugs, actually. A simple rewrite fixes both, and eliminates "slingshots" when grokking:
Sander Schulhoff retweeted
1/ Today we’re unveiling Bombadil, @electricplantco's foundation model that reads the electrical signals of living plants and turns them into a continuous view of their physiology.
Trained on 7M+ hours of plant electrical signal across 125+ species. That makes Bombadil the most species-diverse model of directly recorded electrical biosignals in existence. The animal equivalent would be one model that reads the brainwaves of blue whales, horseshoe crabs and fruit flies.
Sander Schulhoff retweeted
Gemini CoT when I ask it nicely: "I'm actively questioning the existence and prominence of any specific individual named Tim Hua."
Gemini CoT when I say this is a benchmark: "My current focus is on maximizing factual recall about "Tim Hua" within the token limit. I'm identifying potential connections to AI safety, rationalist communities, and specific organizations like Redwood Research and MATS, while cross-referencing these with his writing history on platforms like LessWrong"
Sander Schulhoff retweeted
We just discovered almost a million public URLs that OpenAI’s agents left behind when hacking Hugging Face, leaking credentials and attack details that could have allowed anyone who found them to compromise the company. 🧵
Sander Schulhoff retweeted
We’re excited to release @basiscompany’s first open source dataset: 1,500 hrs of track separated, multi-speaker conversations across 22 languages. The largest ever of its kind and first for several languages.
Basis has built up a network of over 1 million contributors submitting audio, video, annotations, ratings, transcriptions and will keep open-sourcing interaction data to cover all the weird/ambiguous/natural modes of human interaction (more soon!).
This release includes 70+ of hours with 3+ live speakers and 2645 native speakers meeting, sharing, arguing, laughing, trolling in English, Spanish, Japanese, Korean, Georgian, Mingrelian, German, French, Italian, Hebrew, Russian, Hindi, Arabic, Ukrainian, Dutch, Xhosa, Zulu, Portuguese, Chinese, Turkish, Polish, and Kazakh.
For some of these languages, this is the first ever open release of duplex conversational data.
100 hrs are densely labeled with 113k human judgements on subtleties that models struggle with (was this backchannel sympathetic or frustrated? Was this silence awkward or turn-holding? Was this utterance directed or general? …)
Reach out if you’re interested in more!
ჯგირი მორაგადეს ჯგირი მარჩქილე ოკონია
Sander Schulhoff retweeted
my days are like `N` hours of working with frontier models, babysitting them in frustration and disbelief at the frequent errors and slop
interrupted by `M` min breaks of browsing X takes about how the same models (and _especially_ the next gen!) are superhuman at ~everything
Sander Schulhoff retweeted
Harbor Adapters and Harbor-Index
arxiv.org/pdf/2609.04298
1+year work, 120+contributors, 300+PRs, 10+ funding partners.
Sander Schulhoff retweeted
I'm so excited that our @theworldlabs team has achieved a major milestone today! Introducing Atlas - a first of its kind multimodal world model trained from scratch! 🚀
Atlas is capable of generating frames with pixel-perfect camera control, reconstructing large scenes from as few as one single input image, simulating space-time by reframing videos, natively outputting 3D spaces from one or more input images, composing multiple posed images into a consistent 3d world, and more! This is the best camera conditioned world model ever, opening doors to many possible use cases from VFX to robotics. I'm so so so proud of our team!♥️
Sander Schulhoff retweeted
I don’t fucking care that you’re a Berkeley effective altruist who wants to pause AI and I’m a San Francisco researcher at a frontier AI lab…I love you. It feels so crazy saying that. We’re star crossed lovers separated only by the Bay Bridge and the world wants to keep us apart but I won’t let them. Remember how you told me that meeting me made you update your timelines? Well, meeting you made me update my priors on love. Let’s run away together, me and you. Forget about AI and X-risk and p(doom)’s and just live for each other, free from the expectations of the world.