
CEO @ https://xitter.gq/t.co/Uv4i6N1WuD // Author of Introduction to Machine Learning https://xitter.gq/t.co/Pq0NnnFdrE Previously: Head of ML @WolframResearch // Physicist @ENS_ULM & @MIT
Boston, MA
Joined June 2014
- Tweets294
- Following813
- Followers708
- Likes3.1K
Pinned Tweet
We are releasing NuExtract3, a 4B open-source (Apache 2.0) reasoning OCR & Structured Extraction VLM 🧠📷📄
NuExtract3 is the first reasoning VLM specialized in both OCR (converting PDFs/Scans/Spreadsheets into Markdown) and structured extraction (document to JSON via a schema).
We trained NuExtract3 from Qwen3.5 4B via an SFT and an RL phase to provide its reasoning abilities which can be turned on and off. We find that NuExtract3 outperforms similar size models (<30B) in both the structured and OCR tasks, including specialized OCR models, making it the new reference for open-source document extraction😀.
Congrats Alexandre Constantin, Sören Dréano, and @NathanFradet for this model! Looking forward to the Pro version :)
Available on @huggingface. Links in the reply.
🤖 Made with AI
Etienne Bernard retweeted
The ideal outcome for science isn't a couple of labs using proprietary models and millions of dollars to crack trophy problems, it's to have millions of AI-aided human scientists who follow their curiosity and become way more ambitious
Etienne Bernard retweeted
ENOUGH PESSIMISM IN AI PLEASE
I feel that we have become unreasonably pessimistic in our field.
1. I keep hearing AI engineers saying we have to make money quickly because there’s only like 2 years left before we’re automated. Depressing.
2. I see a constant obsession with “having a moat”. This is an incredibly sad mental frame.
3. I keep hearing “we have to catch up”. Soulless.
And so on. People: Every solution creates the possibility to attack new real problems. We face gargantuan engineering challenges in our world. How to capture carbon? How to get rid of teflon and plastics in water? How to invent batteries that are at least 30 times more efficient? How to solve clean energy? Better solar cells? Better ways of producing clean energy so we stop wars and famine? How to eradicate hundreds of diseases? Cures for addiction? And so on. Real engineering is about being brave and truly attacking the many problems we face, to engage with a true desire to improve the lives of others and our environment.
Good engineering is not about protecting your product to make money at the expense of progress (moat thinking). Good engineering is about ensuring your children and grandchildren will be proud of the choices you made in 30 or 50 years. It is about empowering others. It is about advancing science. It is about being one step ahead. Always, one step ahead, meaningfully, proudly.
These are great times. Let’s start thinking positively about all the wonderful things we could achieve together.
Relevant links:
- 📖 blog post: about.nuextract.ai/blog/nuex…
- 🤗 Model: huggingface.co/numind/NuExtr…
- 🤗 Space: huggingface.co/spaces/numind…
- 🖥️ API access: nuextract.ai/
- 🗣️ Discord: discord.com/invite/3tsEtJNCD…
- 🛠️ GitHub: github.com/numindai/nuextrac…
1
162
Etienne Bernard retweeted
We’re excited to introduce Doc-to-LoRA and Text-to-LoRA, two related research exploring how to make LLM customization faster and more accessible.
pub.sakana.ai/doc-to-lora/
By training a Hypernetwork to generate LoRA adapters on the fly, these methods allow models to instantly internalize new information or adapt to new tasks.
Biological systems naturally rely on two key cognitive abilities: durable long-term memory to store facts, and rapid adaptation to handle new tasks given limited sensory cues. While modern LLMs are highly capable, they still lack this flexibility. Traditionally, adding long-term memory or adapting an LLM to a specific downstream task requires an expensive and time-consuming model update, such as fine-tuning or context distillation, or relies on memory-intensive long prompts.
To bypass these limitations, our work focuses on the concept of cost amortization. We pay the meta-training cost once to train a hypernetwork capable of producing tasks or document specific LoRAs on demand. This turns what used to be a heavy engineering pipeline into a single, inexpensive forward pass. Instead of performing per-task optimization, the hypernetwork meta-learns update rules to instantly modify an LLM given a new task description or a long document.
In our experiments, Text-to-LoRA successfully specializes models to unseen tasks using just a natural language description. Building on this, Doc-to-LoRA is able to internalize factual documents. On a needle-in-a-haystack task, Doc-to-LoRA achieves near-perfect accuracy on instances five times longer than the base model's context window. It can even generalize to transfer visual information from a vision-language model into a text-only LLM, allowing it to classify images purely through internalized weights.
Importantly, both methods run with sub-second latency, enabling rapid experimentation while avoiding the overhead of traditional model updates. This approach is a step towards lowering the technical barriers of model customization, allowing end-users to specialize foundation models via simple text inputs. We have released our code and papers for the community to explore.
Doc-to-LoRA
Paper: arxiv.org/abs/2602.15902
Code: github.com/SakanaAI/Doc-to-L…
Text-to-LoRA
Paper: arxiv.org/abs/2506.06105
Code: github.com/SakanaAI/Text-to-…
Etienne Bernard retweeted
Incredible news out today.
The same mRNA tech used in COVID vaccines is now training the immune system to recognize triple-negative breast cancer.
In a new study, 10 /14 patients remained relapse-free at ~5-year mark.
This is why we fund science and support mRNA technology!
Great direction of research. While regular transformers achieve impressive results, I always thought that taking "decisions" (sampling a token) in natural language instead of laten space was limiting their cognition abilities. This seems to be the case.
TL;DR: I made a Transformer that conditions its generation on latent variables.
To do so an encoder Transformer only needs a source of randomness during generation, but then it needs an encoder for training, as a [conditional] VAE.
1/5
1
5
228
Etienne Bernard retweeted
Finally had a chance to listen through this pod with Sutton, which was interesting and amusing.
As background, Sutton's "The Bitter Lesson" has become a bit of biblical text in frontier LLM circles. Researchers routinely talk about and ask whether this or that approach or idea is sufficiently "bitter lesson pilled" (meaning arranged so that it benefits from added computation for free) as a proxy for whether it's going to work or worth even pursuing. The underlying assumption being that LLMs are of course highly "bitter lesson pilled" indeed, just look at LLM scaling laws where if you put compute on the x-axis, number go up and to the right. So it's amusing to see that Sutton, the author of the post, is not so sure that LLMs are "bitter lesson pilled" at all. They are trained on giant datasets of fundamentally human data, which is both 1) human generated and 2) finite. What do you do when you run out? How do you prevent a human bias? So there you have it, bitter lesson pilled LLM researchers taken down by the author of the bitter lesson - rough!
In some sense, Dwarkesh (who represents the LLM researchers viewpoint in the pod) and Sutton are slightly speaking past each other because Sutton has a very different architecture in mind and LLMs break a lot of its principles. He calls himself a "classicist" and evokes the original concept of Alan Turing of building a "child machine" - a system capable of learning through experience by dynamically interacting with the world. There's no giant pretraining stage of imitating internet webpages. There's also no supervised finetuning, which he points out is absent in the animal kingdom (it's a subtle point but Sutton is right in the strong sense: animals may of course observe demonstrations, but their actions are not directly forced/"teleoperated" by other animals). Another important note he makes is that even if you just treat pretraining as an initialization of a prior before you finetune with reinforcement learning, Sutton sees the approach as tainted with human bias and fundamentally off course, a bit like when AlphaZero (which has never seen human games of Go) beats AlphaGo (which initializes from them). In Sutton's world view, all there is is an interaction with a world via reinforcement learning, where the reward functions are partially environment specific, but also intrinsically motivated, e.g. "fun", "curiosity", and related to the quality of the prediction in your world model. And the agent is always learning at test time by default, it's not trained once and then deployed thereafter. Overall, Sutton is a lot more interested in what we have common with the animal kingdom instead of what differentiates us. "If we understood a squirrel, we'd be almost done".
As for my take...
First, I should say that I think Sutton was a great guest for the pod and I like that the AI field maintains entropy of thought and that not everyone is exploiting the next local iteration LLMs. AI has gone through too many discrete transitions of the dominant approach to lose that. And I also think that his criticism of LLMs as not bitter lesson pilled is not inadequate. Frontier LLMs are now highly complex artifacts with a lot of humanness involved at all the stages - the foundation (the pretraining data) is all human text, the finetuning data is human and curated, the reinforcement learning environment mixture is tuned by human engineers. We do not in fact have an actual, single, clean, actually bitter lesson pilled, "turn the crank" algorithm that you could unleash upon the world and see it learn automatically from experience alone.
Does such an algorithm even exist? Finding it would of course be a huge AI breakthrough. Two "example proofs" are commonly offered to argue that such a thing is possible. The first example is the success of AlphaZero learning to play Go completely from scratch with no human supervision whatsoever. But the game of Go is clearly such a simple, closed, environment that it's difficult to see the analogous formulation in the messiness of reality. I love Go, but algorithmically and categorically, it is essentially a harder version of tic tac toe. The second example is that of animals, like squirrels. And here, personally, I am also quite hesitant whether it's appropriate because animals arise by a very different computational process and via different constraints than what we have practically available to us in the industry. Animal brains are nowhere near the blank slate they appear to be at birth. First, a lot of what is commonly attributed to "learning" is imo a lot more "maturation". And second, even that which clearly is "learning" and not maturation is a lot more "finetuning" on top of something clearly powerful and preexisting. Example. A baby zebra is born and within a few dozen minutes it can run around the savannah and follow its mother. This is a highly complex sensory-motor task and there is no way in my mind that this is achieved from scratch, tabula rasa. The brains of animals and the billions of parameters within have a powerful initialization encoded in the ATCGs of their DNA, trained via the "outer loop" optimization in the course of evolution. If the baby zebra spasmed its muscles around at random as a reinforcement learning policy would have you do at initialization, it wouldn't get very far at all. Similarly, our AIs now also have neural networks with billions of parameters. These parameters need their own rich, high information density supervision signal. We are not going to re-run evolution. But we do have mountains of internet documents. Yes it is basically supervised learning that is ~absent in the animal kingdom. But it is a way to practically gather enough soft constraints over billions of parameters, to try to get to a point where you're not starting from scratch. TLDR: Pretraining is our crappy evolution. It is one candidate solution to the cold start problem, to be followed later by finetuning on tasks that look more correct, e.g. within the reinforcement learning framework, as state of the art frontier LLM labs now do pervasively.
I still think it is worth to be inspired by animals. I think there are multiple powerful ideas that LLM agents are algorithmically missing that can still be adapted from animal intelligence. And I still think the bitter lesson is correct, but I see it more as something platonic to pursue, not necessarily to reach, in our real world and practically speaking. And I say both of these with double digit percent uncertainty and cheer the work of those who disagree, especially those a lot more ambitious bitter lesson wise.
So that brings us to where we are. Stated plainly, today's frontier LLM research is not about building animals. It is about summoning ghosts. You can think of ghosts as a fundamentally different kind of point in the space of possible intelligences. They are muddled by humanity. Thoroughly engineered by it. They are these imperfect replicas, a kind of statistical distillation of humanity's documents with some sprinkle on top. They are not platonically bitter lesson pilled, but they are perhaps "practically" bitter lesson pilled, at least compared to a lot of what came before. It seems possibly to me that over time, we can further finetune our ghosts more and more in the direction of animals; That it's not so much a fundamental incompatibility but a matter of initialization in the intelligence space. But it's also quite possible that they diverge even further and end up permanently different, un-animal-like, but still incredibly helpful and properly world-altering. It's possible that ghosts:animals :: planes:birds.
Anyway, in summary, overall and actionably, I think this pod is solid "real talk" from Sutton to the frontier LLM researchers, who might be gear shifted a little too much in the exploit mode. Probably we are still not sufficiently bitter lesson pilled and there is a very good chance of more powerful ideas and paradigms, other than exhaustive benchbuilding and benchmaxxing. And animals might be a good source of inspiration. Intrinsic motivation, fun, curiosity, empowerment, multi-agent self-play, culture. Use your imagination.
.@RichardSSutton, father of reinforcement learning, doesn’t think LLMs are bitter-lesson-pilled.
My steel man of Richard’s position: we need some new architecture to enable continual (on-the-job) learning.
And if we have continual learning, we don't need a special training phase - the agent just learns on-the-fly - like all humans, and indeed, like all animals.
This new paradigm will render our current approach with LLMs obsolete.
I did my best to represent the view that LLMs will function as the foundation on which this experiential learning can happen. Some sparks flew.
0:00:00 – Are LLMs a dead-end?
0:13:51 – Do humans do imitation learning?
0:23:57 – The Era of Experience
0:34:25 – Current architectures generalize poorly out of distribution
0:42:17 – Surprises in the AI field
0:47:28 – Will The Bitter Lesson still apply after AGI?
0:54:35 – Succession to AI
Etienne Bernard retweeted
The only bitter lesson is that LLMs have succeeded beyond any expert expectations.
Underpinning LLMs is the idea of scaling, which is too often misunderstood as more parameters. Scaling is about using massive compute effectively to maximise the throughput of data ingestion into the learning process to obtain more capable models.
We are still far from hitting the limits in this. We are still compute hungry because there is a ton more we could achieve if only we had more compute, from experimental ablations to data acquisition and curation.
Scaling is largely about data and evals. The models are now trained on almost all the web and equally large (but growing) self generated synthetic data. sifting through such vasts quantities of data (the whole of the human creation) requires formidable engineering and intelligent ideas. This is what differentiates most models.
AI is finally in the hands of billions of users, and with it come billions of tasks - every reasonable user need. This scaling in tasks and evaluations is many orders of magnitude larger than pre-LLMs.
Having the right architecture matters, but we know several alternatives could all work well, eg replacing attention in Transformers for RNNs and interleaving such layers with local layers. What matters is fine ablations to maximise hardware usage. This is the realm of sophisticated high-precision engineering. It encompasses semiconductor design, datacenter design, distributed systems, MFU, etc. There is fascinating work on flow matching, JEPA, sparser MoEs, etc, that is all consistent with scaling.
I’m terrible at predictions, but in this we have stayed the course. There’s been pleasant surprises like the effectiveness of reasoning, which while allowing for less parameters, still demands even more compute.
Sparser multimodal MoEs also will allow for better continual learning. This is an old idea, eg arxiv.org/pdf/1108.3298, which is finally being done at scale.
Successful scaling is mostly about organising people into effective teams for research, development and production. They have to be teams of happy and ambitious people who put the team first. Yes, tech VCs and CEOs: work life balance matters to achieve prologued success, something I think @demishassabis did really well at @GoogleDeepMind and which I promote at @MicrosoftAI.
Bitter lesson: it really is all about scaling and hard work by thousands of amazing people. Hardly bitter, but hopeful and inspiring.
Etienne Bernard retweeted
🚀 We're thrilled to unveil Qwen3-VL — the most powerful vision-language model in the Qwen series yet!
🔥 The flagship model Qwen3-VL-235B-A22B is now open-sourced and available in both Instruct and Thinking versions:
✅ Instruct outperforms Gemini 2.5 Pro on key vision benchmarks
✅ Thinking achieves state-of-the-art (SOTA) performance on multimodal reasoning tasks
✨ Key breakthroughs:
🖥️ Visual Agent: Operates GUIs on PC/phone — understands buttons, calls tools, and completes real-world tasks (SOTA on OS World)
💻 Visual Coding: Transforms screenshots into code (HTML/CSS/JS, Draw.io) — true "what you see is what you get" development
📚 256K+ context (scalable to 1M) → supports 2-hour videos and multi-page long PDFs
🌍 32-language OCR with enhanced robustness for blurry, tilted, or rare characters
📐 Advanced spatial reasoning: 2D → relative coordinates, 3D grounding, occlusion handling, and perspective understanding
🧠 Thinking Mode: Leading performance in STEM/Math — enables deep causal reasoning
🔤 Text capabilities rival top-tier LLMs — a solid language foundation powering its multimodal excellence
From "seeing" to "understanding", from "recognizing" to "reasoning & acting"
Qwen Chat: chat.qwen.ai/?models=qwen3-v…
API: alibabacloud.com/help/en/mod…
Blog:qwen.ai/blog?id=99f0335c4ad9…
ModelScope: modelscope.cn/collections/Qw…
HuggingFace: huggingface.co/collections/Q…
NuMarkdown (reasoning OCR) trending #1 in Image-to-Text models on @huggingface 😀
We created a Space for everyone to try it: huggingface.co/spaces/numind…
Looking for failure modes, so let us know if you manage to make it fail spectacularly!
We are releasing NuMarkdown-8B-Thinking, an open-source (MIT License) reasoning OCR model 🧠✨📄
NuMarkdown-8B-Thinking is apparently the first (!) reasoning VLM specialized in converting PDFs/Scans/Spreadsheets into Markdown files (typically used for RAG applications).
It generates thought tokens to figure out the layout of the document before generating the Markdown file. It is particularly good at understanding documents with weird layouts and complex tables. The number of thinking tokens can vary from 20% to 500% of the final answer, depending on task difficulty.
This model is a fine-tune of @Alibaba_Qwen 2.5-VL-7B on synthetic Doc → Reasoning → Markdown examples, followed by an RL phase (GRPO) with a layout-centric reward.
We find that NuMarkdown is outperforming generalist non-reasoning models like GPT-4o, specialized OCR models like OCRFlux, and is competitive against large reasoning closed-source models like Gemini 2.5!
Congrats to Alexandre Constantin for this tour de force
extra links:
- GitHub repository: github.com/numindai/NuMarkdo…
- Discord server: discord.gg/3tsEtJNCDe
1
16
2,589
And you can try the model here! huggingface.co/spaces/numind…
1
293
Etienne Bernard retweeted
Another VLM-based OCR model!
NuMarkdown-8B-Thinking is special, it reasons through documents before converting to markdown
See it analyse this 1878 medical report step-by-step.
Try with @huggingface Jobs + @astral uv - no GPU needed!
Etienne Bernard retweeted
Really excited about ALHF, new work from our research team that lets users give natural language feedback to agents and optimizes them for it. It sort of upends the traditional supervision paradigm where you get a scalar reward, and it makes AI more customizable for non-experts.
Etienne Bernard retweeted
People realize these are vaccines for CANCER, right?
Like, cancer was going to become a treatable disease because of this funding. Now your cancer will be a death sentence, because of RFK and Trump.
Etienne Bernard retweeted
Insanity.
No other technology promises to provide as much protection against an array of diseases.
Now this anti-vaxx weirdo has shut them all down.
Never has America been more vulnerable.


























