
PhD-Student Software Engineer Researching into Software Quality of LLM-based Agents Generated Code Specification-Driven Development Researcher at @i3labuex
Joined July 2016
- Tweets2.9K
- Following483
- Followers211
- Likes5.3K
I've just updated slmbench with browser use! Check it pi-browser branch! Based on
@PiChangelog and ChromeDevTools MCP :) The example uses LFM3-1.7B from @liquidai ! Small Models are here to stay!
Hey open-source & local LLM community 👋
I built slmbench🤖: benchmark the small language models you run locally, on public benchmarks, from a TUI.
Works with llama.cpp, Ollama, LM Studio, vLLM.
Open source 👇
github.com/emi-dm/slm-bench
🧵
Ha sido un side project en el que he ido currando por las noches, muy entretenido y del que he aprendido mucho sobre SLMs! Now, Smol Course @huggingface
Hey open-source & local LLM community 👋
I built slmbench🤖: benchmark the small language models you run locally, on public benchmarks, from a TUI.
Works with llama.cpp, Ollama, LM Studio, vLLM.
Open source 👇
github.com/emi-dm/slm-bench
🧵
Issues and PRs welcome. If you find it useful, a ⭐ and a RT go a long way 🙏
github.com/emi-dm/slm-bench
3
It’s built on lighteval from @huggingface 🤗
lighteval runs the tasks and metrics; slmbench adds the TUI, the modes and a leaderboard that updates itself after every run.
Thanks to the HF ecosystem for making this possible 🙌
15
Emi retweeted
This whole thing sucks, and I wish none of it had happened. It’s a really sad day!
But I want to make one thing clear: we’ll keep fighting for open-source AI, harder than ever. This is too important to slow down now. Let’s go!
Emi retweeted
A pretty important new primitive just landed in 🤗 Transformers: **decision models**.
I think this could matter much more than adding support for any single model.
Today, different decision models all approach "making a decision" differently:
- @typesafeai’s Jev, the OG: typed decisions + calibrated probabilities, single forward
- @AutoTrustAI’s JEV builds open System One models around a learned decision block, w/ multimodal.
- @jaredpalmer’s KEV uses a pointer head over the options, sharing the computation of the state across questions.
- @Cloudflare’s CLEF uses a joint schema head that routes evidence between the state, questions and options, scoring the whole schema together.
- @perplexity_ai’s Decider uses a separate 255-way readout on top of Qwen, with non-causal attention in its full-attention layers.
- @liquidai’s d1 is exploring small, multimodal decision models that can run all the way down to the edge.
- And @XunzhuoLiu's Semantic Router team is pushing the serving side hard with System One + the D3 family: one typed interface for choice, score and yes/no decisions, with models from sub-1B to 27B.
That experimentation is good, and needed! But without a common interface, every training framework, inference engine and evaluation tool has to understand every implementation separately.
So Transformers is introducing a shared decision-model layer:
`DecisionConfig`
`pipeline("decision")`
and, most importantly:
`model.decide(...)`
The idea is **not** to make every decision model work the same internally, but for them to speak the same language to the ecosystem around them.
This means that, without running custom or remote code:
- inference engines can serve them through one interface
- training tools can build against common primitives.
- the Hub can package them consistently.
- evaluations can compare them without model-specific glue everywhere.
This is the part of transformers I'm the most excited about: to grow alongside the ecosystem and the many ideas; and to make all of them more usable by everyone.
github.com/huggingface/trans…
Emi retweeted
Everything that you need to know about decision models, in one place.
> what are decision models
> transformers integration
> trl support
> llama cpp support
huggingface.co/blog/decision
Emi retweeted
I gave Fable 5.2 a PDF of one of Leonardo da Vinci’s 500-year-old notebooks and it made his inventions real for me to interact with.
Such an exciting time for education. Anything from history that’s been stuck on a page can now be something you explore.















