Emi (@SrEdm00) profile picture

PhD-Student Software Engineer Researching into Software Quality of LLM-based Agents Generated Code Specification-Driven Development Researcher at @i3labuex

Joined July 2016
Photo by Emi (@SrEdm00)Photo by Emi (@SrEdm00)Photo by Emi (@SrEdm00)Photo by Emi (@SrEdm00)Photo by Emi (@SrEdm00)Photo by Emi (@SrEdm00)Photo by Emi (@SrEdm00)Photo by Emi (@SrEdm00)Photo by Emi (@SrEdm00)Photo by Emi (@SrEdm00)
I've just updated slmbench with browser use! Check it pi-browser branch! Based on @PiChangelog and ChromeDevTools MCP :) The example uses LFM3-1.7B from @liquidai ! Small Models are here to stay!
Hey open-source & local LLM community 👋 I built slmbench🤖: benchmark the small language models you run locally, on public benchmarks, from a TUI. Works with llama.cpp, Ollama, LM Studio, vLLM. Open source 👇 github.com/emi-dm/slm-bench 🧵
2
1
51
More cases!
7
Hey open-source & local LLM community 👋 I built slmbench🤖: benchmark the small language models you run locally, on public benchmarks, from a TUI. Works with llama.cpp, Ollama, LM Studio, vLLM. Open source 👇 github.com/emi-dm/slm-bench 🧵
11
5
1
9
566
Btw, @liquidai LFM3-1.7B is a really good model for tool calling 👀 Congrats!
1
2
1
47
2,671
Also leaderboard compared to Qwen 3.5 0.8B
2
157
More views!
17
Ha sido un side project en el que he ido currando por las noches, muy entretenido y del que he aprendido mucho sobre SLMs! Now, Smol Course @huggingface
Hey open-source & local LLM community 👋 I built slmbench🤖: benchmark the small language models you run locally, on public benchmarks, from a TUI. Works with llama.cpp, Ollama, LM Studio, vLLM. Open source 👇 github.com/emi-dm/slm-bench 🧵
1
2
99
More: ✅ pass@1 and pass@5 ✅ Saved answers: rescore or resume without calling the model again ✅ Failure explorer: see why each problem failed ✅ Live view with tokens/s and ETA ✅ Leaderboard per quantization (Q4_K_M ≠ Q8_0)
1
Model-written code is untrusted code. So it only ever runs inside a @Docker Sandboxes microVM: no network, no access to your files. 🐳 Compilers for Python, Node, C++, Java, Go and Rust live inside the sandbox, never on your machine.
7
It’s built on lighteval from @huggingface 🤗 lighteval runs the tasks and metrics; slmbench adds the TUI, the modes and a leaderboard that updates itself after every run. Thanks to the HF ecosystem for making this possible 🙌
15
Day in the life of Jensen Huang
246
2,001
199
23,538
2,808,213
This whole thing sucks, and I wish none of it had happened. It’s a really sad day! But I want to make one thing clear: we’ll keep fighting for open-source AI, harder than ever. This is too important to slow down now. Let’s go!
448
941
161
10,549
606,741
A pretty important new primitive just landed in 🤗 Transformers: **decision models**. I think this could matter much more than adding support for any single model. Today, different decision models all approach "making a decision" differently: - @typesafeai’s Jev, the OG: typed decisions + calibrated probabilities, single forward - @AutoTrustAI’s JEV builds open System One models around a learned decision block, w/ multimodal. - @jaredpalmer’s KEV uses a pointer head over the options, sharing the computation of the state across questions. - @Cloudflare’s CLEF uses a joint schema head that routes evidence between the state, questions and options, scoring the whole schema together. - @perplexity_ai’s Decider uses a separate 255-way readout on top of Qwen, with non-causal attention in its full-attention layers. - @liquidai’s d1 is exploring small, multimodal decision models that can run all the way down to the edge. - And @XunzhuoLiu's Semantic Router team is pushing the serving side hard with System One + the D3 family: one typed interface for choice, score and yes/no decisions, with models from sub-1B to 27B. That experimentation is good, and needed! But without a common interface, every training framework, inference engine and evaluation tool has to understand every implementation separately. So Transformers is introducing a shared decision-model layer: `DecisionConfig` `pipeline("decision")` and, most importantly: `model.decide(...)` The idea is **not** to make every decision model work the same internally, but for them to speak the same language to the ecosystem around them. This means that, without running custom or remote code: - inference engines can serve them through one interface - training tools can build against common primitives. - the Hub can package them consistently. - evaluations can compare them without model-specific glue everywhere. This is the part of transformers I'm the most excited about: to grow alongside the ecosystem and the many ideas; and to make all of them more usable by everyone. github.com/huggingface/trans…
15
28
7
153
15,551
Everything that you need to know about decision models, in one place. > what are decision models > transformers integration > trl support > llama cpp support huggingface.co/blog/decision
8
25
4
182
17,704
Descubrí a Opus5.5 hardcodeando un test. Ya tiene nivel de Senior.
5
15
597
16,133
I gave Fable 5.2 a PDF of one of Leonardo da Vinci’s 500-year-old notebooks and it made his inventions real for me to interact with. Such an exciting time for education. Anything from history that’s been stuck on a page can now be something you explore.
146
291
70
2,510
142,914
Emi retweeted
Me using Fable 5.2 to rename a variable
42
296
24
5,188
225,507
Taking advantage of the bad weather! My head just exploded 🤯 I promise to publish the model
36