Sa7f (@sa7f_f) profile picture

Sa7f

@sa7f_f
Joined November 2024
  • Tweets23
  • Following56
  • Followers3
3Media
Photo by Sa7f (@sa7f_f)Photo by Sa7f (@sa7f_f)Photo by Sa7f (@sa7f_f)
Run Qwen 3.8 27b at 240+ tok/s on a 12 GB RTX 5070 (Full Guide)

Run Qwen 3.8 27b at 240+ tok/s on a 12 GB RTX 5070 (Full Guide)

Article · @sa7f_f
1
1
4
155
Weights: huggingface.co/WaveCut/Terna… Speed test script: gist.github.com/Sypher760-gi… Engine: NInfer by @WaveCut
WaveCut/Ternary-Bonsai-2-27B-NInfer-v3 · Hugging Face

WaveCut/Ternary-Bonsai-2-27B-NInfer-v3 · Hugging Face

Image-Text-to-Text model. Download, explore files and run it on Hugging Face.

huggingface.co
1
1
30
242 tok/s qwen 3.8 27b on a 12 gb RTX 5070
2
2
257
more replies
setup: bonsai 27b by @PrismML (ternary quant of qwen 3.8 27b) on ninfer by @WaveCut 64k ctx, ~10.7 of 12 gb used 7800X3D, 64 gb ram, win 11 + wsl2 github.com/iamwavecut/ninfer… runs: localmaxxing.com/en/models/p…
GitHub - iamwavecut/ninfer-all: One line of NInfer for the Blackwell, Ada Lovelace, Ampere, Turing...

GitHub - iamwavecut/ninfer-all: One line of NInfer for the Blackwell, Ada Lovelace, Ampere, Turing...

One line of NInfer for the Blackwell, Ada Lovelace, Ampere, Turing and Volta NVIDIA GPUs, consolidated from the forks that carry it: GGUF block formats, ternary Bonsai 2, device route profiles, MTP...

github.com
2
1
2
88
the app in the video is local-arena, which i just open sourced github.com/Sypher760-gif/loc…
GitHub - Sypher760-gif/local-arena: Chat with, compare and test local language models on your own...

GitHub - Sypher760-gif/local-arena: Chat with, compare and test local language models on your own...

Chat with, compare and test local language models on your own PC. Side-by-side Arena, 704 real benchmark questions, live speed and memory readings. - Sypher760-gif/local-arena

github.com
1
36
Load more