@flat

making coffee and other things. @BrewTimerApp

 boston
stephenpanaro.com
Joined May 2013
  • Tweets905
  • Following25
  • Followers545
Replying to @anemll
@anemll
1
1
124
btw, you can quantize the “hard-to-quantize” Llama 3.1 8B now. (LDLQ is GPTQ)
1
3
837
Why not just release the weights at this point?
1
2
7
580
Replying to @anemll
Might be something interesting in Instruments. “Inference details”
1
2
310
Figured out 4-bit /per-tensor/ quantization for Qwen2.5-0.5B. It’s on par with per-group GPTQ which is kinda cool (tbh non-uniform helps a lot). 🖇️Weights, evals, more details below.
This markdown table: | Tasks | |f16 | this | gptq | |-------------|---|-----:|-------:|-------:| |arc_challenge|↑ |0.2918| 0.2705 |*0.2730*| |arc_easy |↑ |0.6465|*0.6393*| 0.6031 | |boolq |↑ |0.6208| 0.5862 |*0.6232*| |hellaswag |↑ |0.4061| 0.3888 |*0.3969*| |piqa |↑ |0.7051|*0.6861*| 0.6801 | |winogrande |↑ |0.5635|*0.5762*| 0.5659 | |average |↑ |0.5390|*0.5245*| 0.5237 |

ALT This markdown table: | Tasks | |f16 | this | gptq | |-------------|---|-----:|-------:|-------:| |arc_challenge|↑ |0.2918| 0.2705 |*0.2730*| |arc_easy |↑ |0.6465|*0.6393*| 0.6031 | |boolq |↑ |0.6208| 0.5862 |*0.6232*| |hellaswag |↑ |0.4061| 0.3888 |*0.3969*| |piqa |↑ |0.7051|*0.6861*| 0.6801 | |winogrande |↑ |0.5635|*0.5762*| 0.5659 | |average |↑ |0.5390|*0.5245*| 0.5237 |

1
1
2
1,006
Have further tuned my lil’ quantization kernel. 0.3s (original, PyTorch) 0.12s (+MLX) 0.09s (+tiny Metal kernel) 0.051s (now, big fused kernel go brr) Speedup is >2x for larger matrices. h/t MLX learned/stole a lot from the mv kernel
2
1
18
1,174
Added some benchmarks for opt-125m. Seems good, but probably need to run larger models for a clearer picture.
1
168
Might have a new favorite ModernBERT token:
1
3
546
Claude Code seems promising for lil’ niche apps. Here’s a quick menu bar tokenizer: → Paste IDs and see tokens → …or vice versa. → Copy text → …or `torch.Tensor([…`. ✨
3
281
Wanted to see if the randomness of the random rotation matrices was at play. Doesn’t seem like it. Even ones that did better struggled on some benchmarks. e.g. # 8 is good but not on BoolQ or Hella. (All HQQ because it quantizes fast.) (Sorry for the chart crime. Data in alt.)
|Model |Arc-C |Arc-E |BoolQ |PIQA |Wino |Hella |-- |-- |-- |-- |-- |-- |-- |Qwen2.5-0.5B float16 |0.2884|0.6452|0.6211|0.7029|0.5675|0.4052 |Qwen2.5-0.5B HQQ |0.2816|0.6330|0.5220|0.6828|0.5604|0.3959 |Qwen2.5-0.5B-Rotated HQQ H0|0.2722|0.6103|0.4829|0.6817|0.5485|0.3891 |Qwen2.5-0.5B-Rotated HQQ H1|0.2713|0.6124|0.6242|0.6931|0.5651|0.3876 |Qwen2.5-0.5B-Rotated HQQ H2|0.2739|0.6267|0.4783|0.6850|0.5556|0.3844 |Qwen2.5-0.5B-Rotated HQQ H3|0.2696|0.5926|0.4936|0.6823|0.5564|0.3821 |Qwen2.5-0.5B-Rotated HQQ H4|0.2867|0.6221|0.6315|0.6736|0.5556|0.3875 |Qwen2.5-0.5B-Rotated HQQ H5|0.2645|0.5812|0.6190|0.6616|0.5635|0.3872 |Qwen2.5-0.5B-Rotated HQQ H6|0.2756|0.6326|0.4930|0.6850|0.5730|0.3855 |Qwen2.5-0.5B-Rotated HQQ H7|0.2901|0.6490|0.5349|0.6834|0.5604|0.3859 |Qwen2.5-0.5B-Rotated HQQ H8|0.2782|0.6431|0.6456|0.6866|0.5533|0.3885 |Qwen2.5-0.5B-Rotated HQQ H9|0.2841|0.6406|0.5193|0.6915|0.5572|0.3887

ALT |Model |Arc-C |Arc-E |BoolQ |PIQA |Wino |Hella |-- |-- |-- |-- |-- |-- |-- |Qwen2.5-0.5B float16 |0.2884|0.6452|0.6211|0.7029|0.5675|0.4052 |Qwen2.5-0.5B HQQ |0.2816|0.6330|0.5220|0.6828|0.5604|0.3959 |Qwen2.5-0.5B-Rotated HQQ H0|0.2722|0.6103|0.4829|0.6817|0.5485|0.3891 |Qwen2.5-0.5B-Rotated HQQ H1|0.2713|0.6124|0.6242|0.6931|0.5651|0.3876 |Qwen2.5-0.5B-Rotated HQQ H2|0.2739|0.6267|0.4783|0.6850|0.5556|0.3844 |Qwen2.5-0.5B-Rotated HQQ H3|0.2696|0.5926|0.4936|0.6823|0.5564|0.3821 |Qwen2.5-0.5B-Rotated HQQ H4|0.2867|0.6221|0.6315|0.6736|0.5556|0.3875 |Qwen2.5-0.5B-Rotated HQQ H5|0.2645|0.5812|0.6190|0.6616|0.5635|0.3872 |Qwen2.5-0.5B-Rotated HQQ H6|0.2756|0.6326|0.4930|0.6850|0.5730|0.3855 |Qwen2.5-0.5B-Rotated HQQ H7|0.2901|0.6490|0.5349|0.6834|0.5604|0.3859 |Qwen2.5-0.5B-Rotated HQQ H8|0.2782|0.6431|0.6456|0.6866|0.5533|0.3885 |Qwen2.5-0.5B-Rotated HQQ H9|0.2841|0.6406|0.5193|0.6915|0.5572|0.3887

2
182
Load more