System64 (@System64fumo) header
System64 (@System64fumo) profile picture

System64

@System64fumo

🇺🇸🇮🇱 Icy fairy goofing around with embedded systems for fun!

 Israel
github.com/System64fumo
Joined May 2016
  • Tweets575
  • Following117
  • Followers103
74Media
Photo by System64 (@System64fumo)Photo by System64 (@System64fumo)Photo by System64 (@System64fumo)Photo by System64 (@System64fumo)Photo by System64 (@System64fumo)Photo by System64 (@System64fumo)Photo by System64 (@System64fumo)Photo by System64 (@System64fumo)Photo by System64 (@System64fumo)Photo by System64 (@System64fumo)
I'm proud to announce that github.com/System64fumo/kapp… has hit a major usability milestone. If you're interested in #LocalAI now's the time to start min/maxing your hardware.
GitHub - System64fumo/kappai: High performance inference engine

GitHub - System64fumo/kappai: High performance inference engine

High performance inference engine. Contribute to System64fumo/kappai development by creating an account on GitHub.

github.com
2
8
102
github.com/System64fumo/kapp… now has cooperative prefill and smart CPU cluster scheduling! Here's my Orange Pi 5 plus (rk3588) prefilling using CPU + GPU + NPU on a really long prompt. @orangepixunlong @IloveRockchip #LocalAI #ARM
Image from System64 (@System64fumo)
1
1
41
more replies
Also excuse the poor performance, Vulkan and Rocket are still work in progress. Vulkan runs at half the speed of CPU, So does rocket. Rocket still requires a bit of CPU to run so it sort of eats into CPU matmul performance hence why the numbers aren't additive.
2
7
Probably worth noting that "smart CPU cluster scheduling" here means prefill runs on all cores while decode runs on the fast cores only. This helps with heat/power consumption/efficiency since there's 0 perf gain to decode if you run it on all 8 cores, In fact it even degrades it
3
I feel like more people should just have fun in life.
Image from System64 (@System64fumo)
8
68
1
460
2,478
"Teamwork makes the dream work" - Master node Not very creative i know, Anyway clustering support in kappai is fun.
31
Another milestone! NPU support for rk3588 has landed in kappai. Despite it being very unfinished/unpolished I'd say this is running pretty well on my orangepi 5 plus. #rk3588 @orangepixunlong
Image from System64 (@System64fumo)
1
50
NPUs are weird man.
27
Na, I'm not done here yet, There's still plenty more where that came from.
1
31
Not sure what people care about more, Reducing memory usage or making AI go faster in general.
26
Call me crazy, Call me delusional, But i know for a god damn fact YOUR computer is faster than you were lead to believe.
Image from System64 (@System64fumo)
1
36
Did some work on kappai today and i think i did well? Still very W.I.P, Chat did i cook? 👀
Image from System64 (@System64fumo)
1
41
I'm not pulling off any special tricks here or sabotaging llama.cpp by having some load in the background, Both engines are running with the same parameters same conditions, etc.. Unfortunately (Or rather fortunately) this isn't the end, There's still plenty of headroom left.
26
Compute is expensive.. Let's make it cheaper, And local!
Image from System64 (@System64fumo)
2
17
158
I cooked hard, Here are some things that landed in kappai recently: GLM 5.3 Flash support OpenGL ES 3.1 backend Several new IQ quants TUI setup/run screen & many more There's also a new matrix and discord for this project, Come join us! github.com/System64fumo/kapp…
GitHub - System64fumo/kappai: High performance inference engine

GitHub - System64fumo/kappai: High performance inference engine

High performance inference engine. Contribute to System64fumo/kappai development by creating an account on GitHub.

github.com
1
43
I don't actually use discord btw so if you want to contact me do so on matrix.
20
Holidays are over, Time to cook again. Where are all my cracked C devs at? 👀
30
Ok so i didn’t expect to get this much attention, But damn those are a lot of eyes. 👀 I’m out for holidays right now so sorry for the lack of updates. Thank you Bee fumo! 🐝
Image from System64 (@System64fumo)
2
70
I have some pretty telling evidence that Union Alpha is some sort of GLM model. @Zai_org Can we please "pace the frontier"? /s And while you're at it could you kindly fix the glaring security hole? I wanted to work not dive down this rabbit hole, k? thx.
1
120
Load more