@HiredGunDevDen

This is the ultimate hangout for Homelab enthusiasts, AI tinkerers, and Vibe Coders. Think of this as your digital basement: a place to swap stories.

thehiredgun.net
Joined December 2021
  • Tweets2.8K
  • Following532
  • Followers154
123Media
Somebody told me to get more power, so I did. Turns out a terabyte still doesn't fit in 128GB. But my $3,250 rig runs a 125B MoE at 77 tok/s flat across 2K-128K, 25% ahead of the vendor at depth. Arithmetic beats more power. What does your rig do at 128K? @gospaceport @coldniko xitter.cf/HiredGunDevDen/status/…
Josh Meyer
@HiredGunDevDenOct 18

How Far Can a Home Lab AI Rig Get Toward Frontier Inference?

Article · @HiredGunDevDen
1
3
58
Can't beat that price.... :-\
0xSero
@0xSero9h
Don’t let them gaslight you, Chinese models are absolutely eating up frontier demand. Deepseek is #1 and #2 lol
1
5
The local AI community thanks you for your service!
Digital Spaceport
@gospaceportOct 26
I validated it, NVIDIA did leave a free 60% on the table for Ampere/Ada GPUs 😎 Quad 3090s + Qwen 3.8 27B BF16 in vLLM: Unpatched: 1114–1259 tok/s prefill Patched: 1769–2020 tok/s prefill FREE ~60%+ on prompt processing from 2K all the way to 128K. About 15 mins shaved off every hour for long running agentic tasks in Qwen 3.8 27B and other full precision dense models. Decode, the same. Power utilization, the same. 🔥 risk, the same. Quantized models + Imgen/Vidgen, testing soon to get hard numbers Windows unlocked as well. 50X0, probably 🔜 Kernel module, no driver patching or vbios crap. Easiest free win of the year for Local Ai 🎉 Notes + bench script: digitalspaceport.com/gpu-unl…
3
Anyone else got their AI/SI system structured like a company? @gospaceport @SpaceXSI @NousResearch @coldniko @NetworkChuck @QwenDevs
2
1
12
The key to useful AI is brining it into context for yourself. Structure like this allows you to use familiar tools, like email and job queues.
4
Very Detailed!
Simply Complex Systems
@simcomsystems11h
Replying to @HiredGunDevDen @gospaceport @SpaceXSI @NousResearch @coldniko @NetworkChuck @QwenDevs
We use a traditional organizational structure.
1
8
I think my AI rig is ~3K after all the upgrades I can make for my little B550 (128G DDR4, Risen 9 5900XT, 3 5060TI cards, 500G NVMe)... I spend less than ~100 a year in frontier tokens and I am pushing ~ half a million lines of code shipped. themightyjoshua.com I ran this on 32Gb DDR4 and a Risen 5 5500 before the upgrades last month.
Digital Spaceport
@gospaceportOct 27
These ppl really think its $10k for a rig that can do serious work? They must not have met Strata or Maya yet 🤔
6
Unless of course @coldniko has a list of features he wants built by the community.
Josh Meyer
@HiredGunDevDenOct 25
Just a thought... Strata is forkable.. might be worth doing your own optimizations to contribute.
1
1
1,564
Just a thought... Strata is forkable.. might be worth doing your own optimizations to contribute.
udofu@udofuchiOct 25
qwen3.8 flash nextの最適化は辞めちゃったのかな？
2
1
1
1,806
Intellect just became the currency of the GPU market. Horizontal scales will keep this commodity higher than oil prices the more dependent we become.
Elon Musk
@elonmuskOct 25
The way to think about this is: How many super smart and dedicated people do you want to hire? And they keep getting better.
10
This only makes local AI movements stronger.. It's just a matter of time before we see better distillations being done.
Digital Spaceport
@gospaceportOct 24
Every Anthropic account is an affront to freedom. Not blaming you if you have one, but it is true.
7
Awesome
Niko Veit
@coldnikoOct 24
Deepseek-V4.1-Flash now runs on 64GB of RAM with Strata. Using GSQ-RCO quantizations by @mhelcig
3
85
Because brain hurts.... Farming is less stress.
Mehak
@itsmeMehak_Oct 22
Be honest, Why is every software engineer’s backup plan always farming?
8
Engineers. How do you use Agents? I'm not asking for how they work. I'm asking your method for building and choosing the brain they run on.
1
1
1
40
This is what drove me to local AI. Never paid Claude but grok does the same thing. I've learned if you use DeepSeek v41 and grok as sub agents and local as your main life is much much better. Local holds your project and rag proper like and if you need extra umph DeepSeek for DeepSeek and grok is more than enough at a tenth the expense.
Digital Spaceport
@gospaceportOct 22
The reset thing, it get's old? I have no account at oai or anth so I just watch y'all do this dance and it seems like it would get old? Seems like this is highly arbitrary how it all goes down.
14
REALLY good advice.
Josh Meyer
@HiredGunDevDenSep 16
"1. Make your requirements less dumb. Everyone is wrong no matter who you are some of the time. 2. Try very hard to delete the part or process. If you aren't occasionally adding things back in you aren't deleting enough. Whatever requirement or constraint you have you must have a name attached. 3. Simplify/Optimize: Most common problem in engineering is to optimize something that shouldn't exist. 4. Accelerate Cycle time: your moving too slowly, go faster, but don't go faster until you fix 1-3. 5. Automate" -- @elonmusk
7
Had to issue a correction to my last post! Realized I botched my tests which threw off some of my results and thus, the major conclusion I reached. @coldniko @gospaceport
Josh Meyer
@HiredGunDevDenOct 20

How far can a homelab toward frontier inference, and what actually blocks the rest? (CORRECTION!)

Article · @HiredGunDevDen
1
31

How far can a homelab toward frontier inference, and what actually blocks the rest? (CORRECTION!)

Article · @HiredGunDevDen
1
145
If you have time, come check out my labor of love. This chess site was built by myself, using my Local AI. @gospaceport @coldniko @GMHikaru
Josh Meyer
@HiredGunDevDenOct 20
The Hired Gun is building out a chess site. Come check it out themightyjoshua.com.
1
1
74
The Hired Gun is building out a chess site. Come check it out themightyjoshua.com.
1
88
Load more