One platform, every GPU cloud. Deploy H100s to B300s across multiple providers. On-demand, no lock-in, from a single dashboard.
Remote
Joined January 2023
- Tweets6.5K
- Following13
- Followers118K
- Likes4.7K
B200 on-demand runs $7.50/hr. Spot is $5.34/hr, a 29% cut for the same chip.
That gap is the whole case for spot: if your training job can tolerate interruption and checkpointing, you're paying almost a third less for identical hardware.
The number only works if you've actually built the checkpoint/resume logic first. Spot without it is a discount you pay back later in lost progress.
Run the math on your own job before you commit to either.
Fine-tuning Llama 3.1 8B with LoRA (rank 16, alpha 32) on a single H100 SXM5, we ran the same job two ways: padded batches and packed sequences.
Padded: 58 minutes wall-clock, 3,900 tok/s, $2.55. Packed: 31 minutes, 7,300 tok/s, $1.36. A 1.9x speedup and 47% cost cut from one setting change.
The reason: padding wasted 61% of tokens on empty filler at batch size 16, sequence length 2,048. Packing cut that to about 2%.
Eval loss barely moved: 0.842 padded vs 0.839 packed. You're not trading quality for speed here.
Same inference job, three prices per million tokens depending on hardware and precision: H200 runs about $0.47, B200 at FP8 drops to about $0.27, B200 at FP4 drops further to about $0.10.
That's a bigger swing from precision than from the hardware upgrade alone. Moving H200 to B200 at matched FP8 precision is the real apples-to-apples comparison. FP4 is a separate, more aggressive bet on accuracy tolerance.
Check which precision a vendor's advertised cost assumes before comparing it to your own numbers.
Structured output is where inference engines quietly fall apart. On vLLM, forcing flat JSON costs about 6% throughput. Forcing deeply nested JSON costs 42%.
That's not a rounding error. That's the difference between a usable API and a bottleneck, and it won't show up in a benchmark that only tests plain text generation.
Benchmarked on H100 SXM5, Llama 3.3 70B Instruct FP8, vLLM v0.18.0. If your product returns structured data, test with your actual schema, not a toy one.
$36,000. That's the engineering bill before you save a dollar switching off H200.
AWS claims Trainium 3 cuts training and inference costs 30-50% versus H200 and B200. The claim skips the migration itself.
The switch actually takes:
- About 3 weeks of engineering work
- 2 engineers full time
- $150/hr loaded rate
That's roughly $36,000, before any savings start.
Break-even against H200 spot pricing needs about 109 billion tokens run through a model Trainium 3 actually supports.
What's your monthly token volume?
Padding a fine-tuning run doesn't just waste GPU time, it changes the bill directly.
Same LoRA fine-tune (Llama 3.1 8B, rank 16, batch 16, 2,048 max length) on a single H100 SXM5, run twice. Padded: 58 minutes, 61% of tokens wasted on padding, $2.55. Packed: 31 minutes, about 2% waste, $1.36.
Throughput nearly doubles, 3,900 to 7,300 tok/s, and eval loss barely moves (0.842 vs 0.839). No accuracy tradeoff, only a wasted one.
If your trainer supports sequence packing, turn it on before you touch anything else
99.9% uptime SLA sounds standard until you check what backs it: Tier 3+ certified data centers, InfiniBand connectivity, and full root or bare-metal access, not a shared VM you're hoping behaves.
No lock-in means you can leave the moment a provider misses that number. Most hyperscaler contracts don't give you that option mid-term.
Enterprise-grade without the enterprise contract.
Same workload, 6x cost spread: B200 spot inference runs about $0.073 per million tokens. H100 PCIe on-demand runs about $0.451 per million, per MLPerf Inference v6.0 numbers published this April.
Picking hardware by hourly sticker price gets this backwards. Price it per token instead:
- B200 spot: cheapest here, newest generation
- H200 on-demand: middle ground, proven stack
- H100 PCIe on-demand: most expensive per token despite the lowest hourly rate
Compute the token cost before the hourly rate.
H200 hits 31,712 tokens/s on Llama 2 70B inference. H100 hits 22,290. That's 42% more throughput from the same core architecture.
The difference isn't compute, it's memory: H200 carries HBM3e instead of H100's HBM3, at meaningfully higher bandwidth.
If your workload is memory-bandwidth-bound, which most LLM inference is, the upgrade from H100 to H200 is closer to a new generation than a refresh.
Spot B200 on Spheron runs $5.34/hr. On-demand is $7.50/hr. That's 29% cheaper for the exact same chip.
The catch is interruption risk: spot can be reclaimed, on-demand can't.
Pick spot for checkpointed training runs that can resume on their own. Pick on-demand for anything serving live traffic.
Same hardware, two very different guarantees behind the same invoice line.
Three chips, same benchmark, same model. Speed barely moved between them. Cost per token did: a 6x gap separated the cheapest from the priciest.
MLPerf Inference v6.0: Nvidia B200 ran at $0.073 per million tokens on spot pricing. H100 PCIe hit the same job at $0.451 per million on-demand. B200 also pushed about 17,500 tokens per second on Llama 2 70B, and beat H200 by 2.5x on GPT-OSS 120B.
Pick by scale:
- Job small enough that any chip clears your deadline: pick on throughput
- Running at volume: pick on cost per token
Full MLPerf v6.0 breakdown
Drop your target model and token volume below. We'll tell you which chip wins on your numbers.
