@lmcache

🧪 Open Source Team that maintains LMCache and Production Stack 🤖 Democratizing AI by providing efficient LLM serving for ALL

 Github, Online
lmcache.ai/
Joined September 2024
  • Tweets307
  • Following56
  • Followers2K
150Media
When is it better to fetch a KV cache than to recompute it? New on the LMCache blog: an Internet for the KV cache, by Siddhant Ray, adapted from his arXiv paper with Nick Feamster and Junchen Jiang of the University of Chicago. One example: DeepSeek-V4-Pro, about 100,000 tokens, about 1 GB of compressed KV cache. Sending it over a 10 Gbps link takes about 0.8 s and $0.09. Recomputing it on AWS 8×B200 takes about 15 s and $0.47. Under the post’s assumptions, transfer is about 19× faster and 5.2× cheaper ⚡ (AWS prices are examples only.) blog.lmcache.ai/en/2026/10/2…

An Internet for the KV Cache: Store Anywhere, Use Everywhere

Sending a 1 GB KV cache over the Internet can be 19× faster and 5.2× cheaper than recomputing it. A proposal to store it anywhere, use it everywhere.

blog.lmcache.ai
2
1
4
72
What should we ask Dongjoo Seo about KV cache system deployment? 💬 Dongjoo Seo of Samsung speaks at 4:30 PM today at the LMCache hands-on workshop: "What we should consider for KV Cache system deployment." Reply with one question, and we will bring a few into the room. The workshop starts at 1:30 PM Pacific at Samsung, 3655 N First St, San Jose. It is free, and you can still register: ⏰ na2.hubs.ly/H08rNRd0
Side panel: technical session, Dongjoo Seo, Samsung, 4:30 PM, Thu 22 Oct, San Jose. Large question: What should we ask Dongjoo Seo about KV Cache system deployment? luma.com/cm7eip1b

ALT Side panel: technical session, Dongjoo Seo, Samsung, 4:30 PM, Thu 22 Oct, San Jose. Large question: What should we ask Dongjoo Seo about KV Cache system deployment? luma.com/cm7eip1b

1
1
159
Today: hands-on LMCache in San Jose. ⏰ Doors open at 1:30 PM Pacific at Samsung, 3655 N First St. Bring a laptop (recommended). Parking is available, and signs lead you to check-in. 📍 Four hours: an introduction to LMCache, a 45-minute deep dive with mini-lmcache, two technical sessions, and a reception until 5:30 PM. 🛠️ Registration is free: na2.hubs.ly/H08p3x20
Today, large orange 1:30 PM, doors open, Thu 22 Oct. Agenda at Samsung, San Jose: 2:10 intro, 2:45 deep dive, 4:00 Luis Chamberlain, 4:30 Dongjoo Seo, 5:05 reception. luma.com/cm7eip1b

ALT Today, large orange 1:30 PM, doors open, Thu 22 Oct. Agenda at Samsung, San Jose: 2:10 intro, 2:45 deep dive, 4:00 Luis Chamberlain, 4:30 Dongjoo Seo, 5:05 reception. luma.com/cm7eip1b

129
Today is the last day of PyTorch Conference. Tomorrow, 22 October, stay in San Jose for the LMCache hands-on workshop: 1:30 to 5:30 PM Pacific, at Samsung, 3655 N First St. 🛠️ Today is the last full day to register. It is free. 🎟️ Know someone at the conference who works on LLM inference? Send this to them before they leave San Jose. 🔁 na2.hubs.ly/H08k6110
Last full day to register. Today, the conference; tomorrow, hands-on. Panels: Wed 21 Oct, last day of PyTorch Conference; Thu 22 Oct, LMCache workshop, 1:30 PM, Samsung, San Jose. luma.com/cm7eip1b

ALT Last full day to register. Today, the conference; tomorrow, hands-on. Panels: Wed 21 Oct, last day of PyTorch Conference; Thu 22 Oct, LMCache workshop, 1:30 PM, Samsung, San Jose. luma.com/cm7eip1b

1
136
Tomorrow at 1:30 PM, doors open at Samsung, 3655 N First St, San Jose. 📍 Bring a laptop; it is recommended for the hands-on part. Parking is available, and signs lead you to check-in at the event location. Refreshments are provided. Setup help starts at 1:30 PM, so arrive early if you want someone to check your environment. 🛠️ The LMCache hands-on workshop runs until 5:30 PM Pacific. It is free, and registration is required. ⏰ na2.hubs.ly/H08k60h0
Large orange 1, day to go. Tomorrow at Samsung, San Jose: doors 1:30 PM, laptop recommended, parking, refreshments, follow the signs to check-in. Thu 22 Oct, 3655 N First St. luma.com/cm7eip1b

ALT Large orange 1, day to go. Tomorrow at Samsung, San Jose: doors 1:30 PM, laptop recommended, parking, refreshments, follow the signs to check-in. Thu 22 Oct, 3655 N First St. luma.com/cm7eip1b

1
1
2
192
What is the hardest KV cache question in your stack right now? 🧠 Bring it to the LMCache booth at PyTorch Conference, today or tomorrow, at the San Jose Convention Center. If it needs more time, bring it to our hands-on workshop on Thursday, and test your ideas against a live LMCache environment. Thursday 22 October, 1:30 to 5:30 PM Pacific, at Samsung, 3655 N First St, San Jose. Free. 📍 na2.hubs.ly/H08k5_-0
Dark card, at the LMCache booth: What is your hardest KV cache question? Bring it to the booth at PyTorch Conference, or to the workshop. Thu 22 Oct, 1:30 to 5:30 PM, Samsung, San Jose. luma.com/cm7eip1b

ALT Dark card, at the LMCache booth: What is your hardest KV cache question? Bring it to the booth at PyTorch Conference, or to the workshop. Thu 22 Oct, 1:30 to 5:30 PM, Samsung, San Jose. luma.com/cm7eip1b

1
3
215
Here's a question from our ecosystem of LMCache integrators: When you run several vLLM processes, does each one keep LMCache inside its own process, or can all of them share one LMCache service? Here's one way: In multi-process mode, you start lmcache server with CPU memory as L1 and FlashBlade over GPUDirect Storage (GDS) as L2. Each vLLM process connects to the shared server through LMCacheMPConnector. Josh Lay and Jean-Baptiste Thomas published the full server and vLLM configuration, pinned to lmcache==0.5.3 and vllm==0.24.0, on the Everpure blog: na2.hubs.ly/H08nWCR0 #LMCache #KVCache

How to Integrate LMCache with FlashBlade for Faster LLM Inference

Explore how LMCache and Everpure FlashBlade can enable persistent KV caching to help reduce time to first token and increase LLM inference throughput.

blog.everpuredata.com
1
3
245
PyTorch Conference day 1 starts this morning at the San Jose Convention Center, and LMCache has a booth. 📍 Find us to talk about KV cache management, and register at the booth for our hands-on workshop. The workshop is in 2 days: Thursday 22 October, 1:30 to 5:30 PM Pacific, at Samsung, 3655 N First St, San Jose. Free. ⏰ What should we show you at the booth? 💬 na2.hubs.ly/H08k5_K0
Large orange 2, days to go. PyTorch Conference, day 1: We are at the LMCache booth, San Jose Convention Center. Sign up there for the workshop, Thu 22 Oct, San Jose. luma.com/cm7eip1b

ALT Large orange 2, days to go. PyTorch Conference, day 1: We are at the LMCache booth, San Jose Convention Center. Sign up there for the workshop, Thu 22 Oct, San Jose. luma.com/cm7eip1b

130
Tag one teammate who should spend Thursday afternoon with LMCache. 🔁 22 October, 1:30 to 5:30 PM Pacific, at Samsung, 3655 N First St, San Jose: a live LMCache environment, a hands-on deep dive with mini-lmcache, engineering sessions, and a reception. A laptop is recommended. 🤝 It is free. Each of you registers here: na2.hubs.ly/H08k5PT0
Light card, 3 days to go, with @teammate in large type. Tag the person who should come with you. Free, registration required. Thu 22 Oct, 1:30 to 5:30 PM, Samsung, San Jose. luma.com/cm7eip1b

ALT Light card, 3 days to go, with @teammate in large type. Tag the person who should come with you. Free, registration required. Thu 22 Oct, 1:30 to 5:30 PM, Samsung, San Jose. luma.com/cm7eip1b

2
164
Going to PyTorch Conference this week? Find the LMCache booth at the San Jose Convention Center on 20 and 21 October. 📍 You can register at the booth for Thursday's LMCache hands-on workshop. It is the day after the conference ends: 22 October, 1:30 to 5:30 PM Pacific, at Samsung, 3655 N First St, San Jose. Free. 🎟️ Ask us about KV cache management while you are there. Or register now: na2.hubs.ly/H08k5Pw0
At PyTorch Conference? Find the LMCache booth. Tue 20 to Wed 21 Oct, San Jose Convention Center. Sign up there for the hands-on workshop, Thu 22 Oct, Samsung, San Jose. luma.com/cm7eip1b

ALT At PyTorch Conference? Find the LMCache booth. Tue 20 to Wed 21 Oct, San Jose Convention Center. Sign up there for the hands-on workshop, Thu 22 Oct, Samsung, San Jose. luma.com/cm7eip1b

1
122
In 3 days, LMCache runs a hands-on workshop in San Jose. Is it for you? 🧠 It is for engineers and researchers who work on KV cache management and LLM efficiency. If you run LLM inference and care about time-to-first-token (TTFT) and throughput, the afternoon is built for you. Thursday 22 October, 1:30 to 5:30 PM Pacific, at Samsung, 3655 N First St, San Jose. Free, registration required. ⏰ na2.hubs.ly/H08k5M40
Large teal 3 with the words days to go. Built for people who work on KV cache management and LLM efficiency. Thu 22 Oct, 1:30 to 5:30 PM, Samsung, San Jose. luma.com/cm7eip1b

ALT Large teal 3 with the words days to go. Built for people who work on KV cache management and LLM efficiency. Thu 22 Oct, 1:30 to 5:30 PM, Samsung, San Jose. luma.com/cm7eip1b

1
159
The second half of the LMCache workshop on Thursday 22 October has two Samsung speakers, one after the other. 🤝 4:00 PM: Luis Chamberlain (mcgrof) on asymmetric quantization, dma-buf with io_uring, and learned prefix caching. 4:30 PM: Dongjoo Seo, "What we should consider for KV Cache system deployment." 5:00 PM: closing remarks, then a reception until 5:30 PM. Before that: the Introduction to LMCache and a 45-minute hands-on deep dive with mini-lmcache. 🛠️ Free, registration required: na2.hubs.ly/H08rNNB0
Large orange 4, days to go. The second half, Thu 22 Oct: 4:00 Luis Chamberlain, Samsung; 4:30 Dongjoo Seo, Samsung; 5:00 closing remarks; 5:05 reception. San Jose. luma.com/cm7eip1b

ALT Large orange 4, days to go. The second half, Thu 22 Oct: 4:00 Luis Chamberlain, Samsung; 4:30 Dongjoo Seo, Samsung; 5:00 closing remarks; 5:05 reception. San Jose. luma.com/cm7eip1b

1
159
Which is harder for your stack: long context, or many users at once? 💬 Reply with one word. In 4 days you can bring that answer to a live LMCache environment, and to a room of people who run inference. The LMCache hands-on workshop is on Thursday 22 October, 1:30 to 5:30 PM Pacific, at Samsung, 3655 N First St, San Jose. Free, registration required. ⏰ na2.hubs.ly/H08k5K50
Large orange 4, days to go. Question: Which hurts more in your stack? Answer chips: Long context, or Many users at once. Thu 22 Oct, San Jose. luma.com/cm7eip1b

ALT Large orange 4, days to go. Question: Which hurts more in your stack? Answer chips: Long context, or Many users at once. Thu 22 Oct, San Jose. luma.com/cm7eip1b

1
125
The agenda for Thursday 22 October has three blocks for people: networking at 1:30 PM, a networking break at 3:30 PM, and a reception from 5:05 PM until 5:30 PM. 🤝 Bring a problem you have not solved in your own stack. Someone at Samsung in San Jose may run a similar setup. Which inference problem would you most like to discuss with peers? 💬 The LMCache hands-on workshop is free, and refreshments are provided. 📍 Registration is required: na2.hubs.ly/H08k5JR0
The people part: 1:30 PM registration and networking, 3:30 PM networking break, 5:05 PM networking reception until 5:30. Thu 22 Oct, Samsung, San Jose. luma.com/cm7eip1b

ALT The people part: 1:30 PM registration and networking, 3:30 PM networking break, 5:05 PM networking reception until 5:30. Thu 22 Oct, Samsung, San Jose. luma.com/cm7eip1b

3
256
45 minutes, one live deployment, and your laptop: that is the deep dive Samuel Shen of Tensormesh leads at 2:45 PM on Thursday 22 October. 🛠️ First, the mechanics of a public server deployment. Then hands-on work against the mini-lmcache API endpoints to benchmark and optimize KV cache management. The test environment stays open after the session. It is part of the free LMCache workshop, 1:30 to 5:30 PM Pacific, at Samsung, 3655 N First St, San Jose. 📍 Registration is required: na2.hubs.ly/H08p4vJ0
Deep dive with Samuel Shen, 2:45 PM: public server deployment, then hands-on against the API endpoints. Illustrated terminal. Thu 22 Oct, 1:30 to 5:30 PM, Samsung, San Jose. luma.com/cm7eip1b

ALT Deep dive with Samuel Shen, 2:45 PM: public server deployment, then hands-on against the API endpoints. Illustrated terminal. Thu 22 Oct, 1:30 to 5:30 PM, Samsung, San Jose. luma.com/cm7eip1b

1
120
Never run LMCache before? Start at 2:10 PM on Thursday 22 October. 🧠 Karsten Wade gives the Introduction to LMCache: what the KV cache layer does, and where it is useful. Its PyPI summary says it in one line: "A LLM serving engine extension to reduce TTFT and increase throughput, especially under long-context scenarios." TTFT means time-to-first-token. Then you try it yourself in the hands-on deep dive. 1:30 to 5:30 PM Pacific, Samsung, 3655 N First St, San Jose. Free. 📍 na2.hubs.ly/H08k5Dt0
Light card: New to LMCache? Start with the introduction: what the KV cache layer does, and where it helps. 2:10 PM, Karsten Wade. Thu 22 Oct, 1:30 to 5:30 PM, Samsung, San Jose. luma.com/cm7eip1b

ALT Light card: New to LMCache? Start with the introduction: what the KV cache layer does, and where it helps. 2:10 PM, Karsten Wade. Thu 22 Oct, 1:30 to 5:30 PM, Samsung, San Jose. luma.com/cm7eip1b

119
Open Source AI Week starts today in the Bay Area and runs to 25 October. 📅 If you come for PyTorch Conference at the San Jose Convention Center on 20 and 21 October, stay one more day. On Thursday 22 October, the LMCache hands-on workshop runs 1:30 to 5:30 PM Pacific at Samsung, 3655 N First St, San Jose. 📍 Four hours with a live LMCache environment, engineering sessions, and a reception. Free, registration required: na2.hubs.ly/H08k5yg0
Open Source AI Week starts today. Timeline: Fri 16 Oct, the week starts; Tue 20 to Wed 21 Oct, PyTorch Conference, San Jose Convention Center; Thu 22 Oct, LMCache workshop, Samsung, San Jose. luma.com/cm7eip1b

ALT Open Source AI Week starts today. Timeline: Fri 16 Oct, the week starts; Tue 20 to Wed 21 Oct, PyTorch Conference, San Jose Convention Center; Thu 22 Oct, LMCache workshop, Samsung, San Jose. luma.com/cm7eip1b

1
2
182
New speaker for the LMCache workshop: Luis Chamberlain (mcgrof) of Samsung takes the 4:00 PM session on Thursday 22 October. 🧠 His topics: asymmetric quantization, dma-buf support with io_uring, and learned prefix caching with Cartridges and beyond. Dongjoo Seo of Samsung speaks next, at 4:30 PM. 1:30 to 5:30 PM Pacific, Samsung, 3655 N First St, San Jose. Free. 📍 na2.hubs.ly/H08p3xz0
Speaker card: new speaker Luis Chamberlain, Samsung, mcgrof. Technical session: asymmetric quantization; dma-buf support with io_uring; learned prefix caching, Cartridges and beyond. Thu 22 Oct, 4:00 PM, Samsung, San Jose. Free. luma.com/cm7eip1b

ALT Speaker card: new speaker Luis Chamberlain, Samsung, mcgrof. Technical session: asymmetric quantization; dma-buf support with io_uring; learned prefix caching, Cartridges and beyond. Thu 22 Oct, 4:00 PM, Samsung, San Jose. Free. luma.com/cm7eip1b

1
141
After the workshop on Thursday 22 October, the mini-lmcache test environment stays open for your own experiments. 🛠️ So what would you test first? A long shared prompt? A document that many requests read again? Reply with one line. 💬 The hands-on deep dive is part of a free afternoon, 1:30 to 5:30 PM Pacific, at Samsung, 3655 N First St, San Jose. Registration is required: na2.hubs.ly/H08k5xM0
The test environment stays open after the session. Large question: What would you test first? Thu 22 Oct, 1:30 to 5:30 PM, Samsung, San Jose. luma.com/cm7eip1b

ALT The test environment stays open after the session. Large question: What would you test first? Thu 22 Oct, 1:30 to 5:30 PM, Samsung, San Jose. luma.com/cm7eip1b

1
1
1
129
Ecosystem check-in from our community of LMCache integrators: At 128K tokens of context, time to first token went from 33.5 s with no cache to 1.9 s with LMCache storing the KV cache on FlashBlade over GPUDirect Storage (GDS). Josh Lay and Jean-Baptiste Thomas ran the test on NVIDIA L40S with openai/gpt-oss-20b, vllm==0.24.0, and lmcache==0.5.3. The baseline is a cold prefill with no cache. How do you connect vLLM, LMCache, and FlashBlade to get this result? Their post on the Everpure blog has the configs: na2.hubs.ly/H08nWlC0 #LMCache #KVCache

How to Integrate LMCache with FlashBlade for Faster LLM Inference

Explore how LMCache and Everpure FlashBlade can enable persistent KV caching to help reduce time to first token and increase LLM inference throughput.

blog.everpuredata.com
1
1
4
254
Load more