Guide : onehouse.ai/blog/spark-on-ku…

Spark on Kubernetes is hard to run well. Our live session on the these hard parts had a great turnout. If you missed it, here's the full written guide. Running Spark on Kubernetes looks deceptively easy. Most teams moving Spark from YARN to Kubernetes underestimate what YARN gave them for free. In reality, it's full of interesting problem solving on resource management and failure-handling. Here's what the guide covers: ⚙️ Basics around how the driver, executors, and the K8s scheduler work together, pod mapping 🏛️ Why teams are moving from YARN: unifying all compute on k8s for deploys, access controls, chargebacks, and autoscaling 🧩 spark-submit vs. operators, and where adding YuniKorn or Volcano can help 💥 Production lessons from running this at Onehouse: OOM kills, shuffle storage without an external shuffle service, and driver failures It starts small with SparkPi on a local kind cluster and builds up to what actually breaks at scale. We hope you find it useful. Guide link in the comments 👇

While we are at it, can you all please also publish your numbers on closed vs open formats.

If you want spare EC2 capacity, look at your Apache Spark clusters and then go give quanton.dev a shot.

sign up link: onehouse.ai/events/webinars/…

Spark on Kubernetes at scale is hard. Onehouse runs a lot of it, and tomorrow we're sharing what we've learned. In most places, the knowledge required to get all of this right lives across the data team and the Kubernetes/infra team. We've collected a list of things that looked simple at first, but hard in production: - OOM kills - Shuffle storage - Spot interruptions - Autoscaling - Long-running drivers - Cluster upgrades - Keeping streaming jobs alive while the infra underneath changes Most of it lives in the gap between the data team and the infra team. Webinar tomorrow. Bring your worst production problem. Link in comments.

This has got to be the most hype that gzip has had in a decade or more.

Alright . That is in fact pretty freakin' cool.

I feel this is AI generated. 🙂 What tool are you using?

Last week, I posted on why warehouses still default to closed formats. What I learned: they won't flip the default. Open formats are opt-in. So the fix is with the users. Give every open format (Delta, Hudi, Lance, Vortex, ClickHouse, DuckLake) the same consideration on performance and flexibility that you give the closed ones. Your data will thank you. Back to building on open formats.

Does the technical problem simply boil down to a very large table on object storage with fast writes and fairly fast analytical and point lookups?

I like that the ChatGPT Mac app has a built-in markdown editor. It definitely reduces friction in review cycles.

My wrists are happy that I have to type less. This feeling might be same elsewhere too

Not at all. Customers are stuck. We speak to dozens every week. They have to pay hefty egress costs to do the migration. The question I raised is more about first principle and forward-looking. Why not make open formats the default for any new table created from this point forward? That costs the vendor nothing. Btw the LinkedIn counterpart for this x post has a lot more thoughts if you are curious about this topic .

My thoughts and receipts are in the blog onehouse.ai/blog/snowflake-b…

Last week, Databricks CEO Ali Ghodsi and I debated Iceberg and open table formats on a post. Then I caught myself. Why are we debating each other while Snowflake, BigQuery and Redshift still default customers to closed formats? Two years of Iceberg announcements. Still no public retirement date for their closed formats. Open-source projects get scrutinized for interoperability on every release. The largest warehouses should meet the same bar. We’ve spent a decade building open formats and data lakes. I think it’s time we asked.

lol it's actually pretty basic when you see your servers only have 50% CPU utilization.

To squeeze the performance out of our runtime components, we implemented a good amount of concepts from - SEDA: an architecture for well-conditioned, scalable internet services dl.acm.org/doi/10.1145/50205… The seminal work at MIT around writing high-performance server software and the gift that keeps on giving.

I don't have extreme opinions on unit tests, but I hate the false sense of security they give developers, who only later realize the end-to-end functionality doesn't work at all.

Muse, Dots, Grok bot. Eagerly awaiting the consumer AI/agent app explosion, to watch agents melt our databases at scale.