Apache Spark (@ApacheSpark) header
Apache Spark (@ApacheSpark) profile picture

Lightning-fast unified analytics engine

Joined June 2013
Photo by Apache Spark (@ApacheSpark)Photo by Apache Spark (@ApacheSpark)Photo by Apache Spark (@ApacheSpark)Photo by Apache Spark (@ApacheSpark)Photo by Apache Spark (@ApacheSpark)Photo by Apache Spark (@ApacheSpark)Photo by Apache Spark (@ApacheSpark)Photo by Apache Spark (@ApacheSpark)Photo by Apache Spark (@ApacheSpark)Photo by Apache Spark (@ApacheSpark)
Apache Spark Connect Swift Client 0.8.0 is now available, with support for Apache Spark 4.3! This release: ✅ Uses Swift 6.4 as the minimum supported version ✅ Is tested with Apache Spark 4.0.5, 4.1.4, 4.2.1 and 4.3.0 ✅ Includes many feature-parity improvements across the client The Swift client enables developers to use Apache Spark from Swift applications through Spark Connect’s language-independent protocol. 🔗 Learn more: apache.github.io/spark-conne… Thank you to everyone who contributed to, reviewed and tested this release!
1
1
9
1,489
Apache Spark 4.3 adds a shuffle that streams. Records move to the consumer as the producer emits them. The producer and consumer run as one pipelined shuffle group, scheduled and retried together. Less idle time, lower latency. Your job code doesn't change. 🎥 Watch: youtube.com/watch?v=MbVjt61D… #ApacheSpark #BigData
2
16
1,348
Apache Spark 4.3 introduces stateful operation support in Real-Time Mode with sub-150ms P99 latency. State now persists in memory and updates in place rather than checkpointing on every trigger. Existing stateful operations work with minimal code changes when the trigger is set to real-time mode. 🎥 Watch: youtube.com/watch?v=MbVjt61D…
14
1,034
Millisecond latency on Structured Streaming used to mean a second engine. Next week at QCon San Francisco, Jerry Peng (@databricks) will show how RT brings that capability into Apache Spark itself: same SQL streaming model, lower latency path. In this session: 🔸 What RTM changes under the hood 🔸 How low-latency workloads run without leaving Spark 🔸 Why that matters for consolidating batch, ETL, and real-time on one platform 🗓️ Monday, Nov 16 📍 San Francisco 🔗 Session details: qconsf.com/presentation/nov2…
1
9
1,051
Apache Spark 4.3 is out now, and @lisancao walks through the latest updates. ⚡ This video covers stateful real-time mode, pipeline shuffle, AUTO CDC with SCD Type 2, Spark Connect operation IDs, DataFrame. zip, nested JSON in SQL, and more. 🎥 Watch the full overview: youtu.be/MbVjt61Dv1c #ApacheSpark
2
31
2,816
Apache Spark 4.3 is here, bringing stateful real-time streaming to the engine you already use for batch, analytics, and AI. ⚡ Stateful Real-Time Mode: streaming aggregations, dropDuplicates, and transformWithState run with P99 end-to-end latency under 150 ms in reported benchmarks 🕰️ SCD Type 2 in Auto CDC: preserve history from Python, SQL, and Spark Connect 🪶 Project Feather: opt-in shuffle-free execution for eligible small queries, plus Arrow-native caching 🌐 Spark Connect: REST endpoints for sessions and operations, and a more complete JDBC driver 🤖 SQL: JSON_TABLE, VARIANT transformations, ASOF JOIN, and a preview of nanosecond timestamps 📅 Feature releases move to every three months instead of every six Thank you to the 210+ contributors and the Apache Spark community behind 1,100+ Jira tickets. Read more 👇 databricks.com/blog/introduc…
1
11
76
3,002
Apache Spark Data Source V2 is Spark’s way to support native table formats, not only files. Szehon Ho walks through why that exists. Early Spark read and wrote file formats such as Parquet and used a thin Hive Metastore catalog. Apache Iceberg and Delta Lake added a much richer metadata layer so engines can run large, transactional operations. DSV2 expands that surface so a data source can plug that metadata into Spark, including through catalogs. 🎥 youtube.com/watch?v=MOQ3G8Ul… #ApacheSpark #OpenSource
1
5
35
2,712
We are happy to announce the availability of Apache Spark 4.1.4! 🎉 Spark 4.1.4 is the fourth maintenance release containing security and correctness fixes. This release is based on the branch-4.1 maintenance branch of Spark. We strongly recommend all 4.1 users to upgrade to this stable release. To download Spark 4.1.4, head over to the download page: spark.apache.org/downloads.h… Spark 4.1.4 is also available on PyPI (pyspark, pyspark-connect, pyspark-client) and the Maven Central Repository. To view the release notes: spark.apache.org/releases/sp… We would like to acknowledge all community members for contributing to this release. This release would not have been possible without you.
1
7
62
4,532
Agents are writing a ton of Spark code these days. Most of it is error-ridden, and the errors are the boring kind: a column that doesn’t exist, a table nothing upstream declares, a session config invented from thin air. At OSA Con 2026, @lisancao (@databricks) will cover declarative pipelines so structural mistakes get caught before data moves, Spark Connect so session context and credentials stay out of the agent’s hands, and skills and prompts that make one-shots more successful. She’ll also cover the tradeoffs between model choice, tokens, and deployment infrastructure. 🗓️ Monday, November 2 🕓 10:35-11:00 AM PST 📍 San Francisco + online · free 🔗 Session: osacon.io/sessions/2026/gett… #ApacheSpark #SparkConnect #OSACon
1
3
34
2,529
The core issue is that their source system is both relational and a document database. All of the documents are stored as XML. Figuring out how to handle that in production was a headache, because it is very challenging for end users to parse XML or JSON. That is where Variant comes in. Geir Alstad walks through this in a talk on Apache Spark Variant shredding in production. 🎥 Watch the full session: youtube.com/live/imEjS0UoZ2I #ApacheSpark
2
3
4
1,163
Agents don’t ask one question a day. They ask 10 or 100, against the same data stack and the same SLA. At Open Lakehouse + AI in Bellevue, Felix Cheung (@nvidia) walks through GPU-accelerated Apache Spark with no code change, plus @DeltaLakeOSS, @ApacheIceberg, and Spark Connect. 🎥 Watch: youtube.com/watch?v=OE59djsc… #OpenLakehouse #ApacheSpark
6
1,494
Apache Spark can analyze your query plans and derive the dependencies for you. You don’t need to declare them anymore. @lisancao sits down with Andreas Neumann on Spark Declarative Pipelines. Spark owns the graph, manages session lifecycle and writes, while @ApacheAirflow orchestrates Spark as an application. 🎥 Watch the full conversation: youtube.com/watch?v=dS5AwlC-… #ApacheSpark #ApacheAirflow #DataEngineering
1
6
1,058
In Apache Spark 4.2, retrieval (NEAREST BY), governed semantics (metric views), and agent access (Spark Connect) run inside the engine, next to the data and under the same permissions, instead of in separate services. Download Spark 4.2 👉 spark.apache.org/downloads.h… #ApacheSpark #AI #GenAI
2
797
Felix Cheung (Apache Spark PMC, @nvidia) and @lisancao on running Spark SQL and DataFrames on GPUs with NVIDIA’s cuDF plugin, no job rewrites. Snapchat’s A/B experimentation pipeline: 10 PB of new data a day, processed with Spark. On GPUs, 70-90% lower cost. Also Spark Connect, including how a Spark Connect Gateway can sit in front of multiple servers, GPU ones included. 🎥 Full talk: youtu.be/nL6NVivpIfQ #ApacheSpark #SparkConnect #cuDF
1
11
2,068
Spark SQL picks up several additions in Apache Spark 4.2, out now: SET PATH for object resolution, scripting cursors, QUALIFY for window results, SYSTEM.BUILTIN qualification, plus Theta sketches, time_bucket, and top-K max_by/min_by. More analytical work stays in SQL. 🔗 Download Spark 4.2: spark.apache.org/downloads.h… #ApacheSpark #SparkSQL #SQL
17
1,378
Apache Spark 4.2 is out. Python Data Sources let you build batch or streaming readers and writers in Python, no JVM or Scala, used through the standard spark.read interface. Spark 4.2 adds time and memory profiling so they're easier to tune and operate. 🔗 Download: spark.apache.org/downloads.h… #ApacheSpark #PySpark #Python
1
1
14
1,497
Auto CDC arrives in Spark Declarative Pipelines with Apache Spark 4.2: first-class SCD Type 1. Configure how change events update a target table, and Spark handles deletions and out-of-order events, in place of hand-written merge logic. 🔗 Download Spark 4.2: spark.apache.org/downloads.h… #ApacheSpark #CDC #DataPipelines #ETL
7
2
17
1,757