LinkedIn Scales Apache Spark and Flink Workloads with Apache Celeborn to Revolutionize Remote Shuffle Infrastructure

LinkedIn has successfully deployed Apache Celeborn as a shared, remote shuffle platform across the vast majority of its core Apache Spark and Apache Flink workloads, fundamentally separating intermediate data storage from underlying compute infrastructure. This architectural evolution marks a significant milestone in large-scale data engineering. By decoupling storage and compute, the professional networking platform has achieved a dramatic reduction in shuffle fetch failures—plummeting from approximately 750 daily occurrences down to just 20. Furthermore, the migration has yielded a 10% reduction in overall compute costs and an impressive 14% improvement in application runtimes when measured against LinkedIn’s legacy Magnet implementation.
The sheer scale of LinkedIn’s offline and real-time data architecture necessitates exceptionally robust infrastructure. The company’s offline data platform executes upwards of 300,000 distinct Spark jobs on a daily basis across thousands of production-grade compute nodes, consuming an estimated 200 petabyte-hours of compute resources annually. As LinkedIn systematically transitions toward a unified Apache Flink architecture capable of seamlessly handling both real-time streaming and historical batch processing, massive data backfills introduce substantial new shuffle workloads. In distributed data processing frameworks, the shuffle phase—which involves moving intermediate data between disparate processing stages—frequently serves as a critical bottleneck, placing immense stress on networks, storage disks, and application runtimes.
Background Context: The Evolution of Shuffle Optimization at LinkedIn
Historically, shuffle operations presented persistent performance bottlenecks for large-scale distributed engines. To mitigate shuffle read inefficiencies in Apache Spark, LinkedIn previously developed and deployed Magnet, a push-based shuffle implementation that was eventually contributed upstream to Apache Spark 3.2. Magnet functioned by aggregating shuffle blocks grouped by reducer partition into larger files, which substantially minimized the volume of random disk reads experienced by reducers.
However, under the Magnet architecture, shuffle storage remained tightly coupled to the compute cluster. External Shuffle Services running on compute nodes shared vital hardware resources—including CPU cycles, memory allocations, disk I/O bandwidth, and network interfaces—with the core compute workloads. This resource co-location frequently resulted in contention, latency spikes, and cascading task failures during periods of heavy data processing or infrastructure degradation.
The Architectural Shift to Apache Celeborn
To eliminate the systemic limitations of co-located shuffle storage, LinkedIn adopted Apache Celeborn, a cloud-native, distributed remote shuffle service designed specifically for big data workloads. Under the Celeborn paradigm, map tasks push their intermediate data directly to dedicated Celeborn worker nodes. These workers independently aggregate, sort, and store data partitions outside of the Spark and Flink compute executors. Reducers subsequently fetch this pre-organized data directly from the Celeborn cluster.

This decoupling allows shuffle storage to scale independently of compute resources, effectively insulating application runtimes from executor lifecycles. Unlike Magnet, which relied on merger External Shuffle Services embedded natively within the Spark compute cluster, Celeborn operates as an entirely independent distributed service featuring dedicated storage workers and a highly available, replicated control plane. By removing shuffle operations from the compute layer, LinkedIn has successfully eradicated resource contention between data processing tasks and intermediate data management.
Engineering Resilience and High Availability at Scale
Operating a remote shuffle platform across thousands of distributed nodes and hundreds of thousands of daily production jobs required extensive engineering innovation and hardening beyond the upstream open-source project. Aravind Patnam, a staff software engineer at LinkedIn intimately
involved in the deployment, highlighted the engineering challenges, noting that ensuring extreme reliability across a massive fleet demanded sophisticated architectural enhancements.
To guarantee high availability and fault tolerance, LinkedIn engineered a robust five-node Raft-based control plane distributed strategically across multiple availability zones. This control plane utilizes Apache Ratis to handle critical metadata replication, leader election, and automated failover procedures. Additionally, LinkedIn integrated network-aware replication strategies and coupled Celeborn with the corporate host health platform. This integration enables the infrastructure to proactively deprioritize Celeborn workers that are approaching scheduled maintenance cycles and automatically exclude degraded hardware nodes from active traffic rotation. According to internal engineering metrics, these reliability-focused modifications contributed to a twentyfold reduction in task failures during unexpected infrastructure disruptions compared to the legacy Magnet framework.
Data Plane Optimizations and Performance Metrics
Beyond control plane resilience, LinkedIn implemented aggressive data plane optimizations to maximize throughput and minimize latency. The engineering team migrated shuffle storage workloads to high-performance NVMe SSDs, a transition that reduced overall shuffle duration by 40%. Furthermore, by replacing the traditional LZ4 compression algorithm with Zstandard (Zstd), LinkedIn achieved superior data compaction, reducing the required shuffle worker fleet size by approximately 35%.
To prevent network saturation, the platform was provisioned with robust 100 Gbps network interfaces, and slot selection times were optimized sixfold through targeted Java Virtual Machine (JVM) tuning and data-plane enhancements. Across LinkedIn’s expansive compute fleet, the net cost of shuffle operations decreased by nearly 90% per task. Concurrently, the proportion of executor runtime spent idling while waiting on shuffle operations fell by 10% to 15% from historical peak levels.
/filters:no_upscale()/news/2026/11/linkedin-celeborn-remote-shuffle/en/resources/1Screenshot%202026-10-31%20at%203.44.38 PM-1793488412813.png)
Unifying Real-Time and Historical Processing with Flink
The successful deployment of Apache Celeborn serves as a foundational enabler for LinkedIn’s broader architectural convergence toward Apache Flink. The newly unified Flink platform processes hundreds of millions of events every hour during intensive historical backfills, successfully compressing multi-day analytical workloads into single-digit hours.
Thousands of production data pipelines now leverage unified Flink computations for nearline processing and historical corrections, effectively replacing legacy parallel Spark implementations that required duplicated engineering logic. Looking ahead, LinkedIn’s engineering organization is actively exploring advanced caching mechanisms and faster upstream software upgrades while maintaining its commitment to contributing back to the open-source Apache Celeborn community.
Broader Industry Implications and Future Outlook
LinkedIn’s large-scale migration to Apache Celeborn illustrates a broader architectural trend within modern enterprise data engineering: the disaggregation of storage and compute. As data volumes continue to scale into the exabyte era, traditional co-located shuffle architectures increasingly struggle to meet the elasticity, reliability, and cost-efficiency demands of hyper-scale organizations.
By proving that a dedicated, disaggregated shuffle service can dramatically reduce infrastructure costs, minimize failure rates, and accelerate runtimes for both Spark and Flink, LinkedIn has established a compelling blueprint for other organizations managing massive distributed data ecosystems. The success of this deployment underscores the immense value of open-source collaboration combined with rigorous, production-grade engineering, signaling a transformative shift in how modern big data platforms handle intermediate data exchange.







