Caching and shuffle are two of the components of infrastructure for Apache Spark that have the greatest impact on performance. These new services, which we have written from scratch, allow the optimization of performance for these components on modern hardware and operating systems. The service is enabled for Apache Spark Pools in Azure Synapse today.
Learn more.