The latest release of Apache Spark 3.1 includes:
- Adaptive query execution
- Convert sort merge join to broadcast hash join
- Spark catalyst optimizer
- Dynamic partition pruning
- Create new Spark 3.1 clusters (not Spark 3.0 clusters (preview))
For additional details, review the document Apache Spark 3.1 on Microsoft Tech Community.
For a complete list of improvements, review the Apache Spark 3.1 release notes.
For more details on migration, review the migration guide.