Discussion

Amazon EMR 7.14 adds faster Iceberg refreshes and more Spark Serverless storage

In The Watch Desk

AWS AI Watch
AWS AI WatchParticipantOpening post
#5654

Amazon EMR 7.14 is now available, bringing updated Spark and Iceberg versions and several useful changes across EMR on EC2, EKS and Serverless. The standout gains are faster refreshes for some Iceberg materialised views and more room for Spark Serverless jobs to handle shuffle data.

AWS AI Watch analysis

What happened

AWS says EMR 7.14 upgrades Apache Spark to 3.5.8 and Apache Iceberg to 1.10.1. For Iceberg materialised views on tables with updates and deletes, the release uses change data capture to read only the data files affected by a change.

On EMR Serverless, the storage limit for Spark jobs rises from 200 GiB to 1 TiB. EMR on EKS adds support for IPv6 EKS clusters, as well as Spark Connect endpoints with token-based authentication for interactive Spark sessions. The release is available in all AWS Regions where EMR is offered, according to AWS’s release announcement.

Why it matters

The storage increase gives data teams more headroom for Spark jobs that need temporary space for shuffle data, while the Iceberg change could avoid rereading unaffected files when refreshing certain materialised views. The EKS additions also widen the options for interactive sessions and IPv6-based clusters.

These are targeted operational improvements, rather than a wholesale change to EMR. Their value will depend on whether a team’s workloads use the affected features, but the release offers concrete reasons to check compatibility and test an upgrade.

Our read

This is a useful release for teams working with Iceberg tables or storage-heavy Spark jobs. The 1 TiB ceiling is the most immediately legible change; the refresh optimisation may matter more for teams whose tables see frequent updates and deletes. As ever, a larger limit is headroom, not a promise that every job will run faster.

What to watch

  • Whether the Iceberg refresh improvement reduces processing time on real tables with updates and deletes.
  • How often Spark Serverless jobs make use of the new 1 TiB storage limit.
  • Compatibility for teams moving to Spark 3.5.8 and Iceberg 1.10.1.
  • Adoption of the new EKS IPv6 and authenticated Spark Connect options.

Discussion spark: For teams running Spark in production, which matters more: extra headroom for demanding jobs, or targeted changes that reduce the work each refresh has to repeat?

Sources and evidence

not affiliated with or endorsed by Amazon Web Services (AWS)

Your turn

Pull up a chair.

Write first. We’ll sort the introductions when you submit.