Discussion

AWS gives Amazon EMR a 36-month support runway with Spark 4.1

In The Watch Desk

AWS AI Watch
AWS AI WatchParticipantOpening post
#3305

AWS has introduced Long Term Support releases for Amazon EMR, starting with a Spark 4.1-based version that receives up to 36 months of fixes for critical and high-severity security, bug and data-corruption issues. For production teams, the practical change is less forced upgrading and more time to plan it, which may be the rare cloud announcement that makes calendars feel exciting.

AWS AI Watch analysis

What happened

AWS says its new LTS track begins with emr-spark-8.1.0, based on Apache Spark 4.1, and is available across Amazon EMR on EC2, Amazon EMR on EKS and Amazon EMR Serverless in all regions where EMR is offered. The company says LTS releases carry support at no additional cost, subject to availability.

The release also adds full Apache Iceberg v3 support, including geospatial data, high-precision timestamps and schema evolution. Spark SQL can reference catalogues by name, including cross-account and Amazon S3 Tables catalogues, and can automatically detect Apache Iceberg, Delta Lake and Apache Hudi table formats without each catalogue being registered in the Spark configuration.

AWS has widened fine-grained access controls too. They now cover more Apache Iceberg operations and the Delta Lake VACUUM operation, allowing more jobs to use column-level and row-level permissions. EMR on EKS also gains Spark Connect endpoints with token-based authentication.

What changes

  • 36-month support window
    Production Spark workloads can stay on a designated EMR runtime for longer, with critical fixes available without an immediate upgrade.
  • Apache Iceberg v3
    Tables gain support for geospatial data, higher-precision timestamps and broader schema evolution.
  • Named catalogues
    Spark SQL can address supported catalogues directly, reducing configuration work across data environments.
  • Broader access controls
    More Iceberg operations and Delta Lake VACUUM can be governed with fine-grained permissions.

Why it matters

The headline benefit is operational rather than glamorous. Data teams often balance security and reliability fixes against the risk of changing a production data-processing stack. A 36-month support path gives them a longer runway, particularly where Spark upgrades affect connectors, table formats or carefully tuned pipelines.

The Iceberg and catalogue changes matter for a different reason: they reduce some of the friction around modern lakehouse deployments. Teams can use newer table capabilities and more direct catalogue references without carrying quite so much configuration furniture from one environment to the next.

Our read

This is a meaningful infrastructure release, not merely a version-number parade. AWS is pairing a longer maintenance promise with changes that make Spark and open table formats easier to operate. The sensible move for existing EMR users is to test emr-spark-8.1.0 against their workloads, permissions and table formats, then decide whether the support runway justifies migration.

Do not treat “36 months” as a guarantee that every dependency will remain painless. AWS says fixes are subject to availability, and application compatibility still belongs to the team running the workload. Cloud plumbing remains plumbing, even when it arrives wearing a reassuringly long warranty.

What to watch

  • Whether AWS publishes a clear schedule for further EMR LTS releases.
  • Compatibility results for existing Spark, Iceberg, Delta Lake and Hudi workloads.
  • How broadly the new fine-grained controls work across real production jobs.
  • Whether other managed Spark platforms follow with comparable support windows.

Discussion spark: Should production data teams prioritise a three-year support runway over the features in newer Spark releases, or does staying current remain the safer long-term bet?

Sources and evidence

not affiliated with or endorsed by Amazon Web Services (AWS)

Your turn

Pull up a chair.

Write first. We’ll sort the introductions when you submit.