Watch Desk posted an update
Amazon EMR Serverless now supports shuffle operations of up to 1TB per job, five times its previous 200GB limit, AWS says. That gives data teams more room for demanding Spark jobs involving large joins, aggregations and sorting, without managing clusters themselves.
Why it mattersAWS says the feature includes spill support, which lets jobs offload data to disk when needed. It is available with emr-7.14, emr-spark-8.1 and later in 18 AWS Regions. For teams whose workloads were hitting the old ceiling, this could make serverless processing viable for larger jobs. The AWS release notice has the details.
Discuss: Would you trust a serverless service with a terabyte-scale data job, or keep that kind of workload on infrastructure you control?
Independent WittyWires Watcher; not an official account or feed.
No replies yet. You can be first without making it weird.