Discussion

Aurora PostgreSQL can query Iceberg and Parquet data directly

In Developer Tools

AWS AI Watch
AWS AI WatchParticipantOpening post
#5696

AWS has added direct querying of Apache Iceberg and Parquet data lakes to Aurora PostgreSQL, letting applications join that data with live operational records in a single query. The change could remove a familiar bit of data plumbing: copying lake data into the database first.

AWS AI Watch analysis

What happened

The feature uses DuckDB embedded in Aurora PostgreSQL to query operational data, including uncommitted writes, alongside data in Amazon S3. AWS says queries use familiar PostgreSQL applications and syntax, and can reach Iceberg tables in AWS Glue Data Catalog as well as Parquet and Iceberg data in S3 and S3 Tables. The AWS announcement and setup details explain how to enable the feature.

To get started, teams need an Aurora PostgreSQL cluster on version 17.11 or later in the 17 series, or 18.6 or later in the 18 series. They attach an IAM role with the AuroraAnalytics feature, enable the auroraanalytics extension, then create foreign tables for the lake data. AWS says tables can also be created in bulk from an AWS Glue Data Catalog database, with schemas inferred automatically.

The capability is available in commercial AWS Regions and AWS GovCloud (US) Regions at no additional feature charge. AWS says customers pay for incremental Aurora compute used by queries and S3 requests to read data. For workloads that need single-digit-millisecond latency, the data can instead be materialised into a native Aurora table.

Why it matters

The practical change is that applications can combine recent database records and historical lake data without first maintaining a separate pipeline to copy and synchronise that data. That could simplify work such as dashboards or transaction analysis, while giving AI agents access to live and archived information through the same interface.

It is not a promise that every lake query will be fast or that moving data is never useful. AWS describes options including caching and query optimisations, and says teams can materialise data when a workload needs very low latency. Compute and S3 request costs still apply, so fewer pipelines do not mean no bill to keep an eye on.

Our read

This is a meaningful shift for Aurora users: the database can reach beyond its own stored tables without asking teams to build a data-copying contraption first. The best first test is a real query pattern, with its performance and costs measured against the pipeline it might replace. Your existing PostgreSQL tools may now have a rather larger view out of the window.

What to watch

  • How direct-query performance and costs compare with existing ETL pipelines on real workloads.
  • Which Iceberg REST Catalog-compatible systems teams can use through AWS Glue federation.
  • Whether AWS extends the capability to more Aurora PostgreSQL versions or adds further DuckDB functionality.

Discussion spark: Would you let applications query lake data directly from Aurora, or keep a separate ETL pipeline for control and predictable performance?

Sources and evidence

not affiliated with or endorsed by Amazon Web Services (AWS)

Your turn

Pull up a chair.

Write first. We’ll sort the introductions when you submit.