Eclipse Aerospace has moved its flight-data decoding into Databricks’ Spark workflow, replacing a multi-stage process with a Rust-backed PySpark data source. The change gives engineers quicker access to decades of telemetry while cutting duplicate storage and making pipeline monitoring easier.
Databricks Watch analysis
What happened
Eclipse Aerospace stores nearly 20 years of flight data, covering more than a million flight hours. Its proprietary formats previously required a separate decoding stage: files were converted to Avro through AWS Lambda before being loaded into Spark. That added latency, duplicated storage and split troubleshooting across different tools.
The new approach wraps Eclipse’s existing Rust decoder in a Python data source and returns Apache Arrow record batches directly to Spark. Databricks’ account says the connector commonly processes large, multi-gigabyte batches in under two minutes, including startup time, and can make records available within a minute of upload. Eclipse also reports benchmark performance around twice that of a comparable PyArrow-based pure-Python data source, and nearly 20 times that of a native Python implementation without Arrow.
Why it matters
The practical improvement is not simply a faster decoder. Keeping ingestion inside Spark removes the Lambda-and-Avro detour, avoids storing both original and converted files, and brings decode errors and job metrics into the same workflow engineers use to process the data. The approach also lets the team reuse its Rust decoder without building a separate JVM connector.
Databricks says the pattern could help with other specialised formats, including scientific HDF5 data and aerospace ARINC 429 data. That is a possible application, not evidence that this particular connector has already been deployed across those fields.
Our read
This is a useful example of infrastructure detail turning into an operational gain: less waiting, less duplicated plumbing and fewer places to go looking when a pipeline misbehaves. The performance figures come from the Databricks case study, so they are best read as reported results for this implementation, not a universal promise for every custom data source. Still, bringing a Rust decoder to Spark without making the engineering team build a JVM connector is a neat bit of practical work.
What to watch
- Whether Eclipse reports longer-term latency, storage or cost comparisons against its old pipeline.
- How the connector performs across different file sizes and workloads.
- Whether other teams adopt the same Rust-and-Arrow pattern for specialised data formats.
Discussion spark: When bringing specialised data into a data platform, should teams prioritise faster ingestion, simpler operations or easier reuse of existing code?
Sources and evidence
- Eclipse Aerospace simplifies flight-data ingestion with Databricks (12 November 2026, 19:20 UTC)
Independent WittyWires tracker for public updates about Databricks. Not affiliated with or endorsed by Databricks; this is not an official account.