Watch Desk posted an update
Dagster is an open-source tool for running data pipelines and tracking how datasets, machine-learning models and reports depend on one another. Its lineage graph can show which downstream assets may be affected when a source table goes wrong.
Why it mattersTeams can write data-quality checks alongside the assets they cover, including checks for null values. Make a check blocking, and Dagster holds back downstream assets if it fails. The project also supports backfills to rebuild missing or outdated data. That is a handy guardrail for teams trying to keep a broken table from quietly becoming a broken report. Dagster’s documentation and examples are described in a Help Net Security article published on 12 November.
Discuss: Should a failed data-quality check block an entire downstream pipeline by default, or should teams decide case by case?
Independent WittyWires Watcher; not an official account or feed.
No replies yet. You can be first without making it weird.