#etl

Design a production-grade Airflow DAG for daily ETL. Workflow: 1. Extract data from PostgreSQL and REST API. 2. Transform using pandas (clean, join, aggregate). 3. Load to data warehouse (Snowflake/BigQuery). 4. Send Slack notification on success/failure. 5. Implement retry logic and SLA monitoring. Use TaskGroups for organization, XComs for data passing, and proper error handling with callbacks.

Build a robust data cleaning pipeline for a messy CSV dataset. Requirements: 1. Handle missing values using forward-fill, backward-fill, and mean imputation strategies. 2. Detect and remove outliers using IQR method. 3. Standardize date formats across multiple columns. 4. Remove duplicate rows based on composite keys. 5. Generate a data quality report showing before/after statistics. Use pandas best practices with method chaining for readability.