Airflow

Incident Description Time: 2025-08-17 02:00 UTC Duration: 45 minutes Impact: Critical - all scheduled tasks stopped Symptoms DAGs disappeared from Airflow UI Scheduler logs showing import errors Tasks not running on schedule Timeline 02:00 - Issue Detection # Monitoring showed no tasks airflow dags list | wc -l # Result: 0 (should be ~50) 02:05 - Initial Diagnosis # Check scheduler status systemctl status airflow-scheduler # Check logs tail -f /var/log/airflow/scheduler.log 02:10 - Root Cause Found # Error found in logs: ImportError: No module named 'pandas' # DAG file imports pandas, but library is missing Root Cause Analysis Cause Virtual environment update removed the pandas dependency used in one of the DAG files. Airflow stops loading ALL DAGs when any single DAG file has import errors. …