Automation

Introduction Infrastructure as Code (IaC) is how modern teams build reliable systems. Instead of manually clicking through cloud consoles or SSHing into servers, you define infrastructure in code—testable, version-controlled, repeatable. This guide shows you practical patterns for Terraform, Ansible, and Kubernetes with real examples, not just theory. Why Infrastructure as Code? Consider a production outage scenario: Without IaC: Database server dies You manually recreate it through AWS console (30 minutes) Forgot to enable backups? Another 15 minutes Need to reconfigure custom security groups? More time Total recovery: 2-4 hours Risk of missing steps = still broken With IaC: …

Introduction Slow CI/CD pipelines waste developer time and delay releases. This guide covers proven techniques to optimize pipeline performance including build caching, parallel job execution, and efficient deployment strategies across popular CI/CD platforms. Build Caching Why Caching Matters Without caching: Build 1: npm install (5 min) → tests (2 min) = 7 min Build 2: npm install (5 min) → tests (2 min) = 7 min Build 3: npm install (5 min) → tests (2 min) = 7 min Total: 21 minutes With caching: …

Introduction GitOps is a paradigm that uses Git as the single source of truth for declarative infrastructure and applications. ArgoCD and Flux are the leading tools for implementing GitOps on Kubernetes. This guide covers deployment patterns, rollback strategies, and choosing between the two. GitOps Principles Core Concepts 1. Declarative - Everything defined in Git 2. Versioned - Git history = deployment history 3. Automated - Tools sync Git to cluster 4. Auditable - All changes tracked in Git …

Introduction Terraform state is the source of truth for your infrastructure. Proper state management is critical for team collaboration, preventing conflicts, and maintaining infrastructure integrity. This guide covers remote backends, locking mechanisms, and workspace strategies. Understanding Terraform State What is State? State is Terraform’s way of tracking which real-world resources correspond to your configuration. It’s stored in terraform.tfstate file. State file contains: Resource mappings Metadata Resource dependencies Attribute values Why State Matters Without proper state management: …

Introduction Toil is manual, repetitive, automatable work that scales linearly with service growth. It’s the operational burden that keeps engineers from doing valuable engineering work. Reducing toil is essential for scaling both systems and teams effectively. What is Toil? Google’s SRE Definition Toil has the following characteristics: Manual - Requires human action Repetitive - Done over and over Automatable - Could be automated Tactical - Reactive, interrupt-driven No enduring value - Doesn’t improve the system Scales linearly - Grows with service growth Toil vs Engineering Work Toil (eliminate this): …

Incident Description Time: 2025-08-17 02:00 UTC Duration: 45 minutes Impact: Critical - all scheduled tasks stopped Symptoms DAGs disappeared from Airflow UI Scheduler logs showing import errors Tasks not running on schedule Timeline 02:00 - Issue Detection # Monitoring showed no tasks airflow dags list | wc -l # Result: 0 (should be ~50) 02:05 - Initial Diagnosis # Check scheduler status systemctl status airflow-scheduler # Check logs tail -f /var/log/airflow/scheduler.log 02:10 - Root Cause Found # Error found in logs: ImportError: No module named 'pandas' # DAG file imports pandas, but library is missing Root Cause Analysis Cause Virtual environment update removed the pandas dependency used in one of the DAG files. Airflow stops loading ALL DAGs when any single DAG file has import errors. …

SSL Certificate Renewal Runbook Quick Reference: Duration: 5-15 minutes Impact: Minimal (usually zero downtime) Requires: Sudo access to load balancer or cert-manager admin Severity: HIGH (expired certs = full outage) Prerequisites What You Need SSH access to certificate servers or kubectl access to cluster Let’s Encrypt API credentials (if manual renewal) Backup certificate location documented 30+ days lead time before expiry (not 5 minutes!) Check Current Status # View certificate expiry openssl s_client -connect example.com:443 -showcerts | grep -A5 "Verify return code" # Or via Kubernetes kubectl get certificate -n ingress-nginx kubectl describe certificate prod-cert -n ingress-nginx # Check expiry date specifically openssl x509 -in /etc/ssl/certs/server.crt -noout -enddate Automatic Renewal (Preferred) Using cert-manager (Kubernetes) apiVersion: cert-manager.io/v1 kind: Certificate metadata: name: prod-cert namespace: ingress-nginx spec: secretName: prod-tls-secret issuerRef: name: letsencrypt-prod kind: ClusterIssuer dnsNames: - example.com - "*.example.com" Cert-manager automatically: …