Observability

Introduction Observability is the ability to understand the internal state of a system based on its external outputs. Unlike traditional monitoring, which tells you what is broken, observability helps you understand why it’s broken, even for issues you’ve never encountered before. Core Principle: “You can’t fix what you can’t see. You can’t see what you don’t measure.” The Three Pillars Overview ┌─────────────────────────────────────────┐ │ OBSERVABILITY │ ├─────────────┬──────────────┬────────────┤ │ METRICS │ LOGS │ TRACES │ ├─────────────┼──────────────┼────────────┤ │ What/When │ Why/Details │ Where │ │ Aggregated │ Individual │ Causal │ │ Time-series │ Events │ Flows │ │ Dashboards │ Search │ Waterfall │ └─────────────┴──────────────┴────────────┘ When to Use Each: …

Introduction Chaos Engineering is the discipline of experimenting on a system to build confidence in the system’s capability to withstand turbulent conditions in production. Rather than waiting for failures to happen, chaos engineering proactively injects failures to identify weaknesses before they impact users. Why does this matter? In modern distributed systems (microservices, cloud infrastructure, containers), failures are inevitable. A network can partition, a server can crash, a database can slow down. Traditional testing can’t predict all the ways these components interact when things go wrong. Chaos engineering fills this gap by deliberately causing failures in a controlled way. …

Executive Summary Observability = see inside your systems: metrics (CPU, memory, I/O), logs (audit trail), traces (syscalls, latency). This guide covers: Metrics: node_exporter → Prometheus (system-level health) Logs: journald → rsyslog/Vector/Fluent Bit (aggregation) eBPF tools: 5 quick wins (trace syscalls, network, I/O) Triage: 5-minute flowchart to diagnose CPU, memory, I/O, network issues 1. Metrics: node_exporter & Prometheus What It Is node_exporter: Exposes OS metrics (CPU, memory, disk, network) as Prometheus scrape target Prometheus: Time-series database; collects metrics, queries, alerts Dashboard: Grafana visualizes Prometheus data Install node_exporter Ubuntu/Debian: …

Executive Summary Linux is a layered system: from firmware through kernel subsystems to containerized applications. Understanding these layers—and their interdependencies—is critical for reliable, secure, performant infrastructure. This guide covers: Layered architecture (firmware → kernel → userspace → containers) Core subsystems: process scheduling, memory, filesystems, networking systemd: unit management and service lifecycle Production best practices: security, reliability, performance, observability Note: For detailed boot flow and debugging, see the Linux Boot Flow & Debugging guide. …

Executive Summary Neo4j is a native graph database that stores data as nodes (entities) connected by relationships (edges). Unlike relational databases that normalize data into tables, Neo4j excels at traversing relationships. Quick decision: Use Neo4j for: Knowledge graphs, authorization/identity, recommendations, fraud detection, network topology, impact analysis Don’t use for: Heavy OLAP analytics, simple key-value workloads, document storage Production deployment: Kubernetes + Helm (managed) or Docker Compose + Causal Cluster (self-managed) …

Introduction Service Level Indicators (SLIs), Service Level Objectives (SLOs), and Service Level Agreements (SLAs) are fundamental concepts in Site Reliability Engineering. Understanding and implementing them correctly is crucial for maintaining reliable services. Core Concepts SLI (Service Level Indicator) Definition: A quantitative measure of service reliability from the user’s perspective. Common SLIs: Availability: Percentage of successful requests Latency: Proportion of requests served faster than threshold Throughput: Requests processed per second Error Rate: Percentage of failed requests Example SLI Definitions: …