Systemd

Executive Summary Linux boot is a multi-stage handoff: UEFI → Bootloader → Kernel → systemd → Targets → Units. Each stage has failure points. This guide shows the sequence, where failures occur, and how to capture logs. Why understanding boot flow matters: When a Linux server won’t boot, you need to know WHICH stage failed to fix it effectively. A black screen could mean anything from bad hardware to a typo in /etc/fstab. …

Overview This is a one-page cheat sheet for Linux kernel subsystems. Each subsystem controls a critical resource; understanding them is essential for troubleshooting, optimization, and security. Why understanding subsystems matters: Imagine your server is slow. Without subsystem knowledge, you’re guessing: “Maybe add more RAM?” (might be CPU scheduler issue) “Maybe faster disk?” (might be memory cache problem) “Maybe more CPU?” (might be I/O scheduler misconfiguration) With subsystem knowledge, you diagnose systematically: …

Executive Summary Observability = see inside your systems: metrics (CPU, memory, I/O), logs (audit trail), traces (syscalls, latency). This guide covers: Metrics: node_exporter → Prometheus (system-level health) Logs: journald → rsyslog/Vector/Fluent Bit (aggregation) eBPF tools: 5 quick wins (trace syscalls, network, I/O) Triage: 5-minute flowchart to diagnose CPU, memory, I/O, network issues 1. Metrics: node_exporter & Prometheus What It Is node_exporter: Exposes OS metrics (CPU, memory, disk, network) as Prometheus scrape target Prometheus: Time-series database; collects metrics, queries, alerts Dashboard: Grafana visualizes Prometheus data Install node_exporter Ubuntu/Debian: …

Executive Summary Linux is a layered system: from firmware through kernel subsystems to containerized applications. Understanding these layers—and their interdependencies—is critical for reliable, secure, performant infrastructure. This guide covers: Layered architecture (firmware → kernel → userspace → containers) Core subsystems: process scheduling, memory, filesystems, networking systemd: unit management and service lifecycle Production best practices: security, reliability, performance, observability Note: For detailed boot flow and debugging, see the Linux Boot Flow & Debugging guide. …

Executive Summary Reliability means predictable, auditable behavior. This guide covers: Time sync: Chrony for clock accuracy (critical for logging, security) Networking: Stable interface names & hostnames (infrastructure consistency) Logging: Persistent journald + logrotate (audit trail + disk management) Shutdown: Clean hooks to prevent data loss Patching: Kernel updates with livepatch (zero-downtime), tested rollback 1. Time Synchronization (chrony) Why Time Matters Critical for: Logging: Accurate timestamps for debugging, compliance audits Security: TLS cert validation, Kerberos, API token expiry Distributed systems: Causality ordering (happens-before relationships) Monitoring: Alert timing, metric correlation Cost of poor time sync: …

Executive Summary A security baseline is the foundation: OS-hardened, patched, with restricted access and audit trails. This guide covers minimal-install servers with hardened SSH, firewall (default-deny), LSM enforcement, least-privilege sudo, audit logging, and systemd hardening. Goal: Reduce attack surface, detect breaches, and enforce privilege boundaries. 1. Minimal Install & Patching Minimal Install What it is: Install only required packages (base + SSH + monitoring agent) No GUI, X11, unnecessary daemons Reduces vulnerabilities (fewer packages = fewer CVEs) Install steps (Ubuntu/Debian): …