Linux

Executive Summary Linux boot is a multi-stage handoff: UEFI → Bootloader → Kernel → systemd → Targets → Units. Each stage has failure points. This guide shows the sequence, where failures occur, and how to capture logs. Why understanding boot flow matters: When a Linux server won’t boot, you need to know WHICH stage failed to fix it effectively. A black screen could mean anything from bad hardware to a typo in /etc/fstab. …

Overview This is a one-page cheat sheet for Linux kernel subsystems. Each subsystem controls a critical resource; understanding them is essential for troubleshooting, optimization, and security. Why understanding subsystems matters: Imagine your server is slow. Without subsystem knowledge, you’re guessing: “Maybe add more RAM?” (might be CPU scheduler issue) “Maybe faster disk?” (might be memory cache problem) “Maybe more CPU?” (might be I/O scheduler misconfiguration) With subsystem knowledge, you diagnose systematically: …

Executive Summary Networking baseline = reliable, secure, predictable connectivity with proper tuning for your infrastructure. Why networking configuration matters: Most production outages trace back to network issues: misconfigured firewall blocking traffic, exhausted connection tables, or timeouts set too aggressively. Proper networking prevents these disasters. Real-world disasters prevented by good networking: 1. Firewall accidentally blocks production traffic: Problem: Engineer adds SSH rule, accidentally sets policy to "drop all" Result: Website goes down, SSH also blocked (can't fix it remotely) Prevention: Test firewall rules with policy "accept" first, then switch to "drop" 2. Connection tracking table exhausted: …

Executive Summary Observability = see inside your systems: metrics (CPU, memory, I/O), logs (audit trail), traces (syscalls, latency). This guide covers: Metrics: node_exporter → Prometheus (system-level health) Logs: journald → rsyslog/Vector/Fluent Bit (aggregation) eBPF tools: 5 quick wins (trace syscalls, network, I/O) Triage: 5-minute flowchart to diagnose CPU, memory, I/O, network issues 1. Metrics: node_exporter & Prometheus What It Is node_exporter: Exposes OS metrics (CPU, memory, disk, network) as Prometheus scrape target Prometheus: Time-series database; collects metrics, queries, alerts Dashboard: Grafana visualizes Prometheus data Install node_exporter Ubuntu/Debian: …

Executive Summary Performance baseline = safe defaults that work for most workloads, with clear tuning for specific scenarios. This guide covers: sysctl: Kernel parameters (network, filesystem, VM) with production-safe values ulimits: Resource limits (open files, processes, memory locks) CPU Governor: Frequency scaling & power management on servers NUMA: Awareness for multi-socket systems (big apps, databases) I/O Scheduler: NVMe/SSD vs. spinning disk tuning 1. sysctl Kernel Parameters Why sysctl Matters Problem: Default kernel parameters are conservative (fit laptops, embedded systems) Solution: Tune for your workload (databases, web servers, HPC) Trade-off: More throughput vs. latency / memory vs. stability …

Executive Summary Linux is a layered system: from firmware through kernel subsystems to containerized applications. Understanding these layers—and their interdependencies—is critical for reliable, secure, performant infrastructure. This guide covers: Layered architecture (firmware → kernel → userspace → containers) Core subsystems: process scheduling, memory, filesystems, networking systemd: unit management and service lifecycle Production best practices: security, reliability, performance, observability Note: For detailed boot flow and debugging, see the Linux Boot Flow & Debugging guide. …

Executive Summary Reliability means predictable, auditable behavior. This guide covers: Time sync: Chrony for clock accuracy (critical for logging, security) Networking: Stable interface names & hostnames (infrastructure consistency) Logging: Persistent journald + logrotate (audit trail + disk management) Shutdown: Clean hooks to prevent data loss Patching: Kernel updates with livepatch (zero-downtime), tested rollback 1. Time Synchronization (chrony) Why Time Matters Critical for: Logging: Accurate timestamps for debugging, compliance audits Security: TLS cert validation, Kerberos, API token expiry Distributed systems: Causality ordering (happens-before relationships) Monitoring: Alert timing, metric correlation Cost of poor time sync: …

Executive Summary A security baseline is the foundation: OS-hardened, patched, with restricted access and audit trails. This guide covers minimal-install servers with hardened SSH, firewall (default-deny), LSM enforcement, least-privilege sudo, audit logging, and systemd hardening. Goal: Reduce attack surface, detect breaches, and enforce privilege boundaries. 1. Minimal Install & Patching Minimal Install What it is: Install only required packages (base + SSH + monitoring agent) No GUI, X11, unnecessary daemons Reduces vulnerabilities (fewer packages = fewer CVEs) Install steps (Ubuntu/Debian): …

Executive Summary Storage strategy = reliable data access with recovery guarantees. Choose based on workload (traditional vs. modern, single vs. multi-server). Why storage matters: Most production disasters involve storage—disk fills up and crashes your application, a database corruption loses customer data, or backups fail during a restore attempt. Proper storage management prevents these scenarios. Real-world disasters prevented by good storage: Disk full at 3 AM: Application logs fill /var → server crashes → customers can’t access site. Solution: Separate /var/log partition with monitoring. Failed database restore: Backup runs nightly for 2 years, never tested. Server dies, restore fails (corrupt backup). Solution: Monthly test restores. Database grows unexpectedly: 50GB database grows to 500GB in 3 months. Traditional partitions = downtime to resize. Solution: LVM allows online growth. Ransomware encrypts production data: No snapshots available, last backup is 24 hours old. Solution: ZFS/Btrfs snapshots provide instant point-in-time recovery. This guide covers: …