Executive Summary

Linux boot is a multi-stage handoff: UEFI β†’ Bootloader β†’ Kernel β†’ systemd β†’ Targets β†’ Units. Each stage has failure points. This guide shows the sequence, where failures occur, and how to capture logs.

Why understanding boot flow matters:

When a Linux server won’t boot, you need to know WHICH stage failed to fix it effectively. A black screen could mean anything from bad hardware to a typo in /etc/fstab.

Real-world scenario - The 3 AM boot failure:

02:00 AM - You apply a kernel update, run `apt upgrade`
02:05 AM - Reboot server: `sudo reboot`
02:07 AM - Server doesn't come back up
02:10 AM - Console shows: "Kernel panic - not syncing: VFS: Unable to mount root fs"
02:15 AM - You're panicking. What broke? Where are the logs?

What went wrong: The new kernel’s initramfs was missing LVM drivers. The system could boot into the kernel, but couldn’t find the root filesystem on /dev/mapper/vg0-root.

How boot flow knowledge saved the day:

  • Recognized stage: Panic during Stage 4 (Initramfs β†’ Root mount)
  • Recovery: Booted old kernel from GRUB menu (still available)
  • Fix: Regenerated initramfs: update-initramfs -u
  • Outcome: Server back online in 10 minutes instead of hours

This guide teaches you:

  • How to identify WHICH boot stage failed
  • Where to find logs for each stage
  • How to recover from common failures
  • Which kernel parameters to use for debugging

Key log sources:

  • dmesg / journalctl -b: Kernel ring buffer (first boot)
  • journalctl -b -p err: Error-level messages only
  • /var/log/boot.log: Deprecated, use journalctl instead
  • /proc/cmdline: Kernel command-line parameters passed at boot

Boot Sequence with Failure Points

Visual Guide: What You See on Screen During Boot

Understanding boot stages by what you see on the console helps you quickly identify where the failure occurred.

Stage 1 - UEFI/BIOS (0-5 seconds):

What you see:
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚  Manufacturer logo (Dell, HP, etc.)  β”‚
β”‚  "Press F2 for BIOS, F12 for Boot"  β”‚
β”‚  Memory test: 32GB OK               β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

Stuck here? β†’ Hardware failure (RAM, CPU, disk not detected)

Stage 2 - GRUB Bootloader (5-10 seconds):

What you see:
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚  GNU GRUB version 2.04              β”‚
β”‚                                     β”‚
β”‚  > Ubuntu                           β”‚
β”‚    Advanced options for Ubuntu      β”‚
β”‚    Memory test                      β”‚
β”‚                                     β”‚
β”‚  Use ↑↓ to select, Enter to boot   β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

Stuck here? β†’ GRUB config corrupted, missing /boot partition
Black screen? β†’ Video driver issue (try 'e' to edit, add 'nomodeset')

Stage 3 - Kernel Loading (10-15 seconds):

What you see:
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ [    0.000000] Linux version 5.15.0 β”‚
β”‚ [    0.123456] CPU: Intel Core i7   β”‚
β”‚ [    0.234567] Memory: 32GB         β”‚
β”‚ [    1.345678] PCI: Scanning bus    β”‚
β”‚ [    2.456789] EXT4-fs: mounting... β”‚
β”‚ ... (scrolling kernel messages) ... β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

Stuck here? β†’ "Kernel panic" = wrong kernel, missing drivers
Numbers [X.XXXXX] = timestamp in seconds since boot

Stage 4 - Initramfs / Root Mount (15-20 seconds):

What you see:
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ Loading initial ramdisk...          β”‚
β”‚ [    3.123456] md: RAID autodetect  β”‚
β”‚ [    3.234567] LVM: Scanning VGs    β”‚
β”‚ [    3.345678] EXT4: mounting /dev..β”‚
β”‚ Please unlock disk sda5_crypt:      β”‚
β”‚ Password: ****                      β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

Stuck here? β†’ Missing LVM/RAID drivers, wrong root UUID, LUKS password wrong
"VFS: Unable to mount root" β†’ Root device not found

Stage 5 - systemd Starting (20-30 seconds):

What you see:
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ [ OK ] Started System Logging       β”‚
β”‚ [ OK ] Reached target Local FS      β”‚
β”‚ [FAILED] Failed to start nginx      β”‚
β”‚        Starting OpenSSH daemon...   β”‚
β”‚ [ OK ] Started OpenSSH daemon       β”‚
β”‚ [ OK ] Reached target Multi-User    β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

Stuck here? β†’ Service dependency loop, fstab error
[FAILED] = Service crashed, check journalctl -u SERVICE

Stage 6 - Login Prompt (30+ seconds):

What you see:
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ Ubuntu 22.04 LTS server1 tty1       β”‚
β”‚                                     β”‚
β”‚ server1 login: _                    β”‚
β”‚                                     β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜

Success! Boot complete.
If SSH not working β†’ Check sshd.service with journalctl

Detailed Flowchart

β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ STAGE 1: UEFI/BIOS (Firmware)                                   β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ β€’ Power-on: CPU cold reset                                       β”‚
β”‚ β€’ POST (Power-on Self-Test): Verify hardware                    β”‚
β”‚ β€’ Scan boot devices (HDD, USB, network)                         β”‚
β”‚ β€’ Load MBR/UEFI boot sector                                     β”‚
β”‚                                                                 β”‚
β”‚ ⚠️  FAILURE POINTS:                                             β”‚
β”‚   - CPU/RAM/Chipset failure β†’ hang at POST                     β”‚
β”‚   - No bootable device found β†’ "No bootable media"             β”‚
β”‚   - UEFI Secure Boot: bootloader not signed β†’ access denied    β”‚
β”‚   - CMOS checksum fail β†’ boot with defaults                    β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                            ↓
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ STAGE 2: BOOTLOADER (GRUB 2 or systemd-boot)                   β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚                                                                 β”‚
β”‚ GRUB Stage 1 (512 bytes in MBR):                               β”‚
β”‚   β€’ Set CPU mode (real mode β†’ protected mode)                  β”‚
β”‚   β€’ Locate GRUB stage 2 on /boot partition                     β”‚
β”‚                                                                 β”‚
β”‚ GRUB Stage 2 (/boot/grub/i386-pc/core.img):                   β”‚
β”‚   β€’ Initialized drivers (ext4, RAID, LVM, LUKS)               β”‚
β”‚   β€’ Read /boot/grub/grub.cfg                                   β”‚
β”‚   β€’ Display menu β†’ User selects kernel                         β”‚
β”‚                                                                 β”‚
β”‚ ⚠️  FAILURE POINTS:                                             β”‚
β”‚   - /boot partition missing β†’ "Error: no such device"          β”‚
β”‚   - grub.cfg corrupted β†’ hang at menu or "no suitable video"  β”‚
β”‚   - LUKS encrypted /boot: wrong password β†’ access denied       β”‚
β”‚   - Missing GRUB modules β†’ "error: file not found"             β”‚
β”‚   - LVM/MD misconfigured β†’ can't find /boot                    β”‚
β”‚                                                                 β”‚
β”‚ πŸ“ LOG SOURCE: GRUB messages on console (or video memory)      β”‚
β”‚    No journalctl available yet (kernel not loaded)             β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                            ↓
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ STAGE 3: Kernel Load + Early Boot                              β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚                                                                 β”‚
β”‚ GRUB loads:                                                     β”‚
β”‚   1. Kernel (vmlinuz) into RAM at 0x100000                    β”‚
β”‚   2. Initramfs (initrd) β†’ temporary root filesystem            β”‚
β”‚   3. Passes kernel command-line parameters                     β”‚
β”‚                                                                 β”‚
β”‚ Kernel (start_kernel):                                         β”‚
β”‚   β€’ CPU exception handlers, interrupts                         β”‚
β”‚   β€’ Timer interrupts, scheduler initialization                 β”‚
β”‚   β€’ Memory allocator (buddy system)                            β”‚
β”‚   β€’ VFS, filesystems                                           β”‚
β”‚   β€’ PCI bus scan, device drivers                               β”‚
β”‚   β€’ Network stack                                              β”‚
β”‚                                                                 β”‚
β”‚ ⚠️  FAILURE POINTS:                                             β”‚
β”‚   - Bad kernel: "Kernel panic - not syncing"                   β”‚
β”‚   - Missing initramfs β†’ "Unable to mount root fs"              β”‚
β”‚   - CPU features missing β†’ illegal instruction / kernel panic  β”‚
β”‚   - Device init timeout β†’ "timed out waiting for device"       β”‚
β”‚   - Segfault in kernel code (rare)                             β”‚
β”‚                                                                 β”‚
β”‚ πŸ“ LOG SOURCE: dmesg / journalctl -b (kernel messages)         β”‚
β”‚    [    0.000000] Linux version 5.15.0 (gcc) #1 SMP Fri...   β”‚
β”‚    [    0.123456] clocksource: hpet: mask...                  β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                            ↓
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ STAGE 4: Initramfs (Initial RAM Filesystem)                    β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚                                                                 β”‚
β”‚ Purpose: Load drivers for root filesystem before mounting      β”‚
β”‚                                                                 β”‚
β”‚ Actions:                                                        β”‚
β”‚   1. Mount tmpfs as root filesystem (read-only)               β”‚
β”‚   2. Load filesystem drivers (ext4, xfs, btrfs, etc.)         β”‚
β”‚   3. Load device drivers (RAID, LVM, LUKS, NVMe)              β”‚
β”‚   4. Scan for root device                                      β”‚
β”‚   5. If encrypted: prompt for LUKS passphrase                 β”‚
β”‚   6. Mount actual root filesystem                              β”‚
β”‚   7. Find /sbin/init (systemd binary) on root fs              β”‚
β”‚   8. Execute /sbin/init (hand off to PID 1)                   β”‚
β”‚                                                                 β”‚
β”‚ ⚠️  FAILURE POINTS:                                             β”‚
β”‚   - Wrong root device (rootfs UUID/label not found)            β”‚
β”‚   - LUKS error: wrong passphrase β†’ prompt loops               β”‚
β”‚   - RAID degraded: "md0: not ready"                            β”‚
β”‚   - LVM: missing physical volumes β†’ timeout                    β”‚
β”‚   - Filesystem corruption: "unexpected inconsistency" (fsck)  β”‚
β”‚   - /sbin/init not found β†’ "No such file or directory"        β”‚
β”‚                                                                 β”‚
β”‚ πŸ“ LOG SOURCE: initramfs messages on console (before handoff)  β”‚
β”‚    Captured in journalctl if fsck/mount logs enabled          β”‚
β”‚    /proc/cmdline shows root device parameter                  β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                            ↓
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ STAGE 5: PID 1 (systemd Initialization)                         β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚                                                                 β”‚
β”‚ systemd starts:                                                 β”‚
β”‚   1. Parse /etc/systemd/system.conf                            β”‚
β”‚   2. Set up journald (logging)                                 β”‚
β”‚   3. Parse /etc/fstab β†’ create mount units                    β”‚
β”‚   4. Parse /etc/systemd/system/*.target.wants                 β”‚
β”‚   5. Determine default target (multi-user.target, etc.)       β”‚
β”‚   6. Activate dependency-ordered units:                        β”‚
β”‚       β”œβ”€ Local filesystem mounts (/var, /home, etc.)          β”‚
β”‚       β”œβ”€ Network targets (if needed)                           β”‚
β”‚       β”œβ”€ Services (sshd, nginx, etc.)                          β”‚
β”‚       └─ Targets reach active state                            β”‚
β”‚                                                                 β”‚
β”‚ ⚠️  FAILURE POINTS:                                             β”‚
β”‚   - /etc/systemd/system.conf syntax error β†’ boot hangs        β”‚
β”‚   - fstab invalid: wrong device/mount point β†’ emergency shell β”‚
β”‚   - Missing service dependency β†’ timeout, then skip           β”‚
β”‚   - Service ExecStart fails β†’ service marked failed            β”‚
β”‚   - SELinux/AppArmor denials β†’ service blocked                β”‚
β”‚   - cgroup limits hit early β†’ OOM killer (rare)               β”‚
β”‚                                                                 β”‚
β”‚ πŸ“ LOG SOURCE: journalctl -b (systemd messages)                β”‚
β”‚    systemd[1]: Started System Logging Service.                β”‚
β”‚    systemd[1]: Reached target Local File Systems.             β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                            ↓
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ STAGE 6: Targets & Units (Service Startup)                     β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚                                                                 β”‚
β”‚ Default target brings in units:                                β”‚
β”‚   multi-user.target:                                           β”‚
β”‚     β†’ [email protected] (login prompt)                        β”‚
β”‚     β†’ sshd.service (SSH daemon)                                β”‚
β”‚     β†’ nginx.service (web server)                               β”‚
β”‚     β†’ (all Wants= dependencies)                                β”‚
β”‚                                                                 β”‚
β”‚ systemd respects:                                              β”‚
β”‚   - ExecStart: command to run                                  β”‚
β”‚   - Type: oneshot, simple, forking, notify, dbus              β”‚
β”‚   - Restart: on-failure, always, no                            β”‚
β”‚   - RestartSec: delay between restarts                         β”‚
β”‚   - TimeoutStartSec: max time to reach active (default 90s)   β”‚
β”‚                                                                 β”‚
β”‚ ⚠️  FAILURE POINTS:                                             β”‚
β”‚   - Service ExecStart exits non-zero β†’ marked failed           β”‚
β”‚   - ExecStart timeout (>90s) β†’ killed by systemd              β”‚
β”‚   - Circular dependency: A wants B, B wants A β†’ timeout        β”‚
β”‚   - Service crashes immediately β†’ infinite restart loop       β”‚
β”‚   - RestartSec too long β†’ apparent hang                        β”‚
β”‚                                                                 β”‚
β”‚ πŸ“ LOG SOURCE: journalctl -u SERVICE.service (per-unit logs)   β”‚
β”‚    systemd[1]: sshd.service: Main process exited...           β”‚
β”‚    systemd[1]: sshd.service: Failed with result...            β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                            ↓
                    βœ… LOGIN PROMPT
                   (Boot Complete)

Log Capture at Each Stage

Stage 1-2: UEFI/BIOS β†’ GRUB

No log saved yet (firmware only). Observe on console:

  • UEFI splash screen
  • GRUB menu with kernel options
  • Any error messages before kernel loads

Recovery: Enter GRUB edit mode (press e), verify linux line has correct root=UUID=... and ro flags.


Stage 3-4: Kernel + Initramfs

Captured in: dmesg, journalctl -b

# View kernel messages from current boot
dmesg | head -50

# View only errors
dmesg | grep -i error

# View kernel ring buffer with timestamps
dmesg -T | tail -100

# View boot messages (entire boot)
journalctl -b

# View just kernel messages
journalctl -b -k

# View last 3 boots
journalctl --list-boots

# View specific boot (-1 = previous boot)
journalctl -b -1

Example output:

[    0.000000] Linux version 5.15.0-73-generic on x86_64
[    0.000000] Command line: BOOT_IMAGE=/boot/vmlinuz-5.15.0-73-generic root=UUID=a1b2c3d4 ro
[    0.123456] Memory: 31456M available (14337K kernel code, 2750K rwdata, 4648K rodata...)
[    5.234567] EXT4-fs (sda1): mounted filesystem with ordered data mode
[ 6.345678] systemd[1]: systemd 245.4-4ubuntu3.13 running in system mode

What to look for:

  • Lines with [ X.XXXXXX]: boot stage timing
  • Messages containing “error”, “failed”, “panic”
  • Timeout messages: “timed out waiting for device”
  • Device initialization: “EXT4-fs”, “LVM”, “RAID”

Stage 5-6: systemd β†’ Services

Captured in: journalctl -b (unit messages)

# View all systemd messages from boot
journalctl -b

# View errors only
journalctl -b -p err

# View specific service
journalctl -u sshd.service

# View service with last 50 lines
journalctl -u sshd.service -n 50

# Real-time logs (like tail -f)
journalctl -u sshd.service -f

# View boot sequence timeline
systemd-analyze

# View service startup order
systemd-analyze plot > boot-timeline.svg

Example output:

Oct 16 12:34:56 ubuntu systemd[1]: Starting Cleanup of Temporary Directories...
Oct 16 12:34:56 ubuntu systemd[1]: Started Cleanup of Temporary Directories.
Oct 16 12:34:56 ubuntu systemd[1]: Reached target Local File Systems.
Oct 16 12:34:57 ubuntu sshd[1234]: Server listening on 0.0.0.0 port 22.
Oct 16 12:34:57 ubuntu systemd[1]: Started OpenSSH server daemon.
Oct 16 12:34:57 ubuntu systemd[1]: Reached target Multi-User System.
Oct 16 12:34:57 ubuntu systemd[1]: Startup finished in 1.234s (kernel) + 0.567s (userspace) = 1.801s total.

Common Boot Failures & Recovery

How to use this section: Match what you see on screen to the failure below, then follow the recovery steps.

Failure 1: “Unable to mount root filesystem”

What you see on screen:

[    3.456789] VFS: Cannot open root device "UUID=abc123..." or unknown-block(0,0)
[    3.456790] Please append a correct "root=" boot option; here are the available partitions:
[    3.456791] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)

When this happens:

  • After kernel update (initramfs not regenerated)
  • After moving disk to new machine (UUID changed)
  • After resizing partitions (UUID changed)
  • After converting traditional partition β†’ LVM (root device path changed)

Cause: Root device UUID/label not found, or initramfs missing drivers.

Why it happens: The kernel loads successfully, but initramfs can’t find the root filesystem. This is like starting your car but not knowing where home isβ€”the engine works, but you can’t get there.

Recovery:

# 1. Boot into GRUB edit mode (press 'e' at menu)
# 2. Find the 'linux' line, note the root=UUID=...
# 3. Boot into recovery: Add 'systemd.unit=emergency.target' to kernel line
# 4. Once in emergency shell:

# List available block devices
lsblk

# Check if correct device is there
ls -la /dev/sd*

# Mount manually (if you know the device)
mount /dev/sda1 /mnt

# Check /etc/fstab for mount points
cat /mnt/etc/fstab

# If UUID is wrong, fix it
blkid /dev/sda1  # Find actual UUID
# Edit grub.cfg or /etc/default/grub, regenerate GRUB config
sudo grub-mkconfig -o /boot/grub/grub.cfg

Failure 2: “Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)”

What you see on screen:

[    2.123456] VFS: Unable to mount root fs on unknown-block(0,0)
[    2.123457] Kernel panic - not syncing: VFS: Unable to mount root fs
[    2.123458] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 5.15.0-custom
[    2.123459] ---[ end Kernel panic - not syncing: VFS: Unable to mount root fs ]---

When this happens:

  • After compiling custom kernel without filesystem support
  • After booting kernel meant for different filesystem type
  • After filesystem corruption (rare, usually shows different error)

Cause: Kernel compiled without required filesystem driver (ext4, xfs, etc.).

Why it happens: You built a custom kernel or are using a minimal kernel that doesn’t have the ext4/xfs/btrfs driver compiled in. It’s like trying to open a PDF without PDF reader softwareβ€”the kernel doesn’t know how to read the filesystem format.

Recovery:

# 1. Boot from live USB (ubuntu-desktop or similar)
# 2. Check actual root filesystem type
sudo blkid /dev/sda1
# output: /dev/sda1: UUID="abc123" TYPE="ext4"

# 3. If you have the kernel source, rebuild with ext4 enabled:
make menuconfig
# Search for ext4: File systems > Ext4 filesystem support
# Mark as built-in (not module): [*] not [M]

# Or use a pre-built kernel that supports ext4:
sudo apt install linux-image-generic

Failure 3: “systemd[1]: Reached target Shutdown, Rebooting.” (Boot Loop)

What you see on screen:

[ OK ] Started System Logging Service
[ OK ] Reached target Local File Systems
       Starting Network Service...
[   **] A start job is running for Network (1min 30s / no limit)
[   **] A start job is running for Network (2min / no limit)
[FAILED] Failed to start Network Service
       Stopping System Logging Service...
[ OK ] Stopped System Logging Service
[ OK ] Reached target Shutdown
       Rebooting.

(Then boots again and repeats)

When this happens:

  • After editing /etc/fstab with wrong entry (missing disk, wrong UUID)
  • After enabling service with misconfigured dependencies
  • After network service requires interface that doesn’t exist
  • After circular dependency (Service A requires B, B requires A)

Cause: Boot continues but target never reached, systemd timeout triggers reboot.

Why it happens: systemd waits for a critical service to start (like mounting /var from fstab), times out after 90+ seconds, gives up and reboots. It’s like waiting for a friend who never shows upβ€”eventually you leave and try again.

Recovery:

# 1. Boot with 'systemd.unit=rescue.target' to skip services
# 2. Once in rescue shell:

# Check which service is hanging
systemctl list-units --state=failed

# View service status
systemctl status problematic.service

# View service logs
journalctl -u problematic.service

# Disable problematic service temporarily
sudo systemctl disable problematic.service

# Reboot normally
sudo systemctl reboot

Quick Diagnosis Flowchart

Use this to quickly identify which stage failed:

Boot starts β†’ Nothing on screen at all
    ↓
β”œβ”€ No power, no fans spinning β†’ Power supply / hardware failure
β”œβ”€ Fans spin, no video β†’ GPU failure, try different monitor/cable
└─ Logo appears then black β†’ Continue below

Manufacturer logo appears (Dell, HP, etc.)
    ↓
β”œβ”€ Stuck at logo β†’ Press F2/Del, check BIOS sees disk
└─ Logo passes β†’ Continue below

GRUB menu appears
    ↓
β”œβ”€ Can't select kernel β†’ Keyboard not working (USB issue, try PS/2)
β”œβ”€ Black screen after selection β†’ Video driver (try 'e', add nomodeset)
β”œβ”€ "Error: file not found" β†’ Missing /boot files (USB boot, restore /boot)
└─ Kernel loads β†’ Continue below

Scrolling kernel messages [ 0.123456 ]
    ↓
β”œβ”€ "Kernel panic" appears β†’ Wrong kernel or missing driver (boot old kernel)
β”œβ”€ Stops at "VFS: Unable to mount" β†’ Root filesystem not found (check UUID)
β”œβ”€ "Please unlock disk" appears β†’ LUKS encryption (enter password)
└─ Messages continue β†’ Continue below

systemd messages: [ OK ] / [FAILED]
    ↓
β”œβ”€ [FAILED] immediately reboots β†’ fstab error or critical service (rescue.target)
β”œβ”€ Stops at "A start job is running..." β†’ Service timeout (Ctrl+Alt+Del, rescue.target)
β”œβ”€ [FAILED] but continues β†’ Non-critical service (journalctl -u SERVICE)
└─ [ OK ] Reached target Multi-User β†’ Continue below

Login prompt appears β†’ SUCCESS!
    ↓
β”œβ”€ Can't login β†’ Check username/password, caps lock
β”œβ”€ SSH not working β†’ journalctl -u sshd, check firewall
└─ Everything works β†’ Congratulations!

Kernel Parameters: Boot-Time Control

Common Parameters

# View current kernel parameters
cat /proc/cmdline
# output: BOOT_IMAGE=/boot/vmlinuz-5.15.0-73 root=UUID=a1b2c3d4 ro quiet splash

# Edit parameters (temporarily, for this boot):
# 1. GRUB menu β†’ press 'e'
# 2. Find 'linux' line, add parameter
# 3. Press Ctrl+X to boot

# Edit permanently:
sudo vi /etc/default/grub
# Add parameters to GRUB_CMDLINE_LINUX
# Example:
# GRUB_CMDLINE_LINUX="root=/dev/sda1 ro quiet splash systemd.log_level=debug"

# Regenerate grub.cfg
sudo grub-mkconfig -o /boot/grub/grub.cfg

Useful Parameters for Debugging

ParameterEffect
systemd.log_level=debugVerbose systemd messages (replaces quiet)
systemd.unit=emergency.targetSkip all services, boot to emergency shell
systemd.unit=rescue.targetBoot with minimal services (single-user mode)
roMount root read-only (fsck-safe)
rwMount root read-write (faster, but risky if fsck pending)
console=ttyS0Serial console output (for headless systems)
nosplashDisable splash screen, show kernel messages
nomodesetDisable GPU mode-setting (useful if GPU driver broken)
noapicDisable APIC (some old hardware)
quietSuppress kernel messages (default)

Mermaid Sequence Diagram

sequenceDiagram
    participant UEFI as UEFI/BIOS
    participant GRUB as GRUB 2
    participant Kernel as Kernel
    participant Initramfs as Initramfs
    participant Systemd as systemd (PID 1)
    participant Services as Services/Targets
    participant User as User Login

    UEFI->>UEFI: POST, scan devices
    UEFI->>GRUB: Transfer control (MBR/UEFI boot)
    
    activate GRUB
    GRUB->>GRUB: Load core.img, read grub.cfg
    GRUB->>GRUB: Display menu, wait for selection
    GRUB->>Kernel: Load vmlinuz + initramfs
    GRUB->>GRUB: Pass kernel parameters
    deactivate GRUB

    activate Kernel
    Kernel->>Kernel: start_kernel()<br/>Setup CPU, interrupts
    Kernel->>Kernel: Initialize subsystems<br/>(sched, mm, vfs, net)
    Kernel->>Initramfs: Mount initramfs as root
    Kernel->>Initramfs: Call /sbin/init
    deactivate Kernel

    activate Initramfs
    Initramfs->>Initramfs: Load drivers (ext4, raid, luks)
    Initramfs->>Initramfs: Scan for root device
    Initramfs->>Initramfs: Mount real root filesystem
    Initramfs->>Systemd: Exec systemd (PID 1)
    deactivate Initramfs

    activate Systemd
    Systemd->>Systemd: Parse system.conf<br/>Setup journald
    Systemd->>Systemd: Parse /etc/fstab
    Systemd->>Systemd: Select default target<br/>(multi-user.target)
    Systemd->>Systemd: Sort dependencies
    deactivate Systemd

    activate Services
    Services->>Services: Activate units in order<br/>Local filesystems
    Services->>Services: Network targets (if needed)
    Services->>Services: Start services (sshd, nginx, etc.)
    Services->>Services: Reach target state
    deactivate Services

    Services->>User: Display login prompt

Checklists

Pre-Boot Verification

  • /boot partition has space (df -h /boot)
  • GRUB config is valid (grub-mkconfig -o /tmp/test.cfg)
  • Kernel version matches initramfs
  • /etc/fstab has correct UUIDs (blkid vs fstab)
  • root filesystem driver compiled in kernel (not module)
  • Encryption (LUKS) keys stored securely
  • RAID/LVM metadata on all devices (if using)
  • SELinux/AppArmor not blocking boot-critical services

Boot Failure Diagnosis

  • Check kernel messages: dmesg | grep -i error
  • Check journalctl: journalctl -b -p err
  • Check specific service: systemctl status SERVICE
  • Check boot timeline: systemd-analyze plot
  • Verify root device: /proc/cmdline vs blkid
  • Check initramfs: lsinitramfs /boot/initrd.img | grep -i ext4
  • Boot into emergency.target to rule out service issues
  • Check dmesg for “Unable to mount”, “panic”, “timeout”

FAQ

Q: My system reboots immediately after reaching multi-user.target. Why? A: Likely a service in a restart loop with RestartSec. Check journalctl -xe for failed services, then systemctl status SERVICE to see why it’s restarting.

Q: How do I boot without loading fstab mounts? A: Add systemd.unit=emergency.target to kernel parameters, then manually mount what you need: mount /dev/sda1 /mnt.

Q: Why is boot taking 90+ seconds? A: Use systemd-analyze critical-chain to find the slowest unit. Often it’s a service waiting for a device or network. Check journalctl -b | grep -i timeout.

Q: How do I access the kernel ring buffer if X fails? A: Boot with systemd.unit=rescue.target (single-user mode), then dmesg | tail -100 or journalctl -b.


Further Reading