Linux boot process and management
Grub and it’s pal Dracut
For FredLug September 2016
By Peter Larsen
Agenda
● Boot proces overview
– Bios
– UEFI
● Grub fundamentals
● Initramfs fundamentals
● Diagnosis
● Demos – how to fix a broken system
Boot process overview
● Bios
● UEFI
● Kernel
● Initrd
● Systemd
Boot: BIOS
Source: http://resources.infosecinstitute.com/uefi-and-tpm/
● Traditional/Old. 16Bit, 1MB
addressable space, limited
boot partition size
● “Blind” - easy to inject bad
code
● No real standards
● Very hardware
manufactorer dependent
● Basic UI
Boot: UEFI
● Unified Extensible Firmware Interface
● Trusted Boot
Source: http://www.antapex.org/diskdevices.htm
Bios/UEFI setup
● Pressing a key during boot (DEL, F1, F2 etc – depends on BIOS)
● # dmidecode
Show content of bios settings and discovered hardware
● # lspci
List hardware found on the PCI Bus
● # lsusb
List discovered USB devices
● # lscpu
List discovered CPUs and settings
● # lsscsi
List SCSI devices (most drives today show up as SCSI devices)
● Hardware vendors sometimes provide executables to change BIOS
settings.
Key BIOS/UEFI settings
● Boot Device / Boot Device Order
● ACPI – Advanced Configuration and Power
Interface
● CPU features – important for virtualization
● USB boot enabled/disabled
● UEFI mode – in this mode, each boot device
must be registered to be available. In secure
mode, they must be signed to be available.
Boot: Kernel
● Kernel is loaded into memory and executed
● Bulk of kernel is compressed and will be
uncompressed into specific memory locations.
● Memory and other structures are initialized
● Kernel modules are initialized – PCI and other busses
are found, initialized etc.
● Mounts root device (kernel parameter)
● Executes PID 1 /init -> systemd
● First systemd process is located on initramfs.
Kernel implementation
● Most people do not compile their own kernels
● Kernels are installed via rpm or deb packages
● Located in /boot
● Fedora/CentOS/RHEL puts /boot on a primary
partition of at least 512MB in size. Contains the
current and 2 other kernels.
● /boot is NOT a “bootable” partition in terms of
MBR (Microsoft terminology). The boot flag is
ignored by linux.
Kernel options
● Too many to list here – we’ll cover some in the other sections
● Install kernel-doc for /usr/share/doc/kernel* documentation
● https://www.kernel.org/doc/Documentation/kernel-parameters.txt
● Use Grub to change/set parameters – permanent changes goes
into grub.cfg
● Example (one long line):
kernel /vmlinuz-3.10.0-327.36.1.el7.x86_64
root=/dev/mapper/test-root ro crashkernel=auto
rd.lvm.lv=test/root rd.lvm.lv=test/swap rhgb quiet
LANG=en_US.UTF-8
Boot: Initramfs
● Dynamic kernel module initializer
● Ram disk image used to “kickstart” a system
● The kernel is very large, but would be enourmous if it
contained all device code statically.
● Instead the kernel loads modules based on devices
identified during startup.
● Since the kernel doesn’t have all devices needed for
boot, initramfs is required to initialize the right
modules, so the root device can be activated/read.
GRUB: GRand Unified Bootloader
● Legacy Grub is no more
● Grub2 Staged boot
● Loads Kernel
● Mounts ram disk (initramfs)
GRUB Boot process
● POST – Stage 1/1.5
– BIOS
● Finds bootable device and transfers control to MBR
● GPT/MBR contains stage 1, very small. Fits in less than 512bytes
● GRUB stage 1.5 is located in the first 30 KB of boot device immediately
following MBR.
● Stage 1.5 does better device discovery, supports larger partitions etc. and
passes control to Stage 2
– UEFI
● Loads UEFI boot system, mounts EFI partition, loads GRUB stage 2
● Stage 2
– Read Grub.cfg and displays menu and other actions.
– Grub executes the commands in the boot menu
/boot and /boot/efi
Partition Table: gpt
Disk Flags:
Number Start End Size File system Name Flags
1 1049kB 211MB 210MB fat16 EFI System Partition boot
2 211MB 525MB 315MB xfs
3 525MB 9663MB 9137MB lvm
● EFI is only used when UEFI is enabled. Mounted on
/boot/efi
● /boot is partition 2 – notice NOT a boot partition
● The rest can be partitions, LVM etc. - your choice.
/boot structure
/boot
├── efi
  │ └── EFI
  │ ├── BOOT
  │ └── redhat
  │ └── fonts
└── grub2
└── themes
└── system
● /boot/efi only exists when EFI was detected during
boot
● Grub themes are rather cool.
Install plymouth-theme-change to get cool graphics
on your boot menu!
Grub configuration commands
● # grub2-mkconfig
Create configuration/menu by scanning system for bootable images.
Will detect Windows and lots of different Linux distros and boot loaders
● # grubby
Command line tool for configuring grub. Very advanced. Using grub2-
install is a lot easier if you are looking to fix a boot problem. Can be used
to manually add menu items and set configuration options.
● # grub2-install <device>
Install grub (boot) on device. Injects stage1 and 1.5 boot loaders and
makes them refer to the boot device.
DO NOT USE IF YOU’RE USING EFI!
● # efibootmgr
Manage boot setup of EFI. List existing systems, reorder, add new
system, delete systems etc.
Configuration grub.cfg
● Located in /boot/grub2/ or /boot/efi/EFI/redhat
● Created using grub2-mkconfig
● If manually editing, use /etc/grub.d to add files,
then run grub2-mkconfig to generate a new
configuration.
Example configuration
menuentry 'Red Hat Enterprise Linux Server (3.10.0-
327.36.1.el7.x86_64) 7.2 (Maipo)' --class red --class
gnu-linux --class gnu --class os --unrestricted
$menuentry_id_option 'gnulinux-3.10.0-327.el7.x86_64-
advanced-6e59c4d3-dbf6-4ba7-bd9e-7104482f66cd' {
load_video
set gfxpayload=keep
insmod gzio
insmod part_gpt
insmod xfs
linux /vmlinuz-3.10.0-327.36.1.el7.x86_64
root=/dev/mapper/test-root ro crashkernel=auto
rd.lvm.lv=test/root rd.lvm.lv=test/swap rhgb quiet
LANG=en_US.UTF-8
initrd /initramfs-3.10.0-327.36.1.el7.x86_64.img
}
Kernel: Linux begins!
● Fundamentals
– Processes
– Memory Management
– Device Management
– File systems
– Etc.
● Kernel starts PID 1 from root file system
Initramfs – managed by dracut
● Small root device loaded during boot right after
the kernel
● Created and managed using “dracut”
● Initramfs file located in /boot with kernel
● Must match kernel version
● “lsinitrd” shows/lists content of initramfs
● Extract all using zcat | cpio -i
Extract single file using “lsinitrd -f <file>”
Dracut
● Use modules to enable boot features
– Examples: lvm, mdraid, luks, iscsi etc.
● List all modules: dracut –list-modules
● If transposing to new hardware, you may need to
generate a new initramfs with modules fitting
the hardware
September 2016 Grub and Dracut
22
Customizing initramfs on boot
● LVM
– rd.lvm=0
disable LVM detection
– rd.lvm.vg=<volume group name>
only activate the volume groups with
the given name.
– rd.lvm.lv=<logical volume name>
only activate the logical volumes with
the given name.
– rd.lvm.conf=0
remove any /etc/lvm/lvm.conf, which
may exist in the initramfs
Diagnostics
● Grub Shell
● Initramfs Shell (RD)
● Emergency Shell
● Single User Mode (mode 1)
● MultiUser mode (non graphical) (mode 3)
● Graphical mode (mode 5)
Diagnostics: Grub Shell
● Configuration errors in grub.cfg is the most common
cause for going to the shell
● grub>
● The cfg file is nothing but a list of commands that the
grub-shell understands and executes.
● Basic commands to boot a system:
set root=’hd0,msdos1’
kernel /vmlinuz-<version> root=/dev/<root partition> ro
initrd /initramfs-<version>.img
boot
Diagnostics: Initramfs Shell (RD)
● Initramfs is very minimal and a bad shell to do recovery in. Use for
boot diagnostics, to see why a device isn’t found, responding etc.
● RD = Root Device
● Parameters are given on the kernel command line
– rd.break
Drop to shell
Options: cmdline, pre-udev, pre-mount, mount, cleanup
– rd.info
Print debug information
– rd.shell
Drop to shell if root mounting fails
– rd.udev.{info|debug}
Set udev to info/debug level in output
Diagnostics: Emergency Shell/Mode
● Starts minimal shell on a system where the root
device is mounted (ro).
● Full system command line options – but data
disks, network and other key features are not
enabled.
● Will prompt for root password!
● Kernel Parameter:
systemd.unit=emergency.target
Diagnostics: Systemd debug
● Systemd startup process has problems. Used to
debug services and other system management
features controlled by systemd.
● Advanced system diagnostics
● Kernel Parameter:
systemd.debug-shell
Diagnostics: Single User Mode (mode 1)
● Goes to shell after all core services are started.
Disks mounted, devices active etc. No network.
● Will prompt for root password!!
● Kernel Parameter:
systemd.target=rescue.target
Diagnostics: MultiUser mode (non graphical)
(mode 3)
● Full system running – no graphical sub
component. For most servers, this is the run
target for a normal running system.
● Network active
● Kernel parameter:
systemd.unit=multi-user.target
● Graphical subsystem does not start
● Primary services (web server, data volume) does
not start
Diagnostics: Install Disk/Recovery Boot
● Alternative: Use LiveUSB/CD
● Allows full access to root device, fsck of root device
and full recovery/repair
● Limited commands
– Full LVM
– Full File system support
– Bypass root password
● Potential selinux corruption – use with CARE!
– Always run “fixfiles onboot” if major changes are done to
files
Demo
● Show traditional BIOS boot system
● Show EFI system
● Recover boot failure
How to reset the root password
● The old tricks are no longer valid.
● Add rd.break to kernel command line
● Command line:
– # mount -o remount,rw /sysroot
– # chroot /sysroot
– # passwd
– # touch ./autorelabel
● Exit twice and system will reboot.
● Note, system will relabel ALL files on boot – and this will
take considerable time
Questions

Grub and dracut ii

  • 1.
    Linux boot processand management Grub and it’s pal Dracut For FredLug September 2016 By Peter Larsen
  • 2.
    Agenda ● Boot procesoverview – Bios – UEFI ● Grub fundamentals ● Initramfs fundamentals ● Diagnosis ● Demos – how to fix a broken system
  • 3.
    Boot process overview ●Bios ● UEFI ● Kernel ● Initrd ● Systemd
  • 4.
    Boot: BIOS Source: http://resources.infosecinstitute.com/uefi-and-tpm/ ●Traditional/Old. 16Bit, 1MB addressable space, limited boot partition size ● “Blind” - easy to inject bad code ● No real standards ● Very hardware manufactorer dependent ● Basic UI
  • 5.
    Boot: UEFI ● UnifiedExtensible Firmware Interface ● Trusted Boot Source: http://www.antapex.org/diskdevices.htm
  • 6.
    Bios/UEFI setup ● Pressinga key during boot (DEL, F1, F2 etc – depends on BIOS) ● # dmidecode Show content of bios settings and discovered hardware ● # lspci List hardware found on the PCI Bus ● # lsusb List discovered USB devices ● # lscpu List discovered CPUs and settings ● # lsscsi List SCSI devices (most drives today show up as SCSI devices) ● Hardware vendors sometimes provide executables to change BIOS settings.
  • 7.
    Key BIOS/UEFI settings ●Boot Device / Boot Device Order ● ACPI – Advanced Configuration and Power Interface ● CPU features – important for virtualization ● USB boot enabled/disabled ● UEFI mode – in this mode, each boot device must be registered to be available. In secure mode, they must be signed to be available.
  • 8.
    Boot: Kernel ● Kernelis loaded into memory and executed ● Bulk of kernel is compressed and will be uncompressed into specific memory locations. ● Memory and other structures are initialized ● Kernel modules are initialized – PCI and other busses are found, initialized etc. ● Mounts root device (kernel parameter) ● Executes PID 1 /init -> systemd ● First systemd process is located on initramfs.
  • 9.
    Kernel implementation ● Mostpeople do not compile their own kernels ● Kernels are installed via rpm or deb packages ● Located in /boot ● Fedora/CentOS/RHEL puts /boot on a primary partition of at least 512MB in size. Contains the current and 2 other kernels. ● /boot is NOT a “bootable” partition in terms of MBR (Microsoft terminology). The boot flag is ignored by linux.
  • 10.
    Kernel options ● Toomany to list here – we’ll cover some in the other sections ● Install kernel-doc for /usr/share/doc/kernel* documentation ● https://www.kernel.org/doc/Documentation/kernel-parameters.txt ● Use Grub to change/set parameters – permanent changes goes into grub.cfg ● Example (one long line): kernel /vmlinuz-3.10.0-327.36.1.el7.x86_64 root=/dev/mapper/test-root ro crashkernel=auto rd.lvm.lv=test/root rd.lvm.lv=test/swap rhgb quiet LANG=en_US.UTF-8
  • 11.
    Boot: Initramfs ● Dynamickernel module initializer ● Ram disk image used to “kickstart” a system ● The kernel is very large, but would be enourmous if it contained all device code statically. ● Instead the kernel loads modules based on devices identified during startup. ● Since the kernel doesn’t have all devices needed for boot, initramfs is required to initialize the right modules, so the root device can be activated/read.
  • 12.
    GRUB: GRand UnifiedBootloader ● Legacy Grub is no more ● Grub2 Staged boot ● Loads Kernel ● Mounts ram disk (initramfs)
  • 13.
    GRUB Boot process ●POST – Stage 1/1.5 – BIOS ● Finds bootable device and transfers control to MBR ● GPT/MBR contains stage 1, very small. Fits in less than 512bytes ● GRUB stage 1.5 is located in the first 30 KB of boot device immediately following MBR. ● Stage 1.5 does better device discovery, supports larger partitions etc. and passes control to Stage 2 – UEFI ● Loads UEFI boot system, mounts EFI partition, loads GRUB stage 2 ● Stage 2 – Read Grub.cfg and displays menu and other actions. – Grub executes the commands in the boot menu
  • 14.
    /boot and /boot/efi PartitionTable: gpt Disk Flags: Number Start End Size File system Name Flags 1 1049kB 211MB 210MB fat16 EFI System Partition boot 2 211MB 525MB 315MB xfs 3 525MB 9663MB 9137MB lvm ● EFI is only used when UEFI is enabled. Mounted on /boot/efi ● /boot is partition 2 – notice NOT a boot partition ● The rest can be partitions, LVM etc. - your choice.
  • 15.
    /boot structure /boot ├── efi   │└── EFI   │ ├── BOOT   │ └── redhat   │ └── fonts └── grub2 └── themes └── system ● /boot/efi only exists when EFI was detected during boot ● Grub themes are rather cool. Install plymouth-theme-change to get cool graphics on your boot menu!
  • 16.
    Grub configuration commands ●# grub2-mkconfig Create configuration/menu by scanning system for bootable images. Will detect Windows and lots of different Linux distros and boot loaders ● # grubby Command line tool for configuring grub. Very advanced. Using grub2- install is a lot easier if you are looking to fix a boot problem. Can be used to manually add menu items and set configuration options. ● # grub2-install <device> Install grub (boot) on device. Injects stage1 and 1.5 boot loaders and makes them refer to the boot device. DO NOT USE IF YOU’RE USING EFI! ● # efibootmgr Manage boot setup of EFI. List existing systems, reorder, add new system, delete systems etc.
  • 17.
    Configuration grub.cfg ● Locatedin /boot/grub2/ or /boot/efi/EFI/redhat ● Created using grub2-mkconfig ● If manually editing, use /etc/grub.d to add files, then run grub2-mkconfig to generate a new configuration.
  • 18.
    Example configuration menuentry 'RedHat Enterprise Linux Server (3.10.0- 327.36.1.el7.x86_64) 7.2 (Maipo)' --class red --class gnu-linux --class gnu --class os --unrestricted $menuentry_id_option 'gnulinux-3.10.0-327.el7.x86_64- advanced-6e59c4d3-dbf6-4ba7-bd9e-7104482f66cd' { load_video set gfxpayload=keep insmod gzio insmod part_gpt insmod xfs linux /vmlinuz-3.10.0-327.36.1.el7.x86_64 root=/dev/mapper/test-root ro crashkernel=auto rd.lvm.lv=test/root rd.lvm.lv=test/swap rhgb quiet LANG=en_US.UTF-8 initrd /initramfs-3.10.0-327.36.1.el7.x86_64.img }
  • 19.
    Kernel: Linux begins! ●Fundamentals – Processes – Memory Management – Device Management – File systems – Etc. ● Kernel starts PID 1 from root file system
  • 20.
    Initramfs – managedby dracut ● Small root device loaded during boot right after the kernel ● Created and managed using “dracut” ● Initramfs file located in /boot with kernel ● Must match kernel version ● “lsinitrd” shows/lists content of initramfs ● Extract all using zcat | cpio -i Extract single file using “lsinitrd -f <file>”
  • 21.
    Dracut ● Use modulesto enable boot features – Examples: lvm, mdraid, luks, iscsi etc. ● List all modules: dracut –list-modules ● If transposing to new hardware, you may need to generate a new initramfs with modules fitting the hardware
  • 22.
    September 2016 Gruband Dracut 22 Customizing initramfs on boot ● LVM – rd.lvm=0 disable LVM detection – rd.lvm.vg=<volume group name> only activate the volume groups with the given name. – rd.lvm.lv=<logical volume name> only activate the logical volumes with the given name. – rd.lvm.conf=0 remove any /etc/lvm/lvm.conf, which may exist in the initramfs
  • 23.
    Diagnostics ● Grub Shell ●Initramfs Shell (RD) ● Emergency Shell ● Single User Mode (mode 1) ● MultiUser mode (non graphical) (mode 3) ● Graphical mode (mode 5)
  • 24.
    Diagnostics: Grub Shell ●Configuration errors in grub.cfg is the most common cause for going to the shell ● grub> ● The cfg file is nothing but a list of commands that the grub-shell understands and executes. ● Basic commands to boot a system: set root=’hd0,msdos1’ kernel /vmlinuz-<version> root=/dev/<root partition> ro initrd /initramfs-<version>.img boot
  • 25.
    Diagnostics: Initramfs Shell(RD) ● Initramfs is very minimal and a bad shell to do recovery in. Use for boot diagnostics, to see why a device isn’t found, responding etc. ● RD = Root Device ● Parameters are given on the kernel command line – rd.break Drop to shell Options: cmdline, pre-udev, pre-mount, mount, cleanup – rd.info Print debug information – rd.shell Drop to shell if root mounting fails – rd.udev.{info|debug} Set udev to info/debug level in output
  • 26.
    Diagnostics: Emergency Shell/Mode ●Starts minimal shell on a system where the root device is mounted (ro). ● Full system command line options – but data disks, network and other key features are not enabled. ● Will prompt for root password! ● Kernel Parameter: systemd.unit=emergency.target
  • 27.
    Diagnostics: Systemd debug ●Systemd startup process has problems. Used to debug services and other system management features controlled by systemd. ● Advanced system diagnostics ● Kernel Parameter: systemd.debug-shell
  • 28.
    Diagnostics: Single UserMode (mode 1) ● Goes to shell after all core services are started. Disks mounted, devices active etc. No network. ● Will prompt for root password!! ● Kernel Parameter: systemd.target=rescue.target
  • 29.
    Diagnostics: MultiUser mode(non graphical) (mode 3) ● Full system running – no graphical sub component. For most servers, this is the run target for a normal running system. ● Network active ● Kernel parameter: systemd.unit=multi-user.target ● Graphical subsystem does not start ● Primary services (web server, data volume) does not start
  • 30.
    Diagnostics: Install Disk/RecoveryBoot ● Alternative: Use LiveUSB/CD ● Allows full access to root device, fsck of root device and full recovery/repair ● Limited commands – Full LVM – Full File system support – Bypass root password ● Potential selinux corruption – use with CARE! – Always run “fixfiles onboot” if major changes are done to files
  • 31.
    Demo ● Show traditionalBIOS boot system ● Show EFI system ● Recover boot failure
  • 32.
    How to resetthe root password ● The old tricks are no longer valid. ● Add rd.break to kernel command line ● Command line: – # mount -o remount,rw /sysroot – # chroot /sysroot – # passwd – # touch ./autorelabel ● Exit twice and system will reboot. ● Note, system will relabel ALL files on boot – and this will take considerable time
  • 33.