This document discusses troubleshooting, performance monitoring, and security best practices for Linux systems. It outlines methodologies for troubleshooting including monitoring logs and utilities, proactive and reactive maintenance, and documenting solutions. It describes resolving common hardware, software, and user interface problems. It also discusses using tools like sysstat, top, free, and vmstat to monitor performance, and securing Linux systems through access controls, firewall configuration, encryption, and SELinux.

1. LINUX SHELL SCRIPTING
SEC 210
Module 13
Troubleshooting,Performance,
and Security

2. OBJECTIVES
• Describe/outline good troubleshooting practices
• troubleshoot common hardware/software problems
• Monitor system performance via command-line
• Identify and fix common performance problems
• Describe the different facets of Linux security
• Increase the security of a Linux computer
• Measures & utilities to detect a security breach
2

3. 3
The
maintenance
cycle

4. TROUBLESHOOTING
METHODOLOGY
• Monitoring: observing log files and running
performance utilities system to identify problems
and their causes
• Proactive maintenance: minimizing chance of
future problems
• e.g., perform regular system backups
4

5. TROUBLESHOOTING
METHODOLOGY (CONT)
• Reactive maintenance: correcting problems when
they arise
• Documenting solutions
• Developing better proactive maintenance methods
• Documentation: system information stored in a
log book for future references
• All maintenance actions should be documented
• Troubleshooting procedures: tasks performed
when solving system problems
5

6. COMMON
TROUBLESHOOTI
NG
PROCEDURES
6

7. TROUBLESHOOTING
METHODOLOGY (CONTINUED)
• Two troubleshooting golden rules:
• Prioritize problems according to severity
• Spend reasonable amount of time on each problem
given its priority
• Ask for help if you can’t solve the problem
• Try to solve the root of the problem
• Avoid missing underlying cause
• Justify why a certain solution is successful
7

8. RESOLVING COMMON SYSTEM
PROBLEMS
• Three categories of problems:
• Hardware-related
• Software-related
• User interface-related
8

9. HARDWARE-RELATED
PROBLEMS
• Often involve improper hardware or software
configuration
• SCSI termination
• Video card and monitor configuration
• All hardware is on Hardware Compatibility List
• POST test alerts
• Loose hardware connections
• Problems specific to the type of hardware
• View output of dmesg command
• View content of /var/log/boot.log, /var/log/messages
9

10. HARDWARE-RELATED
PROBLEMS (CONTINUED)
• Absence of device drivers prevent OS from using
associated devices
• dmesg command: displays the hardware that is detected
by the Linux kernel
• lsusb command: displays a list of USB devices detected
by the Linux kernel
• lspci command: displays a list of PCI devices detected
by the Linux kernel
• Compare outputs of commands to output of lsmod to
determine if driver module is missing from kernel
10

11. DMESG | GREP SDA
- list all detected devices
11

12. DMESG | GREP –I MEMORY
• search for lines having string memory
12

13. WATCH "DMESG | TAIL -20"
• real time dmesg monitoring
13

14. SUDO LSHW –CLASS MEMORY
| LESS
• lshw classifies hardware using classes
14

15. LSPCI | LESS
• similar to dmesg | grep –i pci
15

16. LSMOD
• same as cat /proc/modules
• shows currently loaded kernel modules (LKMs)
• modules are pieces of code loaded into the kernel during boot (minus ext .o or
.ko… added with insmod)
• size of memory used by module
• used by: number of instances of module used
16
compare output
with dmesg, etc to
see if driver
module is missing

17. HARDWARE-RELATED
PROBLEMS (CONTINUED)
• Hardware failure can render a device unusable
• HDDs most common hardware components to fail
• If HDD containing partitions mounted on noncritical directories fails:
• Power down computer and replace failed HDD
• Boot Linux system
• Use fdisk (or LVM) to create partitions on new HDD
• Use mkfs to create filesystems
• Restore original data
• Ensure /etc/fstab has appropriate entries to mount filesystems
17

18. HARDWARE-RELATED
PROBLEMS (CONTINUED)
• If HDD containing / filesystem fails:
• Power down computer and replace failed HDD
• Reinstall Linux on new HDD
• Restore original configuration and data files
18

19. SOFTWARE-RELATED PROBLEMS:
APPLICATION-RELATED
PROBLEMS
• Missing program libraries/files, process
restrictions, or conflicting applications
• Dependencies: prerequisite shared libraries or
packages required for program execution
• Programs usually check at installation
• Package files may be removed accidentally
19

20. SOFTWARE-RELATED PROBLEMS:
APPLICATION-RELATED
PROBLEMS (CONTINUED)
• rpm –V command: identify missing files in a
package or package dependency
• dpkg –V bash (for ubuntu)
• ldd command: display shared libraries used by a
program
• ldconfig command: updates list of shared
library directories (/etc/ld.so.conf) and list of
shared libraries (/etc/ld.so.cache)
20

21. $LDD /BIN/BASH
• .so files are “shared object” files
• similar to Windows DLL
21

22. SOFTWARE-RELATED PROBLEMS:
APPLICATION-RELATED
PROBLEMS (CONTINUED)
• Too many running processes
• Solve by killing parent process of zombie processes
• Filehandles: connections programs make to files
• ulimit command: modify process limit parameters
in current shell
• Can also modify max number of filehandles
• by default it’s 1024
• ulimit –n 5000 (increases file handles)
• ulimit –u 30000 (increases max # of user processes in
shell)
22

23. SOFTWARE-RELATED PROBLEMS:
APPLICATION-RELATED
PROBLEMS (CONTINUED)
• /var/log directory: contains most system log files
• Some are hard linked to /var/log directory
• If applications stop functioning due to difficulty
gaining resources, restart using SIGHUP
• Do determine if another process trying to access the
same resources attempt to start application in Single User
Mode
• If resource conflict is the cause of the problem,
download newer version of app or application fix
23

24. SOFTWARE-RELATED PROBLEMS:
OPERATING SYSTEM-RELATED
PROBLEMS
• Most software-related problems related to OS
• X windows, boot loader, and filesystem problems
• Problem detecting video card or monitors by the
kernel
• To isolate problem starting X Windows or gdm:
• View /var/log/Xorg.0.log file
• Execute xwininfo or xdpyinfo
24

25. SOFTWARE-RELATED PROBLEMS:
OS-RELATED PROBLEMS
(CONTINUED)
• LILO (Linux Loader) problems: place “linear” in,
remove “compact” from /etc/lilo.conf file
• ubuntu: /etc/grub.d/10_linux, /etc/default/grub,
/boot/grub/grub.cfg
• GRUB problems: typically result of missing files in
/boot directory
• Ensure Linux kernel resides before 1024th cylinder
and lba32 (large block addressing) keyword is in
configuration file
• Eliminates BIOS problems with large HDDs
25

26. SOFTWARE-RELATED PROBLEMS:
OS-RELATED PROBLEMS
(CONTINUED)
• If filesystem on partition mounted to noncritical
directory becomes corrupted:
• Unmount filesystem
• Run fsck command with –f (full) option
• If fsck command cannot repair filesystem, use mkfs
command to re-create the filesystem
• Restore filesystem’s original data
26

27. SOFTWARE-RELATED PROBLEMS:
OS-RELATED PROBLEMS
(CONTINUED)
• If / filesystem is corrupted:
• Boot from Fedora installation media and enter System
Rescue
• At shell prompt within System Rescue:
• Use mkfs to recreate the filesystem
• Use backup utility to restore original data to the re-
created / filesystem (ie: tar, cpio, dump, restore, etc)
• Exit System Rescue and reboot system
• Knoppix Linux and BBC Linux: bootable Linux
distributions with many filesystem repair utilities
27

28. SOFTWARE-RELATED PROBLEMS:
USER INTERFACE-RELATED
PROBLEMS
• Assistive technologies: tools that users can use to
modify their desktop experience
• Assistive Technologies Preference utility within GNOME
Desktop Environment
• Preferred Applications to configure Web browser,
multimedia player and terminal applications to be
opened automatically
• Mouse Accessibility to configure speed and click
behavior
• Keyboard Accessibility to configure keyboard
related assistive technologies
28

29. SOFTWARE-RELATED PROBLEMS:
USER INTERFACE-RELATED
PROBLEMS (CONTINUED)
29
Figure 14-3: The Assistive Technologies Preferences utility

30. PERFORMANCE MONITORING
• Jabbering: failing hardware components send
large amounts of information to CPU when not in
use
• Other causes of poor performance:
• Software monopolizes system resources
• Too many processes
• Too many read/write requests to HDD
• Rogue processes
30

31. PERFORMANCE MONITORING
(CONTINUED)
• To solve software performance issues:
• Remove software from the system
• Move software to another Linux system
• Add CPU or otherwise alter hardware
• Bus mastering: peripheral components perform
tasks normally executed by CPU
31

32. PERFORMANCE MONITORING
(CONTINUED)
• To increase performance:
• Add RAM (reduces swap time with HDD)
• Upgrade to faster HDDs
• Disk Striping RAID
• Keep CD/DVD drives on a separate HDD controller
• Run performance utilities on a regular basis
• Record results in a system log book
• Eases identification of performance problems
• Baseline: measure of normal system activity
32

33. MONITORING PERFORMANCE
WITH SYSSTAT UTILITIES
• System Statistics (sysstat) package: contains
wide range of system monitoring utilities
• Use apt-get install sysstat command
• includes: mpstat, iostat, sar
33

34. MPSTAT
• multiple processor stats
• Used to monitor CPU performance
• Can specify interval and number of measurements rather than displaying
average values
• %sys should be smaller than %usr and %nice combined
34

35. MONITORING PERFORMANCE
WITH SYSSTAT UTILITIES
(CONTINUED)
• iostat (Input/Output Statistics) command:
measures flow of information to and from disk
devices
• Displays CPU statistics similar to mpstat
• Displays statistics for each disk device on the system
• Output includes:
• Transfers per second
• Number of blocks read and written per second
• Total number of blocks read and written for the
device
35

36. IOSTAT | LESS
36

37. MONITORING PERFORMANCE
WITH SYSSTAT UTILITIES
(CONTINUED)
• sar (System Activity Reporter) command:
displays various system statistics taken in the last
day
• Provides more information than mpstat and iostat
• By default scheduled to run every 10 minutes
• Output logged to a file in /var/log/sa directory
• -f option:View statistics from a specific file
• Can be used to take current system measurements
37

38. sar –q 1 5
• # of processes in q. runq-sz (run queue size)
• 2 or less is normal for Intel architectures
38
sar 2 4
• 4 CPU stats taken every 2 seconds

39. MONITORING PERFORMANCE
WITH SYSSTAT UTILITIES
(CONTINUED)
• Additional sar options:
• -q option: Displays processor queue statistics
• runq -sz value: Number of processes waiting for
execution on processor run queue
• plist -sz value: Indicates number of processes
currently running
• ldavg values: Represent average CPU load
• -W option: Displays number of pages sent to and taken from
swap partition
• Large number causes slower performance
• Add RAM to resolve
39

40. MONITORING PERFORMANCE
WITH SYSSTAT UTILITIES
(CONTINUED)
40
Common options to the sar command

41. OTHER PERFORMANCE
MONITORING UTILITIES
• top command: displays CPU statistics, swap usage,
memory usage and average CPU load
• free command: displays total amounts of physical
and swap memory and their utilizations
• Can be used to indicate whether more physical memory is
required
• vmstat command: displays memory, CPU, and swap
statistics
• Can be used to indicate whether more physical memory is
required
41

42. TOP – REAL TIME
42

43. FREE & VMSTAT
43

44. SECURITY
• Linux systems typically made available across
networks such as the Internet
• More prone to security loopholes and attacks
• Should improve local and network security
• Understand how to detect intruders who breach
the system
44

45. SECURING THE LOCAL
COMPUTER
• Limit access to physical computer itself
• Prevent malicious users from accessing files by directly
booting the computer with their own device
• Server closet: secured room to store servers
• Remove floppy, CD, and DVD drives from
workstations
• Ensure BIOS prevents booting from USB ports
45

46. SECURING THE LOCAL
COMPUTER (CONTINUED)
• Ensure BIOS password is set
• Set boot loader password in LILO (linux loader) or
GRUB configuration file
• Prevents intruder from interacting with boot loader
• Limit access to graphical desktops and shells
• Exit command-line shell before leaving computer
• nohup command: prevents background processes from
being killed when parent shell is killed or exited
• Lock screen using GNOME or KDE
46

47. SECURING THE LOCAL
COMPUTER (CONTINUED)
• Minimize root user’s time logged in
• su (switch user) command: switch current
user account to another
• Used to switch between root user and regular user
• sudo command: perform commands as another
user if you have the rights to do that listed in
/etc/sudoers file
47

48. CAT /ETC/GROUP
48

49. PROTECTING AGAINST
NETWORK ATTACKS
• Always a possibility that hackers can manipulate a
network service by interacting with it in unusual
ways
• Buffer overrun: program information for a network
service altered in memory
49

50. NETWORK SECURITY
ESSENTIALS
• Minimize number of running network services
• nmap (network mapper) command: scans
ports on network computers
• User can determine what network services are running
• Ensure that services that are not needed are not
automatically started when entering the runlevel
50

51. NETWORK SECURITY
ESSENTIALS (CONTINUED)
• Ensure network service daemons for essential
services not run as root user when possible
• Ensure that shell listed in /etc/passwd for
daemons is set to /sbin/nologin
• Hacker will not be able to get BASH shell
• New network service versions usually include
fixes for known network attacks
• Keep network services up-to-date
51

52. NETWORK SECURITY
ESSENTIALS (CONTINUED)
• TCP wrapper: program that can start a network
daemon
• Checks /etc/hosts.allow and /etc/hosts.deny files before
starting a network daemon
• Examine permissions for files and directories
associated with system and network services
52

53. CONFIGURING A FIREWALL
• netfilter/iptables: used to configure a firewall
• Discard network packets according to chains of rules
• Chains: specify general type of network traffic to apply rules to
• Rules: match network traffic to be allowed or dropped
• Three chain types:
• INPUT: incoming packets
• FORWARD: packets passing through computer
• OUTPUT chain: outgoing packets
53

54. CONFIGURING A FIREWALL
(CONTINUED)
• iptables command: creates rules for a chain
• Can be based on source IP, destination IP, protocol used,
or packet status
• Stateful packet filter: Remembers traffic allowed
in an existing session and adjust rules
appropriately
• Easier to use graphical utility to configure
firewalls
54

55. 55
Table 14-2: Common iptables options

56. CONFIGURING A FIREWALL
(CONTINUED)
56
Figure 14-4: The Firewall Configuration utility

57. CONFIGURING SELINUX
• SELinux: Security Enhanced Linux
• By default, configured and enabled during Fedora installation
• Series of kernel patches and utilities created by NSA
• Enforces role-based security
• To enable, edit /etc/selinux/config file
• Configure SELINUXTYPE option
• Reboot and relabel the system
• sestatus command: view current SELinux status
57

58. USING ENCRYPTION TO
PROTECT NETWORK DATA
• Use encryption algorithms to protect data before
it is transmitted on a network
• Asymmetric encryption: uses a pair of keys
uniquely generated on each system
• Public key: freely distributed
• Private key: used only by the system, never distributed
• Can be used to authenticate messages
• Digital signature: message that has been
encrypted using a private key
58

59. WORKING WITH SSH
• By default, SSH uses RSA to encrypt data and DSA to
digitally sign data
• System wide RSA and DSA key pairs are generated
the first time SSH daemon is started
• Tunneling: enclosing network traffic within encrypted SSH
packets
• SSH identity: used to automatically authenticate to
other computers using digital signatures
• Manage keys using Password and Encryption Keys
utility
59

60. WORKING WITH SSH
(CONTINUED)
60
Figure 14-5: The Passwords and Encryption Keys utility

61. WORKING WITH GPG
• Open source version of PGP
• Each user has a key pair used for encryption and
authentication
• Authentication uses trust model
• Typically uses RSA and DSA key pairs for
asymmetric encryption and digital signing
• Can manage GPG keys and encrypt data using:
• gpg command
• Graphical utility such as Passwords and Encryption Keys
utility
61

62. DETECTING INTRUSION
• Log files can contain information or irregularities
indicating an intrusion
• Review log files in /var/log associated with network
services
• At minimum, review system log files associated with
authentication
• Pluggable Authentication Module (PAM): handles
authentication requests by network applications
• Log file in /var/log/secure
62

63. DETECTING INTRUSION
(CONTINUED)
• Check /var/log/wtmp log file
• Lists users who receive BASH shells
• Use who command to view the file
• lsof (list open files) command: lists files
that are currently being edited
• Periodically search for files that have SUID bit set
• Tripwire: monitors important files and directories
• Intrusion Detection System (IDS): program used to
detect intruders on a Linux system
63

64. 64

65. DETECTING INTRUSION
(CONTINUED)
65
Table 14-3: Common Linux Intrusion Detection Systems

66. SUMMARY
• Administrators monitor the system, perform
proactive/reactive maintenance, and document
system information
• Common troubleshooting procedures involve:
• Isolating and determining the cause of system problems and
implementing and testing solutions that can be documented
for future use
• Invalid hardware settings, absence of device drivers,
and hard disk failure are common hardware-related
problems
66

67. SUMMARY (CONTINUED)
• Software-related problems can be application-
related or OS-related
• Users can use assistive technologies to modify
their desktop experience
• System performance is affected by a variety of
hardware and software factors
• Using performance monitoring utilities to create a
baseline is helpful for diagnosing future performance
problems
67

68. SUMMARY (CONTINUED)
• Securing a Linux computer involves:
• Improving local and network security and monitoring to
detect intruders
• Greatly improve local security by:
• Restricting access to the computer and using root account
only when required via su and sudo commands
68

69. SUMMARY (CONTINUED)
• Reduce chance of network attacks by:
• Reducing number of network services, implementing
firewalls, SELinux, service updates, encryption, and TCP
wrappers, and restricting services from running as root
user and permissions on key files
• Analyzing log files and key system files and
running IDS applications can be used to detect
intruders
69