Linux Power Tools

1. Linux Power Tools
Ketan (km0@ornl.gov)
ORNL
1
Contributors:
Kim Wong, Barry Moore, Shervin Sammak, Fangping Mu

2. Table of Contents
• Part 1: Introduction and Overview
• Part 2: Basics
• Part 3: Streams, pipe and redirection
• Part 4: find, grep, sed and awk
• Part 5: xargs and tmux
• Part 6: Scripting and Programming
• Part 7: Miscellaneous and Exercises
2

3. Part 1: Introduction and Overview
3
back to toc

4. Download Slides
The slides will be uploaded to talk announcement soon
4

5. Overview
• Build powerful command-lines and scripts
• We will use Bash
• Efficient data manipulation
• awk, sed, grep
• Goal is to be efficient and effective rather than to be an expert
• I do not cover: core sysadmin, security and networking
5

6. Why Power Tools?
• Save time
• Efficient for the system
• Prevent accidents
• Conform to "community practices"
• Promote learning
• Simple and useful
• good payback over long term
6

7. Mac OS X vs Linux
• Mac is based on BSD (Berkley Systems Distribution) Unix
• Linux mostly is GNU/Linux
• Both are mostly alike … except when they are not:
• different versions of bash and utilities: FreeBSD vs GNU projects
• mac invokes .bash_profile while linux invokes .bashrc on terminal startup
• sed, grep differ slightly
• We will assume GNU/Linux
7

8. Anatomy of a Linux Command
$ ls -lh | grep 'Oct 10' > result.txt
8
command
argument
pipeline
file
redirection
options commandprompt

9. Know the System
• id: know yourself
• w: who is logged in and what are they doing
• lsblk: list block storage devices (try lsblk -d)
• lscpu: display information about the cpu architecture
• free: free and used memory (try free -g)
• lsb_release: distribution information (try lsb_release -a)
9

10. part 2: Basics
10
back to toc

11. Work with Processes
• List the processes: ps (commonly used flags: aux)
• Checkout the memory map of a running process: pmap <pid>
• Be nice and fly under the radar, eg.:
• nice -n 19 tar cvzf archive.tgz somelargearchive
• Checkout the process hierarchy: pstree
• Kill a process: kill <pid>
11

12. Work with Files
• less is more than more
• tail -f: follow the growth of a file
• What can you do about binary files? (not much)
• strings will print the printable strings of file
• od will print file in octal format
• Reverse read a file with tac
• useful eg. to look at logs from bottom upwards
• Compare files with cmp (byte by byte), comm (line by line), diff
(differences line by line)
• In linux everything is a file
• what does this mean: there is a wikipedia article about it:
https://en.wikipedia.org/wiki/Everything_is_a_file
12

13. Work with Web: curl, wget, links
• curl: a tool to transfer data to/from web
• curl is commonly used as a quick command to download files
from the web
• curl -O http://www.gutenberg.org/files/4300/4300-0.txt
• libcurl is a curl based library that could be used in program
development
• wget is a similar utility; it has no library
• wget http://www.gutenberg.org/files/4300/4300-0.txt
• links is a useful text-based browser:
• over remote connections and when curl/wget won't work
• on a slow internet and only care about text
• want to avoid pesky ads on the web
13

14. Be a command line ninja: Navigation
14
ls -lh | grep 'Oct 10' > result.txt
cursor
ctrl-xx ctrl-ectrl-a
alt-falt-b
MAC users: terminal pref > profile > keyboard settings > Use option as meta key

15. Be a command line ninja: Deletion
15
ls -lh | grep 'Oct 10' > result.txt
cursor
ctrl-kctrl-u
ctrl-w alt-d
ctrl-y to paste back the last deleted item

16. Some Useful Shortcuts
16
• !! repeats the last command
• !$ change command keep last argument:
• cat a.txt #too long to fit screen
• less !$ #reopen it with less
• !* change command keep all arguments:
• head a.txt | grep -i 've' #should be tail
• tail !* #no need to type the rest of the command
• alt-. #paste last argument of previous command
• alt-<n>-alt-. #paste nth argument of previous command

17. More Useful Shortcuts
• >x.txt #quickly create an empty file
• fc #invoke an editor to fix last command
• ctrl-l #clear terminal
• cd - #change to previous dir
• cd #change to homedir
17

18. Part 3: Streams, pipe and redirection
18
back to toc

19. The pipe: run second command using output of first!
• a pipe is a Linux concept that automates redirecting the output of one
command as input to a next command.
• use of pipe leads to powerful combinations of independent
commands. eg.:
find | less #read a long list of files page wise
head afile.txt | grep -i 'admin'
history | tail #my last 10 commands
19

20. Streams and Redirection
20
standard input standard output

21. Essentials of Streams and Redirection
21
• Three streams: standard input (stdin), standard output (stdout) and
standard error (stderr)
• Standard streams are represented by "file descriptors":
• 0 for stdin
• 1 for stdout
• 2 for stderr
• & is used to "write into" a stream, eg. &1 to write into stdout
• Angle brackets are used for redirection:
• > to send
• < to receive
• >> to append
• << to in-place append (used in "heredoc")

22. Anatomy of Redirection
nohup cmd > /dev/null 2>&1 &
22
command null device
run cmd1
"beyond hangup"
send stderr
send
run in the background
"address of" stdout

23. Redirection Quickref
• Send stdout of cmd to a file: cmd > stdout.txt
• Send stderr of cmd to a file: cmd 2> stderr.txt
• Send stdout and stderr to a file:
cmd > stdouterr.txt 2>&1
• Send stdout and stderr to a file and get the prompt back:
cmd > stdouterr.txt 2>&1 &
• Disregard stdout and stderr: cmd > /dev/null 2>&1
• Disregard stdout and stderr and get the prompt back immediately:
cmd > /dev/null 2>&1 &
• Disregard stdout and stderr and let it run after you log out:
• nohup cmd > /dev/null 2>&1 &
23

24. tee: send stdout to file and pipe/console
24

25. Exceptions
• Most commands receive input from stdin (so, pipe) and file, eg.
• wc a.txt #ok
• wc < a.txt #ok
• All produce error messages on stderr
• There are some exceptions though
• Some receive input only from stdin and not from file, eg.
• tr 'N' 'n' a.txt #NOT OK
• tr 'N' 'n' <a.txt #ok
• Some receive input neither from stdin nor from file
• they only receive input as command line args, eg.
• echo <states.txt #NOT OK
• echo states.txt #NOT OK (if you want to print file contents)
• echo "Hello miss, howdy? " #ok, takes literal args
25

26. Part 4: find, grep, sed and awk
26
back to toc

27. The Versatile find
• Recursively examines a directory tree to look for files matching
criteria and optionally takes action on found files
• Flexible: multiple criteria may be combined to build a very specific
description of the file being searched
• Efficient: simple find is often faster than ls—very handy on slower
filesystems
27

28. Anatomy of find
find ~ -iname "README*" -exec wc -l {} +
28
path
criteria
action

29. Features of find
• path: may have multiple paths, eg. find /usr /opt -iname "*.so"
• criteria
• -name, -iname, -type (f,d,l), -inum <n>
• -user <uname>, -group <gname>, -perm (ugo)
• -size +x[c], -empty, -newer <fname>
• -atime +x, -amin +x, -mmin -x, -mtime -x
• action
• -print -- default action, display
• -exec cmd -- execute command cmd
• -ls -- run ls -lids command on each resulting file
• -ok cmd like exec except that command executed after user confirmation
29

30. Some Useful find Examples
• find . -type f -name "*.txt"
• find ./somedir -type f -perm 0777 -print
• find . -perm /u=r
• find . -type d -perm 777 -print -exec chmod 755 {} +
• find . -type f -name "*.tmp" -exec rm -f {} +
• find /usr/local/lib -atime +50 #<50 days
• find . -mtime +50 –mtime -100 #<50 & <100 days
30

31. grep: Search for patterns in text
• grep originally was a command "global regular expression print" or
'g/re/p' in the ed text editor
• It was so useful that a utility was developed
• grep will fetch lines from a text that has a match for a specific pattern
• Useful to find lines with a specific text in a large body of text, eg.:
• look for a process in a list of processes
• spot check a large number of files for occurrence of a pattern
• exclude some text from a large body of text
31

32. Anatomy of grep
grep -i -n 'col' a.txt
32
options
regular expression
input file

33. Some Useful grep Options
• -i: ignore case
• -n: display line numbers along with lines
• -v: print inverse ie. lines that do not match the regular expression
• -c: print a count of number of occurrences
• -A<n>: include n lines after the match
• -B<n>: include n lines before the match
• -o: print only the matched expression (not the whole line)
• -E: allows "extended" regular expressions that includes (more later)
33

34. Regular Expressions
• a regular expression is an expression that matches a pattern.
• example pattern:
• regular expression:  no match
• regular expression:  one match  "learning"
• regular expression:  two matches  "why" and "when"
• regular expression:  one match  "why"
34
e a
why waste time learning, when ignorance is instantaneous?
r
^ $
b a r
w h
^ w h

35. Regular Expressions-contd.
• Special characters:
• ^ will match from beginning of a line
• $ will match up to end of line
• . will match any character
• Character class: one of the items in the [] will match, sequences allowed
• '[Cc]at' will match Cat and cat
• '[f-h]ate' will match fate, gate, hate
• 'b[^eo]at' will match "brat" but not "boat" or "beat"
• Extended regular expressions (use with egrep or grep -E)
• '*' matches zero or more, '+' matches one or more, '?' matches zero or one
occurrence of the previous letter
• '|' is a delimiter for multiple patterns, '(' and ')' let you group patterns
35

36. grep Examples
• Lines that end with two vowels: grep ' [aeiou][aeiou]$' a.txt
• Count occurrence of term ’max': grep -c 'max' a.txt
• Check 5 lines before and after the term ‘node’: grep -A5 -B5 ‘node’ a.txt
• A classic demo of the power of regular expressions
• M[ou] '?am+[ae]r ([AEae]l[-])?[GKQ]h?[aeu]+([dtz][dhz]?){1,2}af[iy]
• This regular expression matches the dictator's name used by various news agencies at
the time:
• Muammar al-Kaddafi (BBC)
• Moammar Gadhafi (Associated Press)
• Muammar al-Qadhafi (Al-Jazeera)
• Mu'ammar Al-Qadhafi (US Department of State)
36

37. awk: Extract and Manipulate Data
• Old and powerful tool created by Aho, Weinberger and Kernighan
• A programmable filter that reads and processes input line by line
• Equipped with features such as variables, loops, conditionals, arrays and
built-in functions
• May read from file as well as standard input (so, pipes are good)
• May be run as command as well as an independent program
37

38. anatomy of awk
38
awk 'awk_prog' in.txt
awk -f awk_progfile in.txt
prog in file
"inline" prog input file
Inline prog structure
BEGIN{action}
pattern{action}
pattern{action}
...
pattern{action}
END{action}
#BEGIN, END and
patterns are optional
#At least one action
must be present

39. awk patterns and actions
• A pattern is a regex that matches (or not) to an input line, eg.
• /admin/ {action} #any line that contains ‘admin’
• /^[0-9]+ / {action} #beginning with numbers
• /(POST|PUT|DELETE)/ {action} #containing specific
words
• An action is a sequence of ops performed on matching lines, eg.
• {print $1;} #print first field
• {next;} #skip to the next line of input
• {for (i=1;i<x;i++) {sum += $3;}} #run a loop
• User defined functions may be defined in any action block
39

40. awk Feature Highlights
• Fields addressed by: $1, $2,…,$NF
• $0 means the whole line
• Pattern specified as /regex/ or $n~/regex/
• Special variables
• may be modified by user: FS(Field Sep), RS(Record Sep),
OFS(Output FS), ORS(Output RS)
• may not be modified by user: NF(num fields), NR(num records)
• Built-in functions, eg.: sin, cos, log, rand, substr
40

41. Some Useful awk Examples
• awk -F ':' '{print $1}' a.txt
• awk -F ':' '/New/{print $1}' a.txt
• ps aux | awk 'END{print NR}' #how many processes
• awk NF>0 a.txt #skip blank lines
• awk '{print NF, $0}’ a.txt
• awk '{print length($0)}' a.txt
41

42. sed: parse and transform text
• sed is a stream editor
• Looks for a pattern one line at a time and applies changes (edit) to
them
• Kind of a non-interactive editor
• Reads from file or stdin (so, pipes are good) one line at a time
• Lines are changed one line at a time
• The original input file is unchanged (sed is a filter), results are sent to
standard output
42

43. Anatomy of sed
43
sed '1,5s/search/replace/g' in.txt
delimiter input file
search regex replacement
address
modifiercommand

44. sed Options
• address: may be a line number or a range, defaults to whole file
• command: s:substitute, p:print, d:delete, a:append, i:insert, q:quit
• regex: A regular expression
• delimiter: Does not have to be /, can be | or : or any other character
• modifier: may be a number n which means apply the command to nth
occurrence. g means be "greedy" and apply to all
• Common sed flags: -n (no print), -e (multiple ops), -f (read sed
commands from file), -i (in place)
44

45. Some Useful sed Examples
• sed -n '1,3p' a.txt #print lines 1 through 3
• sed -n '$p' a.txt #print last line
• sed 's/[Nn]ew/old/g' a.txt
• sed '1,3d' a.txt #delete first 3 lines
• sed '/^$/d' a.txt #delete all blank lines
• sed '/^$/,$d' a.txt #delete from the first
blank line through last line
• [negation]sed '/York/!s/New/Old/' a.txt #change
New to Old except when York appears in the line
45

46. Part 5: xargs and tmux
46
back to toc

47. xargs: When pipeline is not enough!
• Some commands do not read from standard input, pipe or file; they
need arguments
• cp, touch, echo, rm, chmod
• Additionally, there is a limit on number of arguments on command
line
• for example: rm tmpdir/*.txt will fail if there are too many txt files
• xargs fixes both problems
• Appends standard input to commands as arguments
• Partitions the list of arguments to a limited number and runs the command
over them repeatedly as needed
47

48. Some xargs Examples
• Create a file titled the words that start with letter 'C':
• grep -i '^c' a.txt | awk '{print $4}' |touch #NOT OK
• grep -i '^c' a.txt | awk '{print $4}' |xargs touch #ok
• Remove system wide temporary files:
• find . -iname '*.tmp' | rm #NOT OK
• find . –iname '*.tmp' | xargs rm #ok
• Create a directory for all running processes
• ps | awk ‘NR != 1 {print $4}’| mkdir #NOT OK
• ps | awk ‘NR != 1 {print $4}’| xargs mkdir #ok
48

49. Workspace Management with tmux
• tmux is a terminal multiplexer that lets you create multiple,
persistent terminals within one login
• Useful
• when eg. a compilation or a remote copy operation will take a long
time
• for interactive multitasking
• for exotic stuff such as pair programming (not covered today)
49

50. A Short tmux Tutorial
• Typical tmux workflow
tmux new -s mysession #start a new session
# run any command that takes hours to finish
ctrl-b :detach #detach the session, logout, go home
#later, log in again
tmux a -t mysession #get the same session back
• Other useful tmux commands
ctrl-b ( #switch to previous session
ctrl-b ) #switch to next session
tmux ls #list all sessions
tmux kill-session -t myname #kill a session
50

51. part 6: Scripting and Programming
51
back to toc

52. Shell Scripting Basics
• Set of shell commands in a file that constitute an executable
• Arithmetic operations may be performed
• Variables and constants may be defined
• Conditionals and loops may be defined
• Functions may be defined
• Commands and utilities such as grep, sed, awk may be invoked
• A simple shell script:
52
#!/bin/sh
#prints hello
echo "Hello World"

53. Digression: heredoc
• handy when you need to create "inplace" files
• example:
• sh << END
echo "Hello World"
END <press enter>
• Uses of heredoc
• Multiline message using cat
• Use custom variables to create files
53
cat << feed >afile.txt
message line1
message line2
feed
#!/bin/sh
now=$(date)
cat <<END>timestamped.txt
The script $0 was last executed at $now.
other stuff
END

54. Shell Variable and Assignment
• Variables are implicitly typed
• May be a literal value or command substitute
• vname=value #assign value to variable vname
• $vname #read value of variable vname
• Command substitution:
• curdir=$(PWD)
• curdate=$(date +%F)
• echo "There are $(ls -1 | wc -l) items in the current dir"
54
#!/bin/sh
msg="Hello World"
echo $msg

55. Command line Parameters
• Parameters may be provided to a script at command line
• Accessible as $1, $2, …
• Special parameters:
• $0: program name
• $#: number of parameters
• $*: all parameters concatenated
55
#!/bin/sh
echo "Program: $0" #$0 contains the program name
echo "number of parameters specified is $#"
echo "they are $*" #all parameters stored in $*
grep "$1" $2

56. Conditionals
• if-then-else construct to branch into a script similar to programming
languages
• Two forms of conditional
evaluation mechanisms:
• test and [ … ]
56
#!/bin/sh
if test $USER = 'ketan'
then
echo "I know you"
else
echo "who are you"
fi
#!/bin/sh
if [ -f /etc/yum.conf ]
then
echo "yum conf exists"
if [ $(wc -l < /etc/yum.conf) -gt 10 ]
then
echo "and is more than 10 lines"
else
echo "and is less than 10 lines"
fi
else
echo "file does not exist"
fi

57. Conditional test summary
• string based tests
• -z string: length of string 0
• -n string: length of string not 0
• string1 = string2: strings are identical
• numeric tests
• int1 -eq int2: first int equal to second
• -ne, -gt, -ge, -lt, -le: not-equal, greater-than, -greater-or-equal, -less-than
• file tests
• -r file: file exists and is readable
• -w file: file exists and is writable
• -f, -d, -s: regular file, directory, exists and not empty
• logic
• !, -a, -o: negate, logical and, logical or
57

58. Loops
• Basic structure:
• Typically used with positional params or a list of files:
58
#!/bin/sh
for var in list
do
command
done
#!/bin/sh
sum=0
for var in "$@"
do
sum=$(expr $sum + $var)
done
echo The sum is $sum
#often used as one-liners on command-line
for file in $(ls -1 *.txt) ; do echo we have "$file"; done
#!/bin/sh
for i in $(seq 0 9)
do
echo $i
done

59. Should I write a script or a program
program
• (often) Structured and "systemic"
• Broader functionality
• Fast
• Compiled
• Good fit for algorithms
• example: C, C++
59
script
• (often) Quick and dirty
• Limited functionality
• Slow
• Interpreted
• Poor fit for algorithms
• example: bash, R

60. Elements of program development
• Four phases:
1. development (aka program writing)
2. compile
3. execution (aka runtime)
4. debug (optional but almost inevitable)
• Tangible elements:
• Source code
• Object code
• Config files
• Shared and static lib(s)
• Executable(s)
60

61. Program development tools
• Most Linux installations have all you need to develop simple programs
• Editor: nano, vim, emacs
• Compiler: cc, gcc, g++
• Program libraries tools: ar
• A build system: make
• C code
• Compile: gcc -c example.c -o example.o
• Link: gcc main.o example.o -o example
• Build a shared library: gcc example.o -shared -o example.so
• Build an archive: ar -rv example.a main.o example.o
61

62. A Makefile
all: example
example: main.o example.o
gcc main.o example.o -o example
main.o: main.c
gcc -c main.c -o main.o
example.o: example.c
gcc -c example.c -o example.o
clean:
rm *.o example
install: example
cp example ${HOME}
62

63. Program debugging tools
• strace: trace system calls and signals of a running executable
• strace -o strace.out ls
• gdb: catch bugs in the act
• start program, specifying anything that might affect its behavior
• make program stop on specified conditions.
• examine what has happened, when program stopped.
• make changes in program, experiment with correcting the effects of bug.
• a simple shell option to debug shell scripts:
• sh -x myscript.sh
63

64. part 7: Miscellaneous and Exercises
64
back to toc

65. Miscellaneous useful utilities
• Factorize an integer for fun using factor
• factor 300
• Record your session with script
• watch a changing variable
• watch -n 5 free -m
• Say yes and save time
• yes | rm -r largedir #do not try on critical data
• yes '' | pdflatex report.tex
• Work with xml files with xmllint
65

66. wildcards
• * and ?
• The character class
• The numeric class
• Negation (!)
• Escaping and quoting
66

67. Exercises: 1
• list only all the directories in your $HOME directory
• hints: use long form of ls, grep the first letter
• create a background process using command: sleep 100; kill it using the kill
command
• use find to
• copy all files modified within the last 24 hours to an existing directory named 'day1'
• change all directory permissions to 755 and all file permissions to 644 in your home
directory tree.
• use grep to search for all lines of file foo.txt containing a word of length
four or more starting with the same two characters it is ending with. You
may use extended regular expressions (-E)
67

68. Exercises: 2
• tmux: create 3 sessions: s1, s2 and s3. kill all sessions with one
command
• hints: use pipeline, grep and xargs
• list all running process names in one line (no newlines)
• hints: use ps, awk, tr
• using awk, list only the postal codes for all states from the states.txt
• how do you display the date output with each field on a separate
line using tr? do the same using sed? do the same using awk?
68

69. Exercises: 3
• display states.txt such that the capitals appear in the first column and
the state names in the last.
• look up the tar man page to find out whether the command tar -
cvfb 20 foo.tar *.c is legitimate or not. Will the command
work without the - symbol?
• write a shell script to find all the prime numbers between 1000 and
10000
• hints: use for, if, factor, wc
• add a new target 'static' to the Makefile to build the archive
example.a
69

70. Credits, references and resources
• wikipedia articles: unix, linux, Bash_(Unix_shell)
• http://www.cs.nyu.edu/~mohri/unix08
• bash
• bash ref. manual: https://www.gnu.org/software/bash/manual/bashref.html
• grep
• www.gnu.org/software/grep/manual/grep.html
• sed
• www.catonmat.net/blog/worlds-best-introduction-to-sed
• awk
• ferd.ca/awk-in-20-minutes.html
• tmux
• gist.github.com/MohamedAlaa/2961058
70

71. Thank you for your attention
71

72. environment variables
• What is the difference between .bashrc, .bash_profile and .profile
files
• Environment variables on command line vs .bash* files
• LD_X_PATH, CDPATH, rpath
72

73. Get things done at specific times, repetitively
• at
• batch
• cron, crontab
73

74. the big picture (s)
74
/
home
tom
nancy
etc
proc
boot
usr
bin
sbin
libdev
bin
lib
mnt storedisk
user home dirs
config files
cpu, mem,
process info
kernel is here
devices are here
executable progs
prog libraries
storage
more executables
more libs

75. more about man
• always look how the tool interacts with standard streams
• man pages has sections
• section 1: general command
• section 2: system calls
• section 3: library functions
• section 4: special files and drivers
• section 5: file formats and conventions
• section 6: games and screensavers
• section 7: miscellanea
• section 8: sysadmin commands and daemons
• you can read the manual of man by typing:
man man
• invoke a particular section by providing section number as argument, eg.
man 3 sleep; man 2 kill
75

76. final words
• Pay attention to details
• each typed (and not typed) letter matters
• Question everything
• why?
• Do not reinvent the wheel .. unless you have to
• find existing wheels and reuse them
• Do not rediscover America .. unless you have to
• find Columbus’ route and follow it
76

77. • for i in $(seq 1000 10000); do if test $(factor $i|wc -w) -eq 2; then
echo $i; fi ; done
77

78. 78
ls -lh | grep 'Oct 10' > result.txt
cursor
ctrl-xx ctrl-ectrl-a
alt-falt-b
ls -lh | grep 'Oct 10' > result.txt
cursor
ctrl-kctrl-u
ctrl-w alt-d
ctrl-y to paste back the last deleted item
tmux new -s mysession #start a new session
ctrl-b :detach #detach the session
tmux a -t mysession #get the same session back
watch -n 5 free -m#!/bin/sh
now=$(date)
cat <<END>timestamped.txt
The script $0 was last executed at $now.
other stuff
END

79. thehistory
79
mac os x
linux