Your SlideShare is downloading. ×
  • Like
  • Save
Unix shell scripting
Upcoming SlideShare
Loading in...5

Thanks for flagging this SlideShare!

Oops! An error has occurred.


Now you can save presentations on your phone or tablet

Available for both IPhone and Android

Text the download link to your phone

Standard text messaging rates apply

Unix shell scripting


This presentation is made for beginners who wish to write a simple script and also help those who already working in the sys-admin, support roles to increase their scope for more experiments in …

This presentation is made for beginners who wish to write a simple script and also help those who already working in the sys-admin, support roles to increase their scope for more experiments in different areas where a developer, sys-admin, app-admin, web-admin can use it.

Published in Education
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
No Downloads


Total Views
On SlideShare
From Embeds
Number of Embeds



Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

    No notes for slide
  • Intro Shell in Unix/LinuxChoosing shell for ur environmentCompare root, regular user execute shellWhat are the shell script needs?
  • These system calls request services to be provided by the kernel. Such services would include accessing a file: open close, read, write, link, or execute a file; changing ownership of a file or directory;
  • What is UNIX actually?
  • Expr and $(()) works similar but see the best choice
  • Ref:
  • History -20History +20 Will give the number of previous commands used
  • Therm command is very dangerous it is not like windows, where we don’t have recycle bin here.Interactive mode rm –i is better option.
  • More Loop examples:
  • Tilda code will execute the command inlineAnother way is prefixing $ and () will also execute the command inline
  • Exec can be shown with find and ls command
  • In Unix every thing is a file.
  • Most often question is : "What is necessary to installing Linux for first time?". You must have a free disk space. The best and easiest way is if the whole hard disk drive is dedicated to Linux. Linux uses "its own file system" (FS) and most used are ext2, ext3, ext4 and RAISERFS. Can be used from other operating system, but best is they native fs. There is another FS, which is important for Linux - SWAP FS.Di-spite the fact that is not necessary for Linux SWAP partition, it is common practice to create separate partitions for SWAP. SWAP or Page file concept is known to all those who worked on Windows OS and marks one part of the Virtual memory stored on hard disk. Usually the size of SWAP is 1.5 to 2 times of RAM (physical memory), but to my personal experience, a reasonable upper limit for SWAP is 3G of RAM.Next on what should pay attention is the term - Partition. If we look Windows, we have partition, which we recognize them by letter of the English alphabet. Linux, as Unix-like system, use completely different system of labeling.It could be said to we have devices that are attached to the Unix tree. Tree is something that is typical for Unix-like systems and has a root of tree (labeling - / ) and his directories (tree). Each device gets its mountpoint - a place from where you can access it.
  • you don’t mention –s that means hardlink
  • Ksh uses POSIXBash uses general class
  • Different variants of find options will help you to write simple scripts
  • “This is the Unix philosophy: Write programs that do one thing and do it well. Write programs to work together. Write programs to handle text streams, because that is a universal interface.” --Doug McIlroy, inventor of Unix pipes
  • Use the top command for the process that uses high CPU and memory.


  • 1. UNIX & LINUX Shell Scripting
  • 2. Course Outline What | Who | How
  • 3. Course objective • Know about UNIX, shell, types of shells • Identifying the shell that suites to your needs • Users and groups in a shell, permissions • Use archiving and compression commands • Use basic regular expressions • Identify process management and job control • Identify the Linux Filesystem • Resource management • Network resources, remote communications • Use text editors like vi, vim • Write the automation scripts
  • 4. Scope : Who can do this? • System Administrators – Beginner Linux Admin – DBA – WebLogic Admin – WebSphere Admin – Jboss Admin • Application Developers • Configuration & Release managers (SCM Practice) • Support team members • Technical lead, Managers • Automation Specialist • Web Administrators • Security team
  • 5. Overview of UNIX & Linux UNIX, History, influencers
  • 6. Overview of UNIX & Linux • UNIX Operating System • UNIX Evolution • UNIX and Linux Influencers • Flavors Of UNIX • Why UNIX? • Unix and Linux Features
  • 7. UNIX Operating System • Unix is a layered operating system. The innermost layer is the hardware that provides the services for the OS. • The operating system, referred to in Unix as the kernel, interacts directly with the hardware and provides the services to the user programs. • User programs interact with the kernel through a set of standard system calls.
  • 8. UNIX History
  • 9. UNIX and Linux Influencers Linus Torvalds Richard Stallman Dennis Riche Ken Thompson
  • 10. Flavors of UNIX
  • 11. Trend : Virtual Boxes • Virtual Box
  • 12. Why UNIX? • PC magazine report
  • 13. UNIX and Linux Features • Its first real use is as a text processing tool for the patent department at Bell Labs. It was designed with these features: • Programmers environment • Simple user interface • Simple utilities that can be combined to perform powerful functions • 1973 Unix is re-written mostly in C, a new language developed by Dennis Ritchie. Being written in this high-level language greatly decreased the effort needed to port it to new machines. • And transparent to the user.
  • 14. Directory structure
  • 15. CUI Editors Text editors emacs GUI Editors
  • 16. The vi editor
  • 17.
  • 18. Kernel • Core of the Unix system. • Interacts directly with the OS • Largely written in C, with some parts written in Assembly language. • Insulates other parts of OS from hardware. • Performs all low level functions. • All the applications deal with Kernel. • Handles all the devices.
  • 19. Kernel Functions • Memory management. • Process scheduling. • File management and security. • Interrupt handling and error reporting. • Input / Output services. • System accounting
  • 20. Shell • Acts as an interface between the user and the kernel. • Is the command interpreter of UNIX. • Provides powerful programming capabilities. • System administration. • Text processing. • File management. • UNIX Utilities and Applications. • User can customize his/her own shell. • Users can use different shells on the same machine.
  • 21. Book References
  • 22. Online shells • Linux zoo : • There is 15 min shell : • Short time shell : • Side by side shell & executed outputs : • Cloud Shells: – Amazon EC2 – Ibm cloud instance
  • 23. Shell Basics SHELL hierarchy, types, Login shell
  • 24. UNIX Shell basics • Shell hierarchy • Types of Shells • Know about Shell • Login Shell • Why Use Shell Script?
  • 25. Types of shells
  • 26. Shell Features • Filename completion • Command-line history. • Multiple job control. • Redirection (program to file). • Pipes (program to program). • Storage variables. Some of the common features that most all shells have to offer:
  • 27. Booting Process • Hardware configuration check. • The operating system file /vmunix ( the UNIX kernel) gets loaded. • The kernel brings up the swapper and the init program. • Jobs of init are: – Takes the system into the multi-user mode. – Checks the integrity of the file systems. – Mounts the file systems – The login process gets executed. – After successful login, the shell appears.
  • 28. Run Levels • UNIX system has several modes of operation called system states ( or run levels). These are: 0 : shutdown state 1 : administrative state s or S : single user state 2 : multi-user state 6 : stop and reboot state • These states are passed to the init program, which operates the system in the appropriate state.
  • 29. Login shell
  • 30. I am in which shell? • The current shell program that is in use is stored in environment variable SHELL. • To find the current shell program, use following command • $ echo $SHELL • /bin/bash • $
  • 31. Why Use Shell scripts ? • Batch processing of commands • Repetitive tasks • You can use shell scripts to automate administrative tasks. • Encapsulate complex configuration details. • Get at the full power of the operating system. • The ability to combine commands allows you to create new commands Adding value to your operating system.
  • 32. Shell command-line Variables, Math, Commands
  • 33. Shell command line • Variables and strings • Math in Scripting • Shell Commands
  • 34. Variable and Strings • Variables prefix with $ is a variable. – $VAR ${VAR} • Examples: ${FILE}.txt • echo “my name is $NAME” • echo “my name is “$NAME • read VAR • read VAR1 VAR2 • Strings are enclosed in “ ” or just written with spaces escaped by VAR=”How are you” • VAR=How are you • VAR=HOW are you
  • 35. Math in scripting • Use $((…)) instead of expr to evaluate arithmetic equation • x=1 • x=$((x+1)) • echo $((1+1)) • echo $[1+1] • echo `expr 1 + 1` • A=$((B*C)) • let A+=1 • echo "3.4+7/8-(5.94*3.14)" | bc bc let $(( )) expr
  • 36. An aside: $(( )) for Math % echo $(( 1 + 2 )) 3 % echo $(( 2 * 3 )) 6 % echo $(( 1 / 3 )) 0
  • 37. Math with bc • bc ("basic calculator?") bc is a handy but full-featured calculator. Even includes a programming language! Use the -l option to have floating point support. • z=10; z=`echo "$z + 1" | bc`
  • 38. Shell commands • date: Displaying the System Date • cal: The Calendar • echo: Displaying a Message • printf: An Alternative echo • script: Recording Your Session • Email Basics: mailx: The Universal Mailer • passwd: Changing Your Password • who: Who are the users? • uname: Knowing Your Machine's Characteristics • tty: Knowing Your Terminal • stty: Displaying and Setting Terminal Characteristics
  • 39. The date command • The date command returns the current date or time. • Useful in many scripts to record when commands started or completed. Example: % DATESTRING=`date +%Y%m%d` % echo $DATESTRING 20060125 % man date
  • 40. Customizing ENV path, display, expor t
  • 41. Customizing Environment • How to Setup Environment? • Environment variables • The history command options • Shortcuts : Defining alias • Utility scripting • Scoping with export
  • 42. How to Setup Environment? • The Common Environment Variables • Aliases (bash and ksh) • Command History (bash and ksh) • In-line Command Editing (bash and ksh) • To override the value of an environment variable for a single command, just specify the desired value before the command: ENV[„PATH‟] • The Initialization Scripts
  • 43. Environment variables • $HOME home directory • $PATH path • $PS1 user prompt (normally %) • $PS2 secondary prompt (normally >) • $$ process id of the script • $# number of input parameters • $0 name of the script file • $IFS separation character (white space) • Use „env‟ to check the value
  • 44. The history commands • The history command will show you a list of the most recently used commands, with command sequence numbers. To specify a previously issued command: • use the up and down arrow keys to scroll through the history • use ”!” and the command number, or “!!” for the most recent command. • use [ctrl-r]: recall a previous command containing a desired string by typing [ctrl-r] and the string. Then continue to press [ctrl-r] to cycle through the matching commands. They can be edited.
  • 45. Shortcuts :Defining alias • Global alias can be defined in .profile file you can test in command line as well – Syntax: alias name=value – Here the value can be any command or script paths • Examples: – alias -g L="|less" – alias c='clear„ – alias now='date +"%T' – alias -g ...='../..' – alias ping='ping -c 5„ – alias update='yum update„ – alias meminfo='free -m -l -t' – alias apt-get='sudo apt-get„ • You can do unalias also • unalias aliasname • Example : – unalias L
  • 46. Utility scripts • Frequently used tasks can be turn to utility
  • 47. Scoping with export • Without export • TUTOR=„Shekhar‟ • bash • echo $TUTOR • With export
  • 48. Shell Programming Execution
  • 49. Shell Programming • Shell Programming • Program structure • Shell variables • Quotes - Strings • Shell Output • Print formatted data • Continuing Lines • Exit Status • Logic using test • Expressions
  • 50. How shell works? • Execution cycle for „ls‟ command
  • 51. Shell defaults • $# : number of arguments on the command line • $- options supplied to the • ? exit value of the last command executed • $n argument on the command line, where n is from 1 through 9, reading left to right • $0 the name of the current shell or program • $* all arguments on the command line ("$1 $2 ... $9") • $@ all arguments on the command line, each separately quoted ("$1" "$2" ... "$9") • $argv[n] selects the nth word from the input list • $#argv report the number of words in the input list • $$ process number of the current process • $! process number of the last command done in background
  • 52. Shell programming • We can write a script containing many shell commands • Interactive Program: – grep files with POSIX string and print it % for file in * > do > if grep –l POSIX $file > then > more $file  fi  done Posix There is a file with POSIX in it – „*‟ is wildcard % more `grep –l POSIX *` % more $(grep –l POSIX *) % more –l POSIX * | more
  • 53. Shell variables • Variables needed to be declared, • Note it is case-sensitive (e.g. foo, FOO, Foo are different values) • Add „$‟ for storing values % salutation=Hello % echo $salutation Hello % salutation=7+5 % echo $salutation 7+5 % salutation=“yes dear” % echo $salutation yes dear % read salutation Hola! % echo $salutation Hola!
  • 54. Quotes - Strings • Edit a “” file #!/bin/sh clear myvar=“Hi there” echo $myvar echo “$myvar” echo `$myvar` echo $myvar echo Enter some text read myvar echo „$myvar‟ now equals $myvar exit 0 Output Hi there Hi there $myvar $myvar Enter some text Hello Scripting $myvar now equals Hello Scripting
  • 55. Shell Output - echo • echo command used to print string and variable values • -n do not output the trailing newline • -e enable interpretation of backslash escapes – 0NNN the character whose ACSII code is NNN – backslash – a alert – b backspace – c suppress trailing newline – f form feed – n newline – r carriage return – t horizontal tab – v vertical tab Try following examples % echo –n “string to n output” % echo –e “string to n output”
  • 56. Print formatted data • printfformat and print data • Escape sequence – backslash – a beep sound – b backspace – f form feed – n newline – r carriage return – t tab – v vertical tab • Conversion specifies – %d decimal – %c character – %s string – %% print % Example: % printf “%sn” hello Hello % x=100 % printf “%s %dt%s” “Hi There”, $x people Hi There 100 people
  • 57. An aside: Quoting % echo „$USER‟ $USER % echo “$USER” pavanusr % echo “”” ” % echo “vybhavatrainings” Vybhavatrainings % echo „”‟ ”
  • 58. Continuing Lines: % echo This Is A Very Long Command Line This Is A Very Long Command Line %
  • 59. Exit Status • $? • 0 is True % ls /does/not/exist % echo $? 1 % echo $? 0
  • 60. Exit Status: exit • exit nending the script • 0 means success • 1 to 255 means specific error code • 126 means not executable file • 127 means no such command • 128 or >128 signal % cat > <<END exit 3 END % chmod +x % ./ % echo $? 3
  • 61. Logic: test • test or „ [ ] „ if test –f fred.c then ... Fi % test 1 -lt 10 % echo $? 0 % test 1 == 10 % echo $? 1 If [ -f fred.c ] then ... fi -d file if directory -e file if exist -f file if file -g file if set-group-id -r file if readable -s file if size >0 -u file if set-user-id -w file if writable -x file if executable need space !
  • 62. Logic: test • test • [ ] – [ 1 –lt 10 ] • [[ ]] – [[ “this string” =~ “this” ]] • (( )) – (( 1 < 10 ))
  • 63. Expressions • Conditional operations – expression1 –eq expression2 – expression1 –ne expression2 – expression1 –gt expression2 – expression1 –ge expression2 – expression1 -lt expression2 – expression1 –le expression2 – !expression • String validation – String1 = string2 – String1 != string 2 – -n string (if not empty string) – -z string (if empty string) • [ -f /etc/passwd ] • [ ! –f /etc/passwd ] • [ -f /etc/passwd –a –f /etc/shadow ] • [ -f /etc/passwd –o –f /etc/shadow ]
  • 64. Shell Script Controls If, case, while, for loop
  • 65. Shell script controls • Shell script if control • Switching with case • while: Looping • for: Looping with a List • break and continue in loops • trap: Interrupting a program • Debugging Shell Scripts • Exercise
  • 66. If-else Control Syntax if condition then statement else statement fi 66 #!/bin/sh echo “Is it morning? Please answer yes or no” read timeofday if [ $timeofday = “yes” ]; then echo “Good morning” else echo “Good afternoon” fi exit 0 Is it morning? Please answer yes or no yes Good morning
  • 67. If-elif-else control 67 #!/bin/sh echo “Is it morning? Please answer yes or no” read timeofday if [ $timeofday = “yes” ]; then echo “Good morning” elif [ $timeofday = “no” ]; then echo “Good afternoon” else echo “Sorry, $timeofday not recognized. Enter yes or no” exit 1 fi exit 0
  • 68. If with exit 68 #!/bin/sh echo “Is it morning? Please answer yes or no” read timeofday if [ “$timeofday” = “yes” ]; then echo “Good morning” elif [ $timeofday = “no” ]; then echo “Good afternoon” else echo “Sorry, $timeofday not recongnized. Enter yes or no” exit 1 fi exit 0 If input “enter” still returns Good morning
  • 69. If and case Controls • If then, else, elif, fi 1. if [ -d $F ]; then 2. rm -rdf $F 3. elif [ -f $F ]; then 4. rm $F 5. else 6. echo “Unknown file type” 7. fi • case, esac – while read l; do – case $l in 1) echo "One";; – 2) echo "Two";; – 3) echo "Three";; – *) echo "Invalid"; break;; – esac done
  • 70. Case Statement Syntax case variable in pattern [ | pattern ] …) statement;; pattern [ | pattern ] …) statement;; … esac 70 #!/bin/sh echo “Is it morning? Please answer yes or no” read timeofday case “$timeofday” in yes) echo “Good Morning”;; y) echo “Good Morning”;; no) echo “Good Afternoon”;; n) echo “Good Afternoon”;; * ) echo “Sorry, answer not recongnized”;; esac exit 0
  • 71. Case Statement • A much “cleaner” version 71 #!/bin/sh echo “Is it morning? Please answer yes or no” read timeofday case “$timeofday” in yes | y | Yes | YES ) echo “Good Morning”;; n* | N* ) echo “Good Afternoon”;; * ) echo “Sorry, answer not recongnized”;; esac exit 0 But this has a problem, if we enter „never‟ which obeys n* case and prints “Good Afternoon”
  • 72. Improved Case Statement 72 #!/bin/sh echo “Is it morning? Please answer yes or no” read timeofday case “$timeofday” in yes | y | Yes | YES ) echo “Good Morning” echo “Up bright and early this morning” ;; [nN]*) echo “Good Afternoon”;; *) echo “Sorry, answer not recongnized” echo “Please answer yes of no” exit 1 ;; esac exit 0
  • 73. Loops in shell scripting • While loop • while read VAR; do • echo $VAR; • done • For each loop • for VAR in {1..5}; do • echo $VAR; • done • for VAR in 1 2 3 4 5; do • echo $VAR; • done • for VAR in {0..10..2}; do • echo $VAR; done • for VAR in $(seq 0 2 10); do • echo $VAR; • done • For loop with counter • for (( i=0; i<=10; i+=2 )); do echo $i; • done
  • 74. Control break • Internal: • only in script • break 74 #!/bin/sh rm –rf fred* echo > fred1 echo > fred2 mkdir fred3 echo > fred4 for file in fred* do if [ -d “$file” ] ; then break; fi done echo first directory starting fred was $file rm –rf fred* exit 0
  • 75. Continue loops • Continue Continues next iteration 75 #!/bin/sh rm –rf fred* echo > fred1 echo > fred2 mkdir fred3 echo > fred4 for file in fred* do if [ -d “$skipdir” ]; then echo “skipping directory $skipdir” continue fi echo file is $file done rm –rf fred* exit 0
  • 76. The trap Command • trap action after receiving signal trap command signal • signal explain HUP (1) hung up INT (2) interrupt (Crtl + C) QUIT (3) Quit (Crtl + ) ABRT (6) Abort ALRM (14) Alarm TERM (15) Terminate 76
  • 77. Example of trap Command 77 #!/bin/sh trap „rm –f /tmp/my_tmp_file_$$‟ INT echo creating file /tmp/my_tmp_file_$$ date > /tmp/my_tmp_file_$$ echo “press interrupt (CTRL-C) to interrupt …” while [ -f /tmp/my_tmp_file_$$ ]; do echo File exists sleep 1 done echo The file no longer exists trap INT echo creating file /tmp/my_tmp_file_$$ date > /tmp/my_tmp_file_$$ echo “press interrupt (CTRL-C) to interrupt …” while [ -f /tmp/my_tmp_file_$$ ]; do echo File exists sleep 1 done echo we never get there exit 0
  • 78. Debug scripts • sh –n<script> set -o noexec check syntax set –n • sh –v<script> set -o verbose echo command before set –v • sh –x<script> set –o trace echo command after set –x set –o nounset gives error if undefined set –x set –o xtrace set +o xtrace trap „echo Exiting: critical variable =$critical_variable‟ EXIT 78
  • 79. Exercise • Create script that prints 1-10 all even numbers • Create script that prints 3 multiples skip printing 9 and 18 terminate at 30. • Print prime numbers to given range. • Print Fibonacci numbers given range.
  • 80. Shell & Sub-shells Functions, fork, exec
  • 81. Sub-shells • Function • Shell and sub-shell • The process system calls fork, exec, exit, wait • Array in shell
  • 82. Function • You can define functions for “structured” scripts function_name() { statements } 82 #!/bin/sh foo() { echo “Function foo is executing” } echo “script starting” foo echo “script ended” exit 0 Output script starting Function foo is executing Script ended You need to define a function before using it The parameters $*,$@,$#,$1,$2 are replaced by local value if function is called and return to previous after function is finished
  • 83. Defining a Function 83 #!/bin/sh sample_text=“global variable” foo() { local sample_text=“local variable” echo “Function foo is executing” echo $sample_text } echo “script starting” echo $sample_text foo echo “script ended” echo $sample_text exit 0 Output? Check the scope of the variables define local variable
  • 84. The function returns • Use return to pass a result 84 #!/bin/sh yes_or_no() { echo “Is your name $* ?” while true do echo –n “Enter yes or no:” read x case “$x” in y | yes ) return 0;; n | no ) return 1;; * ) echo “Answer yes or no” esac done } echo “Original parameters are $*” if yes_or_no “$1” then echo “Hi $1, nice name” else echo “Never mind” fi exit 0 Output ./my_name John Chuang Original parameters are John Chuang Is your name John? Enter yes or no: yes Hi John, nice name.
  • 85. Command execution • $(command) to execute command in a script • Old format used “`” but it can be confused with “‟” #!/bin/sh echo The current directory is $PWD echo the current users are $(who)
  • 86. Shells and Sub-shells • () and {}: Sub-shell 0r Current Shell? • export: Exporting Shell Variables • Running a Script in the Current Shell: The . Command • let: Computation-a Second Look (Korn and Bash) • Arrays (Korn and Bash) • String Handling (Korn and Bash) • Conditional Parameter Substitution • Merging Streams • Shell Functions • eval: Evaluating Twice • The exec Statement
  • 87. Fork, exec, exit, wait
  • 88. Array in Shell • Declaring Array • declare -a Linux=('Debian' 'Red hat' 'Suse' 'Fedora'); • Printing Array • echo ${Linux[@]} • Debian Red hat Ubuntu Suse • Array Size • echo ${#Linux[@]} #Number of elements in the array • 4 • echo ${#Linux} #Number of characters in the first element of the array.i.e Debian • 6
  • 90. File commands • What's in a (File) name? • The Parent-Child Relationship – The HOME Variable: The Home Directory – Current Directory – cd: Changing the Current Directory • mkdir: Making Directories • rmdir: Removing Directories • Absolute Pathnames • Relative Pathnames • ls: Listing Directory Contents • The UNIX File System
  • 91.
  • 92. Everything is a file Regular files Almost everything in Unix is a file! Directories Directory are just files listing a set of files Symbolic links Files referring to the name of another file Devices and peripherals Read and write from devices as with regular files Pipes Used to cascade programs cat *.log | grep error Sockets Inter process communication
  • 93. What is file? • In UNIX everything is a file • Files can be tested with file command with filename. • Text file • Executable file • ASCII file • Binary file
  • 94. File system hierarchy • Root and directories / bin etc dev usr home passwd passwd hosts local lib
  • 95. Change directory • > cd /etc # change to absolute path • > cd mypath # change to relative path • > cd ~ # change to home directory • > cd # same as cd ~ (unlike DOS/Windows) • > cd - # toggles current and previous directory • > pushd/popd # directory stack • If the shell finds a CDPATH environment variable, then all directories in that path will be searched for the subdirectory when a relative cd is executed: • > export CDPATH=~/docs • > cd a_doc_subdir # will work even when not in ~/docs
  • 96. Make directory • Creates subdirectories with all necessary intermediate directories in a single command: > mkdir -p a/really/deep/directory/tree • A shortcut for capturing the last parameter in the previously executed command is !$, so you can do this to change to that new directory: > cd !$ • As stated before, to jump back up, you can do: > cd -
  • 97. Remove directory • Remove a file with the rm, remove, • rm [options] filename • -i interactive (prompt and wait for confirmation before proceeding) • -r recursively remove a directory, first removing the files and subdirectories beneath it • -f don't prompt for confirmation (overrides -i)
  • 98. File system basics • / is not allowed in a file name • file extensions are not important • File and folder names are case sensitive • Names containing spaces are escaped by or "" • Hidden files start with . (dot) • Current directory alias . and parent directory alias .. • ls, ls -a, ls -l, cp, mv, rm, rm -rdf, ln, ln -sf
  • 99. The ls command • Lists the files in the current directory, in alphanumeric order,except files starting with the “.” character. • ls -a (all) Lists all the files (including .* files) • ls -l (long) Long listing (type, date, size, owner, permissions) • ls -t (time) Lists the most recent files first • ls -s (size) Lists the biggest files first • ls -r (reverse) Reverses the sort order • ls -ltr (options can be combined) Long listing, most recent files at the end
  • 100. File name pattern • Better introduced by examples! • ls *txt The shell first replaces *txt by all the file and directory names ending by txt (including .txt), except those starting with ., and then executes the ls command line. • ls -d .* Lists all the files and directories starting with . -d tells ls not to display the contents of directories. • cat ?.log Displays all the files which names start by 1 character and end by .log
  • 101. File handling with shell • The File • What's in a (File)name? • The Parent-Child Relationship • The HOME Variable: The Home Directory • Absolute Pathnames • Relative Pathnames • ls: Listing Directory Contents • The UNIX File System
  • 102. File maintenance : cp, mv • Copy the contents of one file to another with the cp command. • cp [options] old_filename new_filename • Common Options • -i interactive (prompt and wait for confirmation before proceeding) • -r recursively copy a directory • Rename a file with the move command, mv. mv [options] old_filename new_filename
  • 103. Output file redirection • Output redirection takes the output of a command and places it into a named file. Input redirection reads the file as input to the command. • > - output redirect • >> append output
  • 104. Input file redirection • < input redirection • <<String read from standard input until "String" is encountered as the only thing on the line.
  • 105. The cat options • Creating Text Files with cat The “cat” command is used to create small text files without using an editor. • The cat utility reads the input from the user and writes it to its output. The Ctrl-D (end-of-file, or EOF) key combination is used to tell cat that there is no additional input. • Display the contents of a file with the concatenate command • cat [options] [file] • -n precede each line with a line number • -v display non-printing characters, except tabs, new-lines, and form-feeds • -e display $ at the end of each line • more, less, and pg - page through a file • more, less, and pg let you page through the contents of a file one screenful at a time
  • 106. File display options • more [options] [+/pattern] [filename] • less [options] [+/pattern] [filename] • pg [options] [+/pattern] [filename] • head - display the start of a file • head displays the head, or start, of the file. • head [options] file • -n number number of lines to display, counting from the top of the file • tail - display the end of a file • tail displays the tail, or end, of the file. tail [options] file • -number number of lines to display, counting from the bottom of the file
  • 107. The head command Command by default displays first 10 lines of the file. The general command line format is: $ head [-n] <file> <Enter> Example: head ae_socket.c  Will display first 10 lines head –5 ae_socket.c  Will display first 5 lines
  • 108. The tail command • tail Command by default displays last 10 lines of the file. The general command line format is: $ tail [+/-n] <file> <Enter> Example: tail ae_socket.c  Will display last 10 lines tail –5 ae_socket.c  Will display last 5 lines tail +5 ae_socket.c  Will display lines starting from 5th line till end of file
  • 109. Compare files • cmp: Comparing Two Files • comm: What is Common? • diff: The diff command outputs the difference, if any, between two text files. > diff before.txt after.txt
  • 110. Compare and Contrast • comm Display common lines in two files. Produces a three-column output: Column 1: Lines that appear only in file1, Column 2: Lines that appear only in file2, Column 3: Lines that appear in both files. The general command line format is: $ comm [-[1] [2] [3]] file1 file2
  • 111. • cmp • Compare two files. • No comment if the files are the same. • If they differ, it announces the byte and line number at which the difference occurred. The general command line format is: $ cmp file1 file2
  • 112. • diff: • Compare directories: –diff sorts the contents of the directories by name. –then runs the regular file diff algorithm on text files that have the same name in each directory but are different. • Compare file: –diff tells what lines must be changed in the files to bring them into agreement. –Option: • -b: ignores trailing blanks (spaces and tabs) and other strings. • -i: ignore uppercase and lower case differences. • -w: ignore all white spaces.
  • 113. Compression archive files • gzip and gunzip: Compressing and Decompressing Files • tar: The Archival Program • zip and unzip: Compressing and Archiving Together • In addition to tar, the zip and unzip command line tool scan be very handy. To see a zip file's content: > unzip -v • Combine with grep to find files whose names end in “html”: > unzip -v | grep html$ • To create a zip file in a backup directory, containing the current directory and all its subdirectories, do: >zip -r ~/bu/ *
  • 114. File system in UNIX • Linux supports the following file systems: minix, ext, ext2, xia, msdos, umsdos, vfat, proc, nfs, iso9660, hpfs, sysv, sm b, ncpfs • man fs • File name no longer than 256 char • Single path name should not exceed 1023 • mount /dev/hda/usr • unmount • File types: • Binary – Gif,jpeg, png • Text – Shell scripts, documents
  • 115. File Security Chmod, chown, chgrp, umask
  • 116. File Permissions 4bits type r w xr w x r w xu | g | s owner group others sticky sgid suid 4 2 1 4 2 1 4 2 1 - file d directory s local socket l symbolic link b block special file c char special file p named piped file
  • 117. Changing Permissions • chmod • chown • chgrp chmod: • Used to change the permissions of files and directory. • Only the owner or super user can use chmod to change the permissions of a file. For example: to remove read write and execute permissions on the file ae_socket.c for the group and others, type: $ chmod go-rwx ae_socket.c
  • 118. Options with chmod.. Symbol Meaning u user g group o other a all r read w write (and delete) x execute (and access directory) + add permission - take away permission
  • 119. Ownership and Permissions • Group and Ownership • chown, chgrp – chown user:group file • sudo chown • Permissions of files and directories can be added with + and revoked with – with access specifier • chmod 755 file • chmod +x file • chmod -x file • (r)ead (w)rite e(x)ecute 4 2 1
  • 120. Using numbers • 1 - Execute • 2 - Write • 4 – Read Example: Original permission for ae_socket.c: -rw----rw- Thus in order to give read write permission to owner, group and only read permission to others we would give: Using character modes: chmod o-w 1.c chmod g+rw 1.c Using numbers: chmod 664 ae_socket.c
  • 121. Common Permissions with Unmask • Applying umask to file
  • 122. The chmod command • chmod -R a+rX linux/ Makes linux directory content and everything in it available to everyone! • R: apply changes recursively • X: x, but only for directories and files already executable Very useful to open recursive access to directories, without adding execution rights to all files.
  • 123. Chmod contd… • chmod a+t /tmp • t: (sticky). Special permission for directories, allowing only the directory and file owner to delete a file in a directory. • Useful for directories with write access to anyone, like /tmp. • Displayed by ls -l with a t character.
  • 124. Groups and users • Users, su, sudo • su, sudo, visudo, /etc/sudoers, su user • sudo bash • /etc/passwd
  • 125. Symbolic link • Find yourself often navigating several levels deep, many times? Create a symbolic link: ln -s work/yadameter/biz/bbsinc/yadameter y • When you don‟t need it anymore, just remove the link with rm command: rm y • The directory pointed to will remain untouched; only the link will be deleted. i Interactive s Symbolic f forcefully
  • 126. • All links to a file access the same physical file. • Modifying contents from any access changes the file. • Links can be of two type: – Hard link. • Created using ln <old file> <link name> • Cannot create a link to directory. • Can not span file systems. – Soft Link. • Created using ln –s <old file> <link name> • Can also create link to directory. • Can span file systems.
  • 127. File Filters Pr, more, less, tail, head, sort, uniq
  • 128. File filters • The Sample Database • pr: Paginating Files • head: Displaying the Beginning of a File • tail: Displaying the End of a File • cut: Slitting a File Vertically • paste: Pasting Files • sort: Ordering a File • uniq: Locate Repeated and Nonrepeated Lines • tr: Translating characters • An example: Displaying a Word-count List
  • 129. The cut filter • select parts of a line • The cut command allows a portion of a file to be extracted for another use. • cut [options] file • -c character_list character positions to select (first character • -d delimiter field delimiter (defaults to <TAB>) • -f field_list fields to select (first field is 1 • Both the character and field lists may contain comma-separated or blank- character-separated numbers (in increasing order), and may contain a hyphen (-) to indicate a range. Any numbers missing at either before (e.g. - 5) or after (e.g. 5-) • the hyphen indicates the full range starting with the first, or ending with the last character or field, respectively. Blank-character-separated lists must be enclosed in quotes. The field delimiter should be enclosed in quotes • is 1
  • 130. cut - Example To display the employee code, firstname and basic salary from the employee file created earlier. $ cut -d" " -f2,3,8 emp_new.txt Where: -d is used to specify the field delimiter. A space in this case. -f represents fields and the numbers after it are the field positions. The specified columns are copied to the standard output.
  • 131. cut - Examples • To display characters 4 to 9 both inclusive, from each record of the file employee: $ cut -c4-9 emp_new.txt • To display from the second to the fourth fields: $ cut -d" " -f2-4 emp_new.txt • To display from the fourth field onwards: $ cut -d" " -f4- emp_new.txt
  • 132. The paste filter • paste - merge files • The paste command allows two files to be combined side-by-side. The default delimiter between the columns in a paste is a tab, but options allow other delimiters to be used. Syntax • paste [options] file1 file2 • Common Options • -d list of delimiting characters • -s concatenate lines
  • 133. tr - translate characters • The tr command translates characters from stdin to stdout. Syntax • tr [options] string1 [string2] • With no options the characters in string1 are translated into the characters in string2, character by character in the string arrays. The first character in string1 is translated into the first character in string2, etc. • A range of characters in a string is specified with a hyphen between the upper and lower characters of the range, e.g. to specify all lower case alphabetic characters use '[a-z]'.
  • 134. Sort file contents • The sort command is used to order the lines of a file. Without any options, the sort compares entire lines in the file and outputs them in ASCII order • sort [options] [+pos1 [ -pos2 ]] file – -k keydef sort on the defined keys (not available on all systems) – -i ignore non-printable characters – -n numeric sort – -b ignore leading blanks (<space> & <tab>) when determining starting and ending characters for the sort key – -d dictionary order, only letters, digits, <space> and <tab> are significant – -f fold upper case to lower case – -o outfile output file – -r reverse the sort – -t char use char as the field separator character – -u unique omit multiple copies of the same line
  • 135. The tee filter • tee - copy command output • tee sends standard in to specified files and also to standard out. It's often used in command pipelines. tee [options] [file[s]] • -a append the output to the files • -i ignore interrupts
  • 136. The uniq command • uniq filters duplicate adjacent lines from a file. Syntax • uniq [options] [+|-n] file [] • -d one copy of only the repeated lines • -u select only the lines not repeated • +n ignore the first n characters • -s n same as above (SVR4 only) • -n skip the first n fields, including any blanks
  • 137. Word count wc • wc - count words in a file • wc stands for "word count"; the command can be used to count the number of lines, characters, or words in a file. Syntax • Common Options • -c count bytes • -m count characters (SVR4) • -l count lines • -w count words
  • 138. Regular Expressions Pattern matching, text filters, grep, find, awk, sed
  • 139. Regular Expressions • Regular Expression Overview • RE Character Classes • RE Quantifiers • RE Parenthesis • RE Examples with grep • Re in awk • RE in sed
  • 140. Regex overview • A regular expression (abbreviated as regexp or regex, with plural forms regexps, regexes, or regexen) is a string that describes or matches a set of strings, according to certain syntax rules. • Syntax – ^ Matches the start of the line – $ Matches the end of the line – . Matches any single character – [] Matches a single character that is contained within the brackets – [^] Matches a single character that is not contained within the brackets – () Defines a "marked subexpression” – {x,y}Match the last "block" at least x and not more than y times
  • 141. How regex works? • Examples of regular expressions: – ".at" matches any three-character string like hat, cat or bat – "[hc]at" matches hat and cat – "[^b]at" matches all the matched strings from the regex ".at" except bat – "^[hc]at" matches hat and cat but only at the beginning of a line – "[hc]at$" matches hat and cat but only at the end of a line
  • 142. Pattern match class • POSIX class similar to meaning • [:upper:] [A-Z] uppercase letters • [:lower:] [a-z] lowercase letters • [:alpha:] [A-Za-z] upper- and lowercase letters • [:alnum:] [A-Za-z0-9] digits, upper- and lowercase letters • [:digit:] [0-9] digits • [:xdigit:] [0-9A-Fa-f] hexadecimal digits • [:punct:] [.,!?:...] punctuation • [:blank:] [ t] space and TAB characters only • [:space:] [ tnrfv]blank (whitespace) characters • [:cntrl:] control characters • [:graph:] [^ tnrfv] printed characters • [:print:] [^tnrfv] printed characters and space • Example: [[:upper:]ab] should only match the uppercase letters and lowercase 'a' and 'b'.
  • 143. Grep • grep print lines matching a pattern (General Regular Expression Parser) grep [options] PATTERN [FILES] option -c print number of output context -E Interpret PATTERN as an extended regular expression -h Supress the prefixing of filenames -i ignore case -l surpress normal output -v invert the sense of matching % grep in words.txt % grep –c in words.txt words2.txt % grep –c –v in words.txt words2.txt
  • 144. Grep usage samples $ grep <string> <file or directory> $ grep 'new FooDao' $ grep Account *.xml $ grep –r 'Dao[Impl|Mock]' src • Quote string if spaces or regex. • Recursive flag is typical • Don't quote filename with wildcards!
  • 145. Common grep options Case-insensitive search: $ grep –i foo bar.txt Only find word matches: $ grep –rw foo src Display line number: $ grep –nr 'new Foo()' src
  • 146. Filtering results Inverted search: $ grep –v foo bar.txt Prints lines not containing foo. Typical use: $ grep –r User src | grep –v svn Using find … | xargs, grep … is faster.
  • 147. More grep options Search for multiple terms: $ grep -e foo –e bar appdata.txt Find surrounding lines: $ grep –r –C 2 foo logs Similarly –A or –B will print lines before and after the line containing match.
  • 148. G‟RE‟P • Unix great utility is grep it uses regular expression • Search for lines ending with “e” % grep e$ words2.txt • Search for “a” % grep a[[:blank:]] word2.txt • Search for words starting with “Th.” % grep Th.[[:blank:]] words2.txt • Search for lines with 10 lower case characters % grep –E [a-z]{10} words2.txt 148 shell programming regex
  • 149. Grep summary • -r recursive search • -i case insensitive • -w whole word • -n line number • -e multiple searches • -A After • -B Before • -C Centered
  • 150. Unix Find NAME, Types, SIZE, MTIME, CTIME, MMIN, PERM Command
  • 151. Basic find examples $ find . –name $ find /etc –name '*.conf' $ find . –name '*.xml' $ find . -not -name '*.java' -maxdepth 4 $ find . (-name '*jsp' –o –name '*xml') • -iname case-insensitive • ! == -not • Quotes keep shell from expanding wildcards.
  • 152. Find and do stuff $ find . –name '*.java' | xargs wc –l | sort Other options: $ find . –name '*.java' -exec wc –l {} ; | sort $ find . –name '*.java' -exec wc –l {} + | sort Use your imagination. mv, rm, cp, chmod . . .
  • 153. -exec vs | xargs • -exec has crazy syntax. • | xargs fits Unix philosophy. • ; is slow, executes command once for each line. • ; not sensible, sorts 'alphabetically.' • | xargs may fail with filenames containing whitespace, quotes or slashes.
  • 154. Find by type Files: $ find . –type f Directories: $ find . –type d Links: $ find . –type l
  • 155. Find by modification time Changed within day: $ find . –mtime -1 Changed within minute: $ find . –mmin -15 Variants –ctime, -cmin, -atime, -amin aren't especially useful.
  • 156. Find Latest file Compare to file $ find . –newer foo.txt $ find . ! –newer foo.txt
  • 157. Find by permissions $ find . –perm 644 $ find . –perm –u=w $ find . –perm –ug=w $ find . –perm –o=x
  • 158. Find by size Less than 1 kB: $ find . –size -1k More than 100MB: $ find . –size +100M
  • 159. Process Management Process Life, termination, scheduling jobs
  • 160. Process management • Process monitoring • Mechanism of Process Creation • Internal and External commands • Process States and Zombies • Running Jobs in Background • nice: Job execution with Low Priority • Killing Processes with Signals • Job Control • at and batch: Execute Later • cron: Running Jobs Periodically
  • 161. View the Processes • ps: A process is an executing program identified by a unique PID (process identifier). • To see information about your processes, with their associated PID and status, type: – $ ps Options: -e Select all processes. -f Show columns user, pid, ppid, cpu, stime, tty, time, and args, in that order. -l Show columns flags, state, uid, pid, ppid, cpu, intpri, nice, addr, sz, wchan, tty, time, and comm, in that order. -x Shows the command line in extended format.
  • 162. Process Life • Kill command will pass the signals
  • 163. Background jobs • To background a process, type an & at the end of the command line. For example, the command sleep waits a given number of seconds before continuing. Type $ sleep 10 • This will wait 10 seconds before returning the command prompt $. Until the command prompt is returned, you can do nothing except wait. To run sleep in the background, type: $ sleep 10 & • The & runs the job in the background and returns the prompt straight away, allowing you do run other programs while waiting for that one to finish.
  • 164. • nohup: The Kernel terminates all active processes by sending a hang-up signal at the time of logout. The nohup command is used to continue execution of a process even after the user has logged out. Syntax: nohup <command> <Enter> • Any command that is specified with nohup, will continue running even after the user has logged out.
  • 165. • Chaining multiple commands: $ command1;command2;command3 • You can suspend the process running in the foreground by holding down the [control] key and typing [z] (written as ^Z) Then to put it in the background, type $ bg Note: background programs that do not require user interaction • top: Display and update information about the top processes on the system.
  • 166. • To see the background jobs running in the terminal: $ jobs • To restart (foreground) a suspended processes, type $ fg %jobnumber • For example, to restart sleep 100, type $ fg %1 • Typing fg with no job number foregrounds the last suspended process.
  • 167. kill – Terminate Process • Occasionally a need may arise to stop the process adhoc • This is done by sending the software, a termination signal to kill the process. $ kill <pid> • Some programs are designed to ignore the interrupts. – We can request the kill command to send a sure kill signal. – The sure kill signal is represented by including a minus nine option (-9) along with the kill command. $ kill –9 1020 • Only the owner of the process or the super user can kill a process.
  • 168. Monitoring processes • ps • ps –ef • ps –u oracle • ps –C sshd • man ps
  • 169. Cron Execution example
  • 170. Scheduling Jobs % crontab –l # listing schedular % crontab –e # editing cron 0 0 * * * 0 * * * * * * * * * 0 1 * * 0 % EDITOR=vi crontab –e % man 5 crontab
  • 171. Resource Management Disk space, RAM, Swap, top, uptime
  • 172. Resource Management • Disk file usage : df • Diskspace utilization : du • Memory space with top, vmstat • Swap space with free • CPU uptime
  • 173. Disk file usage • df [options] [resource] • df - summarize disk block and file usage • df is used to report the number of disk blocks and i-nodes used and free for each file system. The output format and valid options are very specific to the OS and program version in use. • -l local file systems • -k kilobytes • -h human readable form output
  • 174. The df example • df Reports on the space left on the file system. Command Format: df [options] [special directory] Example: $ df –k ~  Will print information about the home directory. • bdf Developed by Berkeley. It displays the free size in a more presentable manner.
  • 175. Disk space utilization • du - report disk space in use • du reports the amount of disk space in use for the files or directories you specify. • du [options] [directory or file] • -a display disk usage for each file, not just subdirectories • -s display a summary total only • -k report in kilobytes (SVR4) Outputs the number of kilobyes used by each subdirectory. Useful if you have gone over quota and you want to find out which directory has the most files. Example: $ du -a
  • 176. RAM size with free -m • The free -m
  • 177. The cpu load average • The uptime command will give you load averages on CPU
  • 178. Network telnet, ftp, finger, ssh, scp, rsync
  • 179. Network • telnet and ftp • Communication check : finger • Remote files : scp, rcp, rsync • Download from command wget • Upload from command curl • Email alerts • Remote script : ssh • Password less connectivity
  • 180. telnet and ftp • TELNET and FTP are Application Level Internet protocols. The TELNET and FTP protocol specifications have been implemented by many different sources, including The National Center for Supercomputer Applications (NCSA), and many other public domain and shareware sources which are also TELNET protocol implementations – telnet [options] [ remote_host [ port] ] – ftp [options] [ remote_host ]
  • 181. ftp options • ftp telnet Action • -d set debugging mode on • -d same as above (SVR4 only) • -i turn off interactive prompting • -n don't attempt auto-login on connection • -v verbose mode on • -l user connect with username, user, on the remote host
  • 182. Example of ftp ftp –n –u <<_FTP_ user username password put FILE _FTP_
  • 183. Download from command • wget • wget –r
  • 184. Upload with curl, ftp curl –T upload-file -u username:password
  • 185. Communication finger • finger displays the .plan file of a specific user, or reports who is logged into a specific machine. The user must allow general read permission on the .plan file. • finger [options] [user[@hostname]] • Common Options • -l force long output format • -m match username only, not first or last names • -s force short output format
  • 186. Remote script SSH • The Secure SHell (SSH) versions of the rcp, rsh, and rlogin programs are freely available and provide much greater security Command Format: $ rlogin <host name> -l <user name> $ rlogin baileys -l snadmin • Password less ssh connection to remote server – ssh –t rsa
  • 187. Email • Email Notification % echo “Message” | mail –s “Here’s your message”
  • 188. Smart directory copy with rsyncrsync (remote sync) has been designed to keep in sync directories on 2 machines with a low bandwidth connection. Only copies files that have changed. Files with the same size are compared by checksums. Only transfers the blocks that differ within a file! Can compress the transferred blocks Preserves symbolic links and file permissions: also very useful for copies on the same machine. Can work through ssh (secure remote shell). Very useful to update the contents of a website, for example.
  • 189. rsync examples (1) rsync -a /home/arvin/sd6_agents/ /home/sydney/misc/ -a: archive mode. Equivalent to -rlptgoD... easy way to tell you want recursion and want to preserve almost everything. rsync -Pav --delete /home/steve/ideas/ /home/bill/my_ideas/ -P: --partial (keep partially transferred files) and --progress (show progress during transfer) --delete: delete files in the target which don't exist in the source. Caution: directory names should end with / . Otherwise, you get a my_ideas/ideas/ directory at the destination.
  • 190. rsync examples (2) Copying to a remote machine rsync -Pav /home/bill/legal/arguments/ User bill will be prompted for a password. Copying from a remote machine through ssh rsync -Pav -e ssh fridge/homer/beer/ User homer will be prompted for his ssh key password.
  • 191. Q & A • Summary • References – – • Feature Automations skills – AWK – SED – PERL – Python