Your SlideShare is downloading. ×
Unix  introduction
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Saving this for later?

Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime - even offline.

Text the download link to your phone

Standard text messaging rates apply

Unix introduction

877
views

Published on


0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
877
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
18
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Introduction to UNIX UNIX Power User Training Unix Courseware Version 1.2 1 System Consultant Introduction to UNIX 2Copyright Department of Computer Science,Northern Illinois University, 2004 01-1
  • 2. Introduction to UNIX What is an Operating System? • Software that manages (allocates and de-allocates) system resources in an efficient and secure manner • System resources consist of hardware (e.g. terminals, printers, storage device, etc) and software (e.g. application programs, language libraries, etc.) 3 System Resources System Resources Hardware Software Components System Application Software Software 4Copyright Department of Computer Science,Northern Illinois University, 2004 01-2
  • 3. Introduction to UNIX Different Types of Operating Systems • Single-user, single-process operating systems: allow only one user at a time to use the computer system. The user can execute/run only one process at a time. – Examples: DOS, Windows 3.1 • Single-user, multi-process operating systems: allow a single user to use the computer system; however, the user can run multiple processes at the same time. – Example: OS/2 5 Different Types of Operating Systems • Multi-user, multi-process operating systems: allow multiple users to use the computer system simultaneously. Each user can run multiple processes at the same time. – Examples: UNIX, Windows XP 6Copyright Department of Computer Science,Northern Illinois University, 2004 01-3
  • 4. Introduction to UNIX UNIX Operating System • UNIX is a multi-user, multi-processing, portable(?) operating system. • UNIX is designed to facilitate programming, text processing and communication 7 Components of UNIX: perspective 1 8Copyright Department of Computer Science,Northern Illinois University, 2004 01-4
  • 5. Introduction to UNIX Components of UNIX: perspective 2 9 History of UNIX • Invented by Ken Thompson at AT&T in 1969 • First version written in assembly language – single user system, no network capability • Thompson, Dennis Ritchie, Brian Kernighan – rewrote Unix in C: processor/architecture independent • Unix evolution: I. Bell Labs, USL, Novell, SCO II. AIX, Ultrix, Irix, Solaris, … III. BSD, FreeBSD, Mach, OS X IV. Linux, Redhat, Suse, … 10Copyright Department of Computer Science,Northern Illinois University, 2004 01-5
  • 6. Introduction to UNIX What Can UNIX Do for You? • User’s Support Tools – Text processing (vi, sed, awk) – Filing system – E-mail and networking – Electronic databases • Programmer’s Support Tools – Programming languages & compilers (C, C++) – Shell scripts – Programming Workbench • Source Code Control System (SCCS) • Revision Control System (RCS) 11 Some Standard UNIX Shells 12Copyright Department of Computer Science,Northern Illinois University, 2004 01-6
  • 7. Introduction to UNIX Command Source and Destination 13 Command Line Structure mp% command [-options] [arguments] Arguments can be: Command Command Command modifier; 1. More information prompt name usually one character 2. Object identifiers preceded by + or - sign 3. Names of files Notes: •UNIX is case sensitive!!!! •Example: the command “ls –l” is not the same as “LS –L” • Must be a space between the command, options and arguments • No space between the plus or minus sign and the option letter • Option letters must be typed exactly as they are indicated, uppercase or lowercase •Fields enclosed in [ ] are optional •Must press [Return] after you have completed entry of a command 14Copyright Department of Computer Science,Northern Illinois University, 2004 01-7
  • 8. Introduction to UNIX Correcting Mistakes Note: UNIX is case sensitive (use lowercase) Key Pressed Result Backspace, Ctrl-h Back up & erase last character Ctrl-c Terminates the current command Ctrl-r Redraws the current command line Ctrl-s Stops scrolling of output on screen (Ctrl-q to resume/start scrolling) Ctrl-w Erases a word on command line Ctrl-u Erases/deletes entire command line 15 Command Line Structure • How do the results of the following commands differ? • According to the synopsis of the “sort” command, what parts of the “sort” command are required? sort list Command sort –f list argument sort –o sorted list Command Command Option name option argument 16Copyright Department of Computer Science,Northern Illinois University, 2004 01-8
  • 9. Introduction to UNIX The UNIX File System 17 A Directory Hierarchy 18Copyright Department of Computer Science,Northern Illinois University, 2004 01-9
  • 10. Introduction to UNIX Sample UNIX File Structure (NIU, CS) UNIX supports “tree-like” directory structure Root Directory Root subdirectories dev etc home usr tty null skel mp ux bin local ucb Home Directory z036473 Files in the z036473 csci330 .cshrc .logout z036473 directory subdirectory 19 Directory Types • Root Directory: / – The first directory in any UNIX file structure – Always begin with the forward slash (/) • Home Directory: $HOME or ~ • Created by system administrator • This is where you are when you first log in! • Under $HOME, you may create your own directory structure • Type: cd [Return] takes you $HOME • Current Working Directory: . – The Directory you are currently working in – Also called Current Working Directory (cwd) • Parent Directory: .. – The directory immediately above your current working directory. 20Copyright Department of Computer Science,Northern Illinois University, 2004 01-10
  • 11. Introduction to UNIX File Types - Text - Root (/) - Binary - Home (~) - Working (.) - Parent (..) 21 Paths and Pathnames Two ways of locating a file or a directory: • By Using Absolute Pathname – Full pathname – Traces a path from root to a file or a directory – Always begins with the root (/) directory! – Example: /home/ux/krush/unix/assignments/assign1.sp04 • By Using Relative Pathname – Traces a path from the ‘cwd’ to a file or a directory – No initial forward slash (/) – Two dots (..) goes up one level on file structure – Dot (.) points to current working directory (cwd) – Example: unix/assignments/assign1.sp04 22Copyright Department of Computer Science,Northern Illinois University, 2004 01-11
  • 12. Introduction to UNIX Relative Pathnames for file3 Absolute Pathname: /usr/staff/joan/file3 23 Directory Operations 24Copyright Department of Computer Science,Northern Illinois University, 2004 01-12
  • 13. Introduction to UNIX Display Current Directory’s Full Pathname • To determine the full pathname of the current working directory, use the command named “pwd” • pwd stands for print working directory Example: To display the full pathname of the current working directory ux% pwd /home/ux/krush/unix 25 The ls Command 26Copyright Department of Computer Science,Northern Illinois University, 2004 01-13
  • 14. Introduction to UNIX Long List Option 27 List ALL Contents of Current Directory ux% ls -la List contents of the current . Points to cwd, total 126 directory in long format .. Points to parent’s drwxr-xr-x 13 krush csci 1024 Apr 26 15:49 . dir. drwxr-xr-x 15 root root 512 Apr 24 15:18 .. File names -rwx------ 1 krush csci 1120 Apr 12 13:11 .cshrc begins with -rwxr--r-- 1 krush csci 885 Dec 2 13:07 .login a dot (.) are -rw-r--r-- 1 krush csci 141 Mar 14 13:42 .logout hidden files --rwx------ 1 krush csci 436 Apr 12 11:59 .profile drwx------ 7 krush csci 512 May 17 14:11 330 Directories drwx------ 3 krush csci 512 Mar 19 13:31 467 drwx------ 2 krush csci 512 Mar 31 10:16 Data -rw-r--r-- 1 krush csci 80 Feb 27 12:23 crontab.cron Regular text file 28Copyright Department of Computer Science,Northern Illinois University, 2004 01-14
  • 15. Introduction to UNIX List Contents of a Specific Directory Listing contents of a subdirectory named ux% ls -l unix/grades “unix/grades” total 10 -rwxr-xr-x 3 krush csci 72 Jan 19 19:12 330assign-graderun -rwxr-xr-x 1 krush csci 70 Jan 19 19:13 330exam-graderun -rwxr-xr-x 2 krush csci 70 Jan 19 19:12 330quiz-graderun -r-x------ 1 krush csci 468 Feb 1 11:55 test-330grade -r-x------ 1 krush csci 664 Feb 1 11:55 test-330grade,v 29 File Name Expansion & Wildcards Allows you to select files that satisfy a particular name pattern (wildcards) Character Description Example * Match zero or more char. ls *.c ? Match any single character ls conf.? [list] Match any single character in list ls conf.[co] [lower-upper] Match any character in range ls lib-id[3-7].o str{str1,str2,…} Expand str with contents of { } ls c*.{700,300} 30Copyright Department of Computer Science,Northern Illinois University, 2004 01-15
  • 16. Introduction to UNIX The mkdir Command 31 Creating a New Directory • To create a directory, use the command named “mkdir” Example: To create two new directories called “csci330” and “test-data” ux% mkdir csci330 test-data ux% mkdir /home/ux/krush/unix/demo must already exist 32Copyright Department of Computer Science,Northern Illinois University, 2004 01-16
  • 17. Introduction to UNIX Directory Names • Use the following characters: – Uppercase letters (A-Z) – Lowercase letters (a-z) – Numbers (0-9) – Underscore ( _ ) – Period/dot ( . ) 33 Directory Names • When naming a directory, avoid the following characters: & * | [] {} $ <> () # ? / “ ‘ ; ^ ! ~ Space Tab 34Copyright Department of Computer Science,Northern Illinois University, 2004 01-17
  • 18. Introduction to UNIX Example: Create a Directory Creation dev etc home usr tty null skel mp ux bin local ucb z036473 You are here csci330 .cshrc .logout Temp Data Create a directory called Data under csci330 a) Using Absolute Pathname: mkdir /home/mp/z036473/csci330/Data b) Using Relative Pathname: 35 mkdir csci330/Data The cd Command 36Copyright Department of Computer Science,Northern Illinois University, 2004 01-18
  • 19. Introduction to UNIX Changing Directory dev etc home usr tty null skel mp ux bin local ucb z036473 csci330 .cshrc .logout Temp You are here Data In the Data directory, go to $HOME directory a) Using Absolute Pathname: cd /home/mp/z036473 b) Using Relative Pathname: 37 cd $home cd ../.. cd cd ~ cd ~z036473 Remove Directories • To remove an empty directory – a directory that does not contain user-created files, use the command named “rmdir” Example: To remove a directory called “test”, which does not contain user-created files. ux% rmdir test • To remove a non-empty directory, use the command named “rm –r” Example: To remove a non-empty directory called “old- data” ux% rm –r old-data 38Copyright Department of Computer Science,Northern Illinois University, 2004 01-19
  • 20. Introduction to UNIX Operations Common to Directories and Regular Files 39 Copying Files • To copy a file, use the command named “cp” • Syntax: cp source-file new-file • Commonly used options: -i if “new-file” exists, the command cp prompts for confirmation before overwriting -p preserve permissions and modification times -r recursively copy files and subdirectories 40Copyright Department of Computer Science,Northern Illinois University, 2004 01-20
  • 21. Introduction to UNIX Copying Files • “source-file” must have read permission. • The directory that contains “source-file” must have execute permission. • The directory that contains “new-file” must have write and execute permissions. • Note that if “new-file” exists, you do not need the write permission to the directory that contains it, but you must have the write permission to “new- file”. 41 Moving Files • To move files from one directory to another directory, or to re-name a file, use the command named “mv”. • The directory that contains the source file and the destination directory must have write and execute access permissions. 42Copyright Department of Computer Science,Northern Illinois University, 2004 01-21
  • 22. Introduction to UNIX Moving Files • Syntax: mv source-file destination-file • If the destination file exists, “mv” will not overwrite exiting file. Example: Move “assign1.txt” a different directory and rename it to “assign1.save” ux% mv assign1.txt $HOME/archive/assign1.save ux% mv assign1.txt $HOME/archive 43 Moving a File 44Copyright Department of Computer Science,Northern Illinois University, 2004 01-22
  • 23. Introduction to UNIX Rename Directories • To change the name of an existing directory, use the command named “mv” Example: To rename the file called “unix” to “csci330” ux% mv unix csci330 • For the above example, what happens if “csci330” already exists in the current directory and it is the name of a directory? 45 The mv Command 46Copyright Department of Computer Science,Northern Illinois University, 2004 01-23
  • 24. Introduction to UNIX Removing/Deleting Files • You should remove un-needed files to free up disk space. • To remove/delete files, use the command named “rm”. • Syntax: rm file-list • Commonly used options: -f force remove regardless of permissions for “file-list” -i prompt for confirmation before removing -r removes everything under the indicated directory 47 Removing/Deleting Files • If “file-list” contains pathname, the directory components of the pathname must have execute permission. • The last directory that contains the file to be deleted must have execute and write permissions. Example: Remove the file named “old-assign” ux% rm unix/assign/old-assign 48Copyright Department of Computer Science,Northern Illinois University, 2004 01-24
  • 25. Introduction to UNIX Recap: Common Operations on Files 49 Finding Files • The command named “find” can be used to locate a file or a directory. • Syntax: find pathname-list expression • “find” recursively descends through pathname-list and applies expression to every file. • For syntax of expression, see Course Notes pp. 13- 9 50Copyright Department of Computer Science,Northern Illinois University, 2004 01-25
  • 26. Introduction to UNIX Finding Files Example 1: Find all files, in your directory hierarchy, that have a name ending with “.bak”. ux% find $home –name “*.bak” –print Example 2: Find all files, in your directory hierarchy, that were modified yesterday. ux% find $home –mtime –1 -print 51 The “ln” command • Allows file to listed in multiple directories • 2 types: – Hard link – Symbolic link • First: understand Unix file storage and organization 52Copyright Department of Computer Science,Northern Illinois University, 2004 01-26
  • 27. Introduction to UNIX Unix file organization • Computer has one or more physical hard drives • Hard drive is divided into partitions • Partition holds file system – File system is set of data blocks – Data blocks contain • general information • actual file data • directory information 53 Blocks in a file system 54Copyright Department of Computer Science,Northern Illinois University, 2004 01-27
  • 28. Introduction to UNIX inode • Index (or information) node: one inode per file • Each inode has unique number • contents: – File type, access permissions, link count – UID, GID – Date and time of the file’s last • Data access (read and execute) • Data modification (written) • I-node modification (permission change) – Data blocks assigned to the file 55 Inodes in a filesystem 56Copyright Department of Computer Science,Northern Illinois University, 2004 01-28
  • 29. Introduction to UNIX inode Contents: where is the file data ? Inode may store: – 10 addresses of data blocks that belong to file – 1 address of a block that contains data block addresses – 1 address of a block that contains addresses of blocks that contain data block addresses – 1 address of a block that contains addresses of blocks that contain addresses of blocks that contain data block addresses 57 I-node Structure I-node blocks blocks Access, Links, and other information 1 2 blocks blocks . . indirect 9 block 10 11 double indirect block blocks 12 blocks 13 blocks triple indirect 58 blockCopyright Department of Computer Science,Northern Illinois University, 2004 01-29
  • 30. Introduction to UNIX Directory representation Directory is a file: – Has inode like regular file, but different file type – Data blocks of directory contains simple table: Name Inode number 59 Example structure . I-node list 2763 Contents of dir1 2764 1076 . 2765 2083 . .. myfile 2764 . Data blocks on disk 60Copyright Department of Computer Science,Northern Illinois University, 2004 01-30
  • 31. Introduction to UNIX Example: user view vs. system view 61 Output: ls -li ux% ls -li crontab.cron 118282 -rw-r--r-- 1 krush csci 80 Feb 27 12:23 crontab.cron I-node 62Copyright Department of Computer Science,Northern Illinois University, 2004 01-31
  • 32. Introduction to UNIX Howto share Files ? • Duplicate shared files. • Create common login name for members of the team. • Set appropriate access permissions on shared files. • Create common group for members of the team. • Share files via links. 63 Duplicate Shared Files • Make copies of shared files and give them to all members of the team. • The simplest way to share files. • Does not work well if members of the team work on the files at the same time. 64Copyright Department of Computer Science,Northern Illinois University, 2004 01-32
  • 33. Introduction to UNIX Create Common Login Name • All team members use a common login that gives them access to shared files. • This is a simple solution that works well if the size of the team is small and stable. • A disadvantage is team members have to use a separate account, not their regular account. 65 Set Appropriate Access Permissions • Team members put all shared files in one place and set access permissions so all team members can access them. • This scheme works well if all team members are in the same group (group permissions can be used). • A disadvantage is if the group used has other users in it, they will also have access to the files. 66Copyright Department of Computer Science,Northern Illinois University, 2004 01-33
  • 34. Introduction to UNIX Create Common Group • This solution is similar to setting appropriate permissions except a new group is created that contains just the members of the team. • All team members have individual logins. • This is an effective solution, especially if it is with version control. 67 Linking Files • To share a single file with multiple users, a link can be used. • A link is: – A reference to a file stored elsewhere on the system. – A way to establish a connection to a file to be shared. • Two types: – Hard link – Symbolic link (a.k.a. “soft link”) 68Copyright Department of Computer Science,Northern Illinois University, 2004 01-34
  • 35. Introduction to UNIX Hard Link Advantages Disadvantages Allow access to original file name via the file name or the I-node Cannot link to a file in a different number file system The original file continues to exist as long as at least one directory contains its I-node Prevents owner from truly deleting it, and it counts against his/her disk Checks for the existence of the quota original file 69 Hard Link home Syntax: ln shared-file link-name z036473 From dir3, link to the file ‘aa’ in dir1 name it ‘bb’: dir1 dir2 % ln /home/z036473/dir1/aa bb aa dir3 bb 70Copyright Department of Computer Science,Northern Illinois University, 2004 01-35
  • 36. Introduction to UNIX The ln Command 71 A Hard Link 72Copyright Department of Computer Science,Northern Illinois University, 2004 01-36
  • 37. Introduction to UNIX Hard Link Contents of dir1 . home 1076 . 2406 2083 2407 z036473 .. 2408 aa 2407 dir1 dir2 . aa Contents of dir3 . dir3 1070 bb . 2050 .. bb 2407 73 Symbolic Link Advantages Disadvantages Allow access to original file name Created without checking the existence of the shared file Can use either relative or absolute path to access the original file Cannot access the shared file if its path has restricted permissions Can cross partition and drives Can be circular linked to another Allows the creation of a link to a symbolic linked file directory 74Copyright Department of Computer Science,Northern Illinois University, 2004 01-37
  • 38. Introduction to UNIX Symbolic Link • A hard link may not be created for a file on a different file system • Use symbolic link • The linked files do not share the same I-node number Syntax: ln –s shared-file link-name Also called source-file Also called target-file 75 Symbolic Links to Different File Systems 76Copyright Department of Computer Science,Northern Illinois University, 2004 01-38
  • 39. Introduction to UNIX User’s Disk Quota • A disk quota is set for each user account • The command: quota –v displays the user’s disk usage and limits • 2 kinds of limits: – Soft limit: ex. 3MB • Maybe exceeded for one week • System will nag – Hard limit: ex. 4MB • Cannot be exceeded 77 Operations Common to Directories and Regular Files 78Copyright Department of Computer Science,Northern Illinois University, 2004 01-39
  • 40. Introduction to UNIX Operations Unique to Regular Files Display Create Edit Print Others Contents 79 Creating New Files Create Regular Files Redirect cat vi pico Command Output See Text Editors See the C shell Section 03 Section 06 80Copyright Department of Computer Science,Northern Illinois University, 2004 01-40
  • 41. Introduction to UNIX Creating A File With cat Example: mp% cat > myfile This is line 1 of input Line 2 of input ^d mp% 81 Editing Text Files Editing Text Files vi pico sed awk See Text Editors See course notes See course notes Section 03 section 08 section 10 82Copyright Department of Computer Science,Northern Illinois University, 2004 01-41
  • 42. Introduction to UNIX Displaying Contents of Text Files Display Text File contents cat more less pg head tail 83 The cat Command 84Copyright Department of Computer Science,Northern Illinois University, 2004 01-42
  • 43. Introduction to UNIX Viewing Contents of Text Files • The command named “cat” can be used to display/concatenate one or more files, displaying the output all at once. Example: Display the contents of a file called “assign1.txt”. ux% cat assign1.txt 85 Viewing Contents of Text Files • The commands named “more” and “less” can be used to display the contents of one or more files one page at a time. Space bar – to advance to next page b – to go back a page Enter Key – to advance to next line Example: Display the contents of a file called “assign1.txt” one page at a time. ux% more assign1.txt 86Copyright Department of Computer Science,Northern Illinois University, 2004 01-43
  • 44. Introduction to UNIX Viewing Contents of Text Files • The command named “head” can be used to display the beginning portion of indicated file(s); the default head size is 10 lines. Example: Display the first 20 lines of a file called “assign1.txt”. ux% head –20 assign1.txt 87 Viewing Contents of Text Files • The command named “tail” can be used to display the ending portion of indicated file(s); the default tail size is 10 lines. Example: Display the last 10 lines of a file called “assign1.txt”. ux% tail assign1.txt ux% tail –10 assign1.txt 88Copyright Department of Computer Science,Northern Illinois University, 2004 01-44
  • 45. Introduction to UNIX Other Operations Unique to Text Files Other File Operations Combine Extract Compare File Compress sort contents contents contents size contents Unique Encrypt/ lines decrypt 89 Combining Contents of Files • Method 1: To vertically concatenate the contents of two or more files, use the command named “cat” with output redirection (>). • Syntax: cat file-1 file-2 file-3 > all-file • “all-file” will contain the combined contents of file-1, file-2, and file-3 in top-down (vertical) fashion • See demo 90Copyright Department of Computer Science,Northern Illinois University, 2004 01-45
  • 46. Introduction to UNIX Combining Contents of Files • Method 2: To horizontally concatenate contents (columns/fields) of two or more files, use the command named “paste”. • Syntax: paste file-1 file-2 • See demo 91 Extracting Contents of Files • To extract one or more fields in a file, use the command named “cut”. Example: Extract the month and year fields from the output of the “date” command. ux% date Mon Feb 2 20:37:38 CST 2004 ux% date | cut -d -f2,6 Feb CST 92Copyright Department of Computer Science,Northern Illinois University, 2004 01-46
  • 47. Introduction to UNIX The cut Command 93 Comparing Files: comm • The command named “comm” can be used to compare lines that are common in two sorted files. • Syntax: comm [options] file-1 file-2 • The output contains three columns: – Column1 contains lines unique to file-1 – Column 2 contains lines unique to file-2 – Column 3 contains lines common to both files • See Demo 94Copyright Department of Computer Science,Northern Illinois University, 2004 01-47
  • 48. Introduction to UNIX Comparing Files: diff • The command “diff” can be used to compare two files line by line. • Syntax: diff [options] file-1 file-2 • If file-1 and file-2 are the same, no output is produced. • If file-1 and file-2 are not the same, diff reports a series of commands that can be used to convert the first file to the second file. • See Demo 95 Determining File Size • Recall: The “ls” command with the option –l gives the file size in bytes. • Use the command named “wc” to display the size of files as number of lines, words, and characters. • Syntax: wc file-list • Commonly used options: -l display the number of lines -w display the number of words -c display the number of characters • See demo 96Copyright Department of Computer Science,Northern Illinois University, 2004 01-48
  • 49. Introduction to UNIX The wc Command 97 Compress File Contents • The command named “compress” can be used to reduce the size of one or more files. • Syntax: compress file-name • To display compression percentage and the names of compressed files, use the option named –v. • The file extension .Z is automatically appended to file-name. • See Demo 98Copyright Department of Computer Science,Northern Illinois University, 2004 01-49
  • 50. Introduction to UNIX Viewing a Compressed File • The command named “zcat” can be used to display the contents of a compressed file in a readable format. Example: Display the contents of a compressed file called “customer-data.Z”. ux% zcat customer-data.Z 99 Un-compress File Contents • To uncompress one or more compressed files, used the command named “uncompress”. • Syntax: uncompress file-list Example: uncompress a compressed file called “customer-data.Z”. ux% uncompress customer-data.Z 100Copyright Department of Computer Science,Northern Illinois University, 2004 01-50
  • 51. Introduction to UNIX Sorting Files • To sort a text file in ascending or descending order, use the command named “sort”. • Syntax: sort [options] file-name • Commonly used options: -r sort in reverse order -n numeric sort +x [-y] specify a field as the sort key; skipping x fields and start sorting with field y -f consider lowercase and uppercase to be equivalent 101 The sort Command 102Copyright Department of Computer Science,Northern Illinois University, 2004 01-51
  • 52. Introduction to UNIX Fields Within A Line 103 Field Specifier Examples 104Copyright Department of Computer Science,Northern Illinois University, 2004 01-52
  • 53. Introduction to UNIX Global and Local Options 105 Removing Repeated Lines • The command named “uniq” can be used to remove repetitious lines from a sorted input file, sending unique (unrepeated) lines to standard output. • Syntax: uniq sorted-file-name • Commonly used options: -c place a count of repeated lines at beginning of each output line -d display the repeated lines -u display the lines that are not repeated 106Copyright Department of Computer Science,Northern Illinois University, 2004 01-53
  • 54. Introduction to UNIX The unique Command 107 Encrypting Files • Encryption is a process that transforms a file to an unreadable form. • The transformed file is called an “encrypted” file. • You encrypt files to prevent other users from reading their contents. • The command named “crypt” can be used to encrypt files. • Syntax: crypt key < original-file > encrypted file • See Demo 108Copyright Department of Computer Science,Northern Illinois University, 2004 01-54
  • 55. Introduction to UNIX Decrypting Files • The process of transforming an encrypted file to its original format is called “decryption”. • To decrypt an encrypted file, use the command named “crypt”. • Syntax: crypt key < encrypted-file > original-file • See Demo 109 UNIX Text Editors 110Copyright Department of Computer Science,Northern Illinois University, 2004 01-55
  • 56. Introduction to UNIX Editor Concepts • Editing – creating a new file and modifying an existing text file. • An editor – a utility that makes the editing task possible. • A text editor differs from a word processor in that it does not perform text formatting, such as bold, center, underline, etc. • A line editor – a utility that applies changes to a line or group of lines; two common line editors: ex and sed. • A screen editor – shows a whole screen of text at a time; we can move cursor or select part of text, search for text, etc. 111 The Vi Editor • A screen editor available on most UNIX systems. • When invoked, it copies the contents of a file to a memory space know as a work buffer. • All editing are applied to the contents in the work buffer. • If the file does not exist, an empty buffer is created. • When we exit vi, the work buffer is erased. • At exit time, we can do two things: 1. Quit without saving – the original contents remain unchanged. 2. Save the file – the original contents are replaced by the new version in the work buffer. 112Copyright Department of Computer Science,Northern Illinois University, 2004 01-56
  • 57. Introduction to UNIX Vi Buffers • Work Buffer – This is where vi performs all editing – Vi makes a copy of your file in the work buffer – Makes all changes to the copied version – Replaces original copy with edited copy when you save (:w) • General Purpose Buffer – Stores recent deleted, or copied text – Used by the ‘undo’ command to restore text • Name Buffers – 26 named buffers, each named by a letter of the alphabet – Used to store a different block of text to recall later 113 The UNIX vi Text Editor Esc Command Input Mode Mode Insert (i, I) Append (a, A), : Return Open (o, O) Change (c), Replace (r, R) Last-Line Mode 114Copyright Department of Computer Science,Northern Illinois University, 2004 01-57
  • 58. Introduction to UNIX Vi Mode • In vi, to find out what mode you are in, you need to create a vi start-up file (.exrc) • In your home directory or in your cwd directory, create “.exrc” file with contents: set showmode 115 File Names • When naming a file, avoid the following characters: & * | [] {} $ <> () # ? / “ ‘ ; ^ ! ~ Space Tab 116Copyright Department of Computer Science,Northern Illinois University, 2004 01-58
  • 59. Introduction to UNIX Commonly Used File Extensions Extension File Type .a An archive or library .c C program source .cc C++ program source .csh C shell script .f FORTRAN program source .sh Bourne shell script .bsh Bash shell script .ksh Korn Shell script 117 Commonly Used File Extensions Extension File Type .o Object file of compiled program .ps Postscript source .shar Shell archive .tar Tar archive .txt ASCII text file .Z Compressed file 118Copyright Department of Computer Science,Northern Illinois University, 2004 01-59
  • 60. Introduction to UNIX Recovering Text After a Crash • If the system crashes while you are editing a file using vi, you may be able to recover your text • The system sends you an e-mail telling you how to recover your file • To recover a file after a system crash: % vi –r filename 119 Vi Demo Demo 120Copyright Department of Computer Science,Northern Illinois University, 2004 01-60
  • 61. Introduction to UNIX Access Permissions 121 Terminology • A user – any one who has Unix account on the system. • Unix recognizes a user by a number called user id. • A super user: – has the maximum set of privileges in the system – also know as system administrator – can change the system – must have a lot of experience and training • Users can be organized into groups. • One or more users can belong to multiple groups. 122Copyright Department of Computer Science,Northern Illinois University, 2004 01-61
  • 62. Introduction to UNIX Users 123 Terminology • To find out group information, use the command named: groups user-id • Example: To find out what groups the user z036473 belongs to. ux% groups z036473 student csci467a csci330c • Information about groups is stored in the Network Information Service (NIS) file named group.org_dir. 124Copyright Department of Computer Science,Northern Illinois University, 2004 01-62
  • 63. Introduction to UNIX Security Levels • There are three levels of security in UNIX: system, directory and file. • System security – controlled by the system administrator, a super user. • Directory and file – controlled by the user who owned them. 125 Security Levels 126Copyright Department of Computer Science,Northern Illinois University, 2004 01-63
  • 64. Introduction to UNIX Access Permission Code • The protection on a file is referred to as its file modes • File modes are set with the “chmod” command • UNIX supports three types of access permissions: r read w write x execute - permission denied 127 Directory and File Permissions 128Copyright Department of Computer Science,Northern Illinois University, 2004 01-64
  • 65. Introduction to UNIX Access Types Access Type Meaning on File Meaning on Dir. r (read) View file contents List directory contents (open, read) w (write) Change file contents - Change directory contents - Be careful !!! x (execute) Run executable file - Make it your cwd - Access files (by name) in it - Permission denied Permission denied 129 Checking Permissions • To check the permissions of an existing file or an existing directory, use the command: ls –l • Example: ux% ls –l unix total 387 drwxr--r-- 1 z036473 student 862 Feb 7 19:22 unixgrades -rw-r--r-- 1 z036473 student 0 Jun 24 2003 uv.nawk -rw-r--r-- 1 z036473 student 0 Jun 24 2003 wx.nawk -rw-r--r-- 1 z036473 student 0 Jun 24 2003 yz.nawk 130Copyright Department of Computer Science,Northern Illinois University, 2004 01-65
  • 66. Introduction to UNIX Changing Permissions 131 The chmod Command 132Copyright Department of Computer Science,Northern Illinois University, 2004 01-66
  • 67. Introduction to UNIX Changing Permissions: Symbolic Mode 133 Changing Permissions: Symbolic Mode mp% chmod who operation permissions filename u for user + for add r for read g for group - for remove w for write o for others = for assign x for execute a for all 134Copyright Department of Computer Science,Northern Illinois University, 2004 01-67
  • 68. Introduction to UNIX Changing Permissions: Symbolic Mode ux% ls -li sort.c 118283 -rw-r--r-- 1 krush csci 80 Feb 27 12:23 sort.c Example 1: To change the permissions on the file “sort.c” using Symbolic mode, so that: a) Everyone may read and execute it b) Only the owner and group may write to it. We want this using symbolic mode: rwx|rwx|r-x Answer: chmod ug=rwx,o=rx sort.c chmod ugo+rx,go+w sort.c 135 The chmod Command: Octal Mode 136Copyright Department of Computer Science,Northern Illinois University, 2004 01-68
  • 69. Introduction to UNIX Changing Permissions: Octal Mode ux% ls -li sort.c 118283 -rw-r--r-- 1 krush csci 80 Feb 27 12:23 sort.c Example 2: Ignoring the original permission settings, change the permissions on the file “sort.c” using octal mode, so that: a) Everyone may read and execute it b) Only the owner and group may write to it. We want this using octal mode: rwx|rwx|r-x Answer: chmod 775 sort.c 137 Changing Permissions: Octal Mode ux% ls -li sort.c 118283 -rw-r--r-- 1 krush csci 80 Feb 27 12:23 sort.c Step Perform… Settings 1 List the desired setting rwx|rwx|r-x 2 Assign binary: 1 for access; 0 for no access 111|111|101 3 List octal values for the corresponding binary 1’s 421|421|401 4 Convert the octal values to a 3- digit number 775 5 Write the command chmod 775 sort.c 138Copyright Department of Computer Science,Northern Illinois University, 2004 01-69
  • 70. Introduction to UNIX Changing Permissions: exercise Ignoring the original permission settings on a file called “myfile”, assign: a) Read, write, and execute permissions to owner b) Read and execute permissions to group c) Execute permission to others We want: rwx|r-x|--x 1) Using Symbolic Mode: chmod u=rwx,g=rx,o=x myfile __________________________________ 2) Using Octal Mode: chmod 751 myfile __________________________________ 139 User Masks (default permissions) • The default permissions are initially set for a file or directory using a three-digit octal system variable called user mask (mask). • This user mask was defined initially by the system administrator when your account is created. • Initially set by system administrator in a start-up file: $HOME/.cshrc • The user mask contains the octal settings for the permissions to be removed from the default when a directory or file is created. • The default permissions are: – 777 for a directory – 666 for a file 140Copyright Department of Computer Science,Northern Illinois University, 2004 01-70
  • 71. Introduction to UNIX The umask Command 141 User Mask User mask Directory File (666) Value Default: 777 Default: 666 000 777 (rwx rwx rwx) 666 (rw- rw- rw-) 111 666 (rw- rw- rw-) 666 (rw- rw- rw-) 222 555 (r-x r-x r-x) 444 (r- - r- - r- -) 333 444 (r- - r- - r- -) 444 (r- - r- - r- -) 444 333 (-wx –wx –rx) 222 (-w- -w- -w-) 555 222 (-w- -w- -w-) 222 (-w- -w- -w-) 666 111 (- -x - -x - -x) 000 (--- --- --- ) 777 000 (--- --- --- ) 000 (--- --- --- ) 142Copyright Department of Computer Science,Northern Illinois University, 2004 01-71
  • 72. Introduction to UNIX The command: umask % umask 000 For regular files: rw-rw-rw- 666 For directories: rwxrwxrwx 777 In $HOME/.cshrc file: umask 022 For regular files: rw-r- - r- - 644 For directories: rwxr-xr-x 755 143 Default Access Permissions • To prevent cheating, do not grant read and write access permissions to group and others! • Set default permissions for all files and directories created in the future, so that only you/owner can rw to it (rw- --- ---) • In your $HOME/.cshrc file: umask 077 144Copyright Department of Computer Science,Northern Illinois University, 2004 01-72
  • 73. Introduction to UNIX Default Access Permissions: exercise Assume your $HOME/.cshrc file contains the command: umask 002. a) What permissions would be set for newly created text files? b) What permissions would be set for newly created directories? 145 Special Permissions • The regular file permissions (rwx) are used to assign security to files and directories. • Three additional special permissions can be optionally used on files and directories. – Set User Id (SUID) – Set Group ID (SGID) – Sticky bit 146Copyright Department of Computer Science,Northern Illinois University, 2004 01-73
  • 74. Introduction to UNIX Special Permissions: SUID • SUID enables regular users to execute a program file and become the owner of the file for the duration of execution. • Example: The “passwd” command is used to change a user’s password. The file “/usr/bin/passwd” (owned by “root”) has this special permission set: r-sr-sr-x • In theory, only the root user has the ability to change passwords. Since “/usr/bin/passwd” has the SUID permission on, when a regular user executes the “passwd” command, the user temporarily becomes the “root” user while the “passwd” command is executing. 147 Special Permissions: GUID • Like SUID, SGID enables regular users to execute a program file and become a member of the group that is attached to the file. • If a file is owned by the system group and also has the SGID permission, then any user who executes that file will be a member of the system group during the execution. 148Copyright Department of Computer Science,Northern Illinois University, 2004 01-74
  • 75. Introduction to UNIX Special Permissions: Sticky Bit • Sticky bit performs a useful function on directories • Recall: Write permission applied to a directory enables you to add and remove any files to or from that directory. • If you had “write” permission to a certain directory but no permissions for files within it, you could delete all of those files. 149 Special Permissions: Sticky Bit Scenario: A company provides a common directory that gives all employees the ability to add and remove files in it. This directory must give user “write” permissions. Unfortunately, the “write” permissions also gives all employees the ability to delete all files and subdirectories within it, including the ones that others have added to the directory. 150Copyright Department of Computer Science,Northern Illinois University, 2004 01-75
  • 76. Introduction to UNIX Special Permissions: Sticky Bit • If the “sticky” bit were applied to this common directory, in addition to “write” permissions, then employees may add files to the directory. However, they may delete only files and directories that they have added/created. • Example: ls –ld /tmp drwxrwxrwt 3 root sys 1077 Jan 25 13:30 /tmp Sticky Bit 151 Special Permissions • The access permission status that is displayed using the “ls –l” command does not have a section for special permissions • However, since special permissions required “execute”, they mask the execute permission when displayed using the “ls –l” command. rwxrwxrwx rwsrwsrwt SUID SGID STICKY BIT 152Copyright Department of Computer Science,Northern Illinois University, 2004 01-76
  • 77. Introduction to UNIX Special Permissions • All special permissions also required the “execute” permission to work properly. • Usually, the SUID and SGID apply to executable files. • The sticky bit applies to directories, which must have execute permission for access. • If special permissions are set on a file or a directory without “execute” permission, the special permissions are shown in capital letters. rw-rw-rw- rwSrwSrwT SUID SGID STICKY 153 BIT Setting Special Permissions suid sgid stb r w x r w x r w x 4 2 1 4 2 1 4 2 1 4 2 1 7 7 7 7 Special user group others Use the “chmod” command with octal mode: chmod 7777 filename 154Copyright Department of Computer Science,Northern Illinois University, 2004 01-77
  • 78. Introduction to UNIX Special Permissions: Exercise • Give the full UNIX command to enable all users including others to add files to directory named “dir1”, but only to delete files that they own in that directory. • Answer: chmod 1777 dir1 ____________________________________ 155 Regular Expressions 156Copyright Department of Computer Science,Northern Illinois University, 2004 01-78
  • 79. Introduction to UNIX Regular Expression Metacharacters • A pattern of special characters used to match strings in a search • Can be controlled by special characters called metacharacters • Metacharacters are used by UNIX pattern matching editors and utilities: – Editors: ed, ex, vi – Utilities: grep, egrep, sed, and awk 157 RE Metacharacters: use with grep RE Metacharacter Matches… ^ A character at beginning of line $ A character at end of line . Any one character, except new line * Zero or more of preceding character char Escape the meaning of char following it [a-z] Any one of the enclosed characters (e.g. a-z) [^] One character not in the set < Beginning of word anchor > End of word anchor ( ) Tags matched characters to be used later (max = 9) x{m} Repetition of character x, m times (x,m = integer) x{m,} Repetition of character x, at least m times x{m,n} Repetition of character x between m and m times 158Copyright Department of Computer Science,Northern Illinois University, 2004 01-79
  • 80. Introduction to UNIX RE Metacharacters: use with egrep RE Metacharacter Matches… ^ A character at beginning of line $ A character at end of line . Any one character, except new line * Zero or more of preceding character char Escape the meaning of char following it [a-z] Any one of the enclosed characters (e.g. a-z) [^] One character not in the set + One or more of the preceding characters ? Zero or one of the preceding characters a|b Either a or b () Groups characters 159 The grep Family 160Copyright Department of Computer Science,Northern Illinois University, 2004 01-80
  • 81. Introduction to UNIX The Command: grep Syntax: grep [options] RE-pattern file(s) • Searches the entire specified input file(s) for a pattern of characters. • If pattern contains white space, it must be in quotes. • All other words following RE-pattern are treated as files. • Sends its output to stdout (default), a file, or a command pipe. • “grep” does not change the original input file(s). • “grep” can receive input from stdin (keyboard), a pipe, or file(s) 161 Commonly Used Grep Options Commonly used “grep” options: -c Print only a count of matched lines. -i Ignore uppercase and lowercase distinctions. -l List all files that contain the specified pattern. -n Print matched lines and line numbers. -s Work silently; display nothing except error messages. Useful for checking the exit status. -v Print lines that do not match the pattern. -w Search for the expression as a word as if surrounded by < > 162Copyright Department of Computer Science,Northern Illinois University, 2004 01-81
  • 82. Introduction to UNIX Regular Expression An atom specifies what text is to be matched and where it is to be found. An operator combines regular expression atoms. 163 Atoms An atom specifies what text is to be matched and where it is to be found. 164Copyright Department of Computer Science,Northern Illinois University, 2004 01-82
  • 83. Introduction to UNIX Single-Character Atom A single character matches itself. 165 Dot Atom A dot matches any single character except for a new line character (n) 166Copyright Department of Computer Science,Northern Illinois University, 2004 01-83
  • 84. Introduction to UNIX Class Atom A class matches only single character that can be any of the characters defined in a set, e.g. [A-C] matches either A, B, or C. Notes: 1) A range of characters is indicated by a dash, e.g. [A-C] 2) Can specify characters to be excluded from the set, e.g. [^0-9] matches any character other than a number. 167 Example: Classes 168Copyright Department of Computer Science,Northern Illinois University, 2004 01-84
  • 85. Introduction to UNIX Anchors Anchors tell where the next character in the pattern must be located in the text data. 169 Back References: n • Used to retrieve saved text in one of nine buffers • Can refer to the text in a saved buffer by using a back reference; e.g. 1 2 3 ...9. • More on back references later in this section. 170Copyright Department of Computer Science,Northern Illinois University, 2004 01-85
  • 86. Introduction to UNIX Operators no oper. 171 Sequence Operator In a sequence operator, if a series of atoms are shown in a regular expression, there is no operator between them. 172Copyright Department of Computer Science,Northern Illinois University, 2004 01-86
  • 87. Introduction to UNIX Alternation Operator: | The alternation operator ( | ) is used to defined one or more alternatives, e.g. A | B matches A or B. 173 Repetition Operator: {…} The repetition operator specifies that the atom or expression immediately before the repetition may be repeated. 174Copyright Department of Computer Science,Northern Illinois University, 2004 01-87
  • 88. Introduction to UNIX Basic Repetition Forms 175 Short Form Repetition Operators: * + ? 176Copyright Department of Computer Science,Northern Illinois University, 2004 01-88
  • 89. Introduction to UNIX Group Operator In the group operator, when a group of characters is enclosed in parentheses, the next operator applies to the whole group, not only the previous characters. 177 Save Operator: (..) • Copies a matched text string to one of nine buffers for later reference. • Back references can then be used to retrieve saved text in one of nine buffers. • Can refer to the text in a saved buffer by using a back reference; e.g. 1 2 3 ...9. • Example: To reverse first name and last name in the following data set; also. John DeVries Bob Flowers s/^([A-Z].*) ([A-Z].*)$/2 1/ DeVries John Flowers Bob 178Copyright Department of Computer Science,Northern Illinois University, 2004 01-89
  • 90. Introduction to UNIX More Examples • The following slides contain examples of using regular expressions with grep, egrep and fgrep. • Review them on your own time. • Ask your instructor or T.A. if you have any questions. 179 Example: Grep with RE: ^ % cat grep-datafile northwest NW Charles Main 300000.00 western WE Sharon Gray 53000.89 southwest SW Lewis Dalsass 290000.73 southern SO Suan Chin 54500.10 southeast SE Patricia Hemenway 400000.00 eastern EA TB Savage 440500.45 northeast NE AM Main Jr. 57800.10 north NO Ann Stephens 455000.50 central CT KRush 575500.70 Print all lines beginning Extra [A-Z]****[0-9]..$5.00 with the letter n. % grep ^n grep-datafile northwest NW Charles Main 300000.00 northeast NE AM Main Jr. 57800.10 north NO Ann Stephens 455000.50 180Copyright Department of Computer Science,Northern Illinois University, 2004 01-90
  • 91. Introduction to UNIX Example: Grep with RE: $ % cat grep-datafile northwest NW Charles Main 300000.00 western WE Sharon Gray 53000.89 southwest SW Lewis Dalsass 290000.73 southern SO Suan Chin 54500.10 southeast SE Patricia Hemenway 400000.00 eastern EA TB Savage 440500.45 northeast NE AM Main Jr. 57800.10 north NO Ann Stephens 455000.50 Print all lines ending with a central CT KRush 575500.70 period and exactly two Extra [A-Z]****[0-9]..$5.00 zero numbers. % grep .00$ grep-datafile northwest NW Charles Main 300000.00 southeast SE Patricia Hemenway 400000.00 Extra [A-Z]****[0-9]..$5.00 181 Example: Grep with RE: char % cat grep-datafile northwest NW Charles Main 300000.00 western WE Sharon Gray 53000.89 southwest SW Lewis Dalsass 290000.73 Print all lines containing the southern SO Suan Chin 54500.10 number 5, followed by a southeast SE Patricia Hemenway 400000.00 literal period and any single eastern EA TB Savage 440500.45 character. The dot northeast NE AM Main Jr. 57800.10 metacharacter represents a north NO Ann Stephens 455000.50 single character, unless it is central CT KRush 575500.70 escaped with a backslash. Extra [A-Z]****[0-9]..$5.00 When escaped, the period is no longer a special character, but represents itself, a literal period. % grep 5.. grep-datafile Extra [A-Z]****[0-9]..$5.00 182Copyright Department of Computer Science,Northern Illinois University, 2004 01-91
  • 92. Introduction to UNIX Example: Grep with RE: [ ] % cat grep-datafile northwest NW Charles Main 300000.00 western WE Sharon Gray 53000.89 southwest SW Lewis Dalsass 290000.73 southern SO Suan Chin 54500.10 southeast SE Patricia Hemenway 400000.00 eastern EA TB Savage 440500.45 northeast NE AM Main Jr. 57800.10 north NO Ann Stephens 455000.50 central CT KRush 575500.70 Extra [A-Z]****[0-9]..$5.00 Print all lines beginning with either % grep ^[we] grep-datafile a “w” or an “e”. western WE Sharon Gray 53000.89 eastern EA TB Savage 440500.45 183 Example: Grep with RE: [^] % cat grep-datafile northwest NW Charles Main 300000.00 western WE Sharon Gray 53000.89 southwest SW Lewis Dalsass 290000.73 southern SO Suan Chin 54500.10 southeast SE Patricia Hemenway 400000.00 eastern EA TB Savage 440500.45 northeast NE AM Main Jr. 57800.10 north NO Ann Stephens 455000.50 Print all lines ending central CT KRush 575500.70 with a period and Extra [A-Z]****[0-9]..$5.00 exactly two non-zero numbers. % grep .[^0][^0]$ grep-datafile western WE Sharon Gray 53000.89 southwest SW Lewis Dalsass 290000.73 eastern EA TB Savage 440500.45 184Copyright Department of Computer Science,Northern Illinois University, 2004 01-92
  • 93. Introduction to UNIX Example: Grep with RE: x{m} % cat grep-datafile northwest NW Charles Main 300000.00 western WE Sharon Gray 53000.89 southwest SW Lewis Dalsass 290000.73 southern SO Suan Chin 54500.10 southeast SE Patricia Hemenway 400000.00 eastern EA TB Savage 440500.45 northeast NE AM Main Jr. 57800.10 north NO Ann Stephens 455000.50 central CT KRush 575500.70 Extra [A-Z]****[0-9]..$5.00 Print all lines where there are at least six consecutive % grep [0-9]{6}. grep-datafile numbers followed by a northwest NW Charles Main 300000.00 period. southwest SW Lewis Dalsass 290000.73 southeast SE Patricia Hemenway 400000.00 eastern EA TB Savage 440500.45 north NO Ann Stephens 455000.50 central CT KRush 575500.70 185 Example: Grep with RE: < % cat grep-datafile northwest NW Charles Main 300000.00 western WE Sharon Gray 53000.89 southwest SW Lewis Dalsass 290000.73 southern SO Suan Chin 54500.10 southeast SE Patricia Hemenway 400000.00 eastern EA TB Savage 440500.45 northeast NE AM Main Jr. 57800.10 north NO Ann Stephens 455000.50 central CT KRush 575500.70 Extra [A-Z]****[0-9]..$5.00 Print all lines containing a word % grep <north grep-datafile starting with “north”. northwest NW Charles Main 300000.00 northeast NE AM Main Jr. 57800.10 north NO Ann Stephens 455000.50 186Copyright Department of Computer Science,Northern Illinois University, 2004 01-93
  • 94. Introduction to UNIX Example: Grep with RE: < > % cat grep-datafile northwest NW Charles Main 300000.00 western WE Sharon Gray 53000.89 southwest SW Lewis Dalsass 290000.73 southern SO Suan Chin 54500.10 southeast SE Patricia Hemenway 400000.00 eastern EA TB Savage 440500.45 northeast NE AM Main Jr. 57800.10 north NO Ann Stephens 455000.50 central CT KRush 575500.70 Extra [A-Z]****[0-9]..$5.00 Print the line if it contains the word % grep <north> grep-datafile “north”. north NO Ann Stephens 455000.50 187 Example: grep with pipe % ls -l | grep ^d Pipe the output of the drwxr-xr-x 2 krush csci 512 Feb 8 22:12 assignments “ls –l” command to grep and list/select drwxr-xr-x 2 krush csci 512 Feb 5 07:43 feb3 only directory entries. drwxr-xr-x 2 krush csci 512 Feb 5 14:48 feb5 drwxr-xr-x 2 krush csci 512 Dec 18 14:29 grades drwxr-xr-x 2 krush csci 512 Jan 18 13:41 jan13 drwxr-xr-x 2 krush csci 512 Jan 18 13:17 jan15 drwxr-xr-x 2 krush csci 512 Jan 18 13:43 jan20 drwxr-xr-x 2 krush csci 512 Jan 24 19:37 jan22 drwxr-xr-x 4 krush csci 512 Jan 30 17:00 jan27 drwxr-xr-x 2 krush csci 512 Jan 29 15:03 jan29 Display the number of % ls -l | grep -c ^d lines where the pattern 12 was found. This does not mean the number of occurrences of the pattern. 188Copyright Department of Computer Science,Northern Illinois University, 2004 01-94
  • 95. Introduction to UNIX Example: egrep with RE: a|b % cat grep-datafile northwest NW Charles Main 300000.00 western WE Sharon Gray 53000.89 southwest SW Lewis Dalsass 290000.73 southern SO Suan Chin 54500.10 southeast SE Patricia Hemenway 400000.00 eastern EA TB Savage 440500.45 northeast NE AM Main Jr. 57800.10 north NO Ann Stephens 455000.50 central CT KRush 575500.70 Extra [A-Z]****[0-9]..$5.00 Prints the line if it contains either the % egrep NW|EA grep-datafile expression “NW” or the northwest NW Charles Main 300000.00 expression “EA”. eastern EA TB Savage 440500.45 189 Example: egrep with RE: + % cat grep-datafile northwest NW Charles Main 300000.00 western WE Sharon Gray 53000.89 southwest SW Lewis Dalsass 290000.73 southern SO Suan Chin 54500.10 southeast SE Patricia Hemenway 400000.00 eastern EA TB Savage 440500.45 northeast NE AM Main Jr. 57800.10 north NO Ann Stephens 455000.50 central CT KRush 575500.70 Extra [A-Z]****[0-9]..$5.00 Print all lines containing one or more 3s. % egrep 3+ grep-datafile northwest NW Charles Main 300000.00 western WE Sharon Gray 53000.89 southwest SW Lewis Dalsass 290000.73 190Copyright Department of Computer Science,Northern Illinois University, 2004 01-95
  • 96. Introduction to UNIX Example: egrep with RE: ? % cat grep-datafile northwest NW Charles Main 300000.00 western WE Sharon Gray 53000.89 southwest SW Lewis Dalsass 290000.73 southern SO Suan Chin 54500.10 southeast SE Patricia Hemenway 400000.00 eastern EA TB Savage 440500.45 northeast NE AM Main Jr. 57800.10 north NO Ann Stephens 455000.50 central CT KRush 575500.70 Print all lines containing Extra [A-Z]****[0-9]..$5.00 a 2, followed by zero or one period, followed by a % egrep 2.?[0-9] grep-datafile number. southwest SW Lewis Dalsass 290000.73 191 Example: egrep with RE: ( ) % cat grep-datafile northwest NW Charles Main 300000.00 western WE Sharon Gray 53000.89 southwest SW Lewis Dalsass 290000.73 southern SO Suan Chin 54500.10 southeast SE Patricia Hemenway 400000.00 eastern EA TB Savage 440500.45 northeast NE AM Main Jr. 57800.10 north NO Ann Stephens 455000.50 central CT KRush 575500.70 Print all lines containing Extra [A-Z]****[0-9]..$5.00 one or more consecutive occurrences of the % egrep (no)+ grep-datafile pattern “no”. northwest NW Charles Main 300000.00 northeast NE AM Main Jr. 57800.10 north NO Ann Stephens 455000.50 192Copyright Department of Computer Science,Northern Illinois University, 2004 01-96
  • 97. Introduction to UNIX Example: egrep with RE: (a|b) % cat grep-datafile northwest NW Charles Main 300000.00 western WE Sharon Gray 53000.89 southwest SW Lewis Dalsass 290000.73 southern SO Suan Chin 54500.10 southeast SE Patricia Hemenway 400000.00 eastern EA TB Savage 440500.45 northeast NE AM Main Jr. 57800.10 north NO Ann Stephens 455000.50 central CT KRush 575500.70 Print all lines containing Extra [A-Z]****[0-9]..$5.00 the uppercase letter “S”, followed by either “h” or % egrep S(h|u) grep-datafile “u”. western WE Sharon Gray 53000.89 southern SO Suan Chin 54500.10 193 Example: fgrep % cat grep-datafile northwest NW Charles Main 300000.00 western WE Sharon Gray 53000.89 southwest SW Lewis Dalsass 290000.73 southern SO Suan Chin 54500.10 southeast SE Patricia Hemenway 400000.00 eastern EA TB Savage 440500.45 northeast NE AM Main Jr. 57800.10 north NO Ann Stephens 455000.50 central CT KRush 575500.70 Find all lines in the file Extra [A-Z]****[0-9]..$5.00 containing the literal string “[A-Z]****[0-9]..$5.00”. % fgrep [A-Z]****[0-9]..$5.00 grep-datafile All characters are treated as Extra [A-Z]****[0-9]..$5.00 themselves. There are no special characters. 194Copyright Department of Computer Science,Northern Illinois University, 2004 01-97
  • 98. Introduction to UNIX The C Shell 195 Standard UNIX Shell Overview 196Copyright Department of Computer Science,Northern Illinois University, 2004 01-98
  • 99. Introduction to UNIX Objectives • C Shell Environment – Startup and logout files – Shell variables – History – Command substitution – Command sequences – Aliases – Directory Stacks – Redirections and pipe • C Shell Programming 197 Password File Login shell ux% nismatch z036473 passwd.org_dir z036473:e6qhWRw.YeK0E:1432:202:rush kaisone:/home/mp/z036473:/bin/csh:12068:: gid Home directory User login id Encrypted User uid name password Number of days between 1/1/1970 and the last day that the user changed their password. 198Copyright Department of Computer Science,Northern Illinois University, 2004 01-99
  • 100. Introduction to UNIX C Shell Startup init Login: getty Login: z036473 Passwd: login Login: z036473 Passwd: csh % 199 NIS Password File • Network Information Service password file named “passwd.org_dir” contains user account information. • The command named “niscat” displays the entire contents of the “passwd.org_dir” file. • The command named “nismatch” displays account information for a specified user. Examples: Display contents of the password file. ux% niscat passwd.org_dir 200Copyright Department of Computer Science,Northern Illinois University, 2004 01-100
  • 101. Introduction to UNIX Login At NIU Default shell: /bin/csh Can be customized via startup scripts 1) /etc/.login (system shell) 2) $HOME/.cshrc (if C shell is the login shell) 3) $HOME/.login 201 Startup File: $HOME/.cshrc • Executed once: – at login and – each time a C shell script is executed (unless #!/bin/csh –f was used) • Located in your $HOME directory • Contains settings such as: – Command search path – Initial permission setting (umask) – Terminal setting – Mail box, etc. 202Copyright Department of Computer Science,Northern Illinois University, 2004 01-101
  • 102. Introduction to UNIX Useful .cshrc content • Define command aliases alias ll “ls –al” alias ap “man –k” • customize common behavior: – set filec when typing a file/directory name, if you press the esc key, it will complete the rest of the name for you. – set ignoreeof prevents accidental logout if you type ctrl-d at the command line. – set noclobber prevents overwritten of files by output direction. 203 Startup File: $HOME/.login • Executed once only during login process • Located in your $HOME directory • Contains settings such as: – Terminal setting: special keyboard character setting stty erase ‘^h’ • Mail directory 204Copyright Department of Computer Science,Northern Illinois University, 2004 01-102
  • 103. Introduction to UNIX Logout File: $HOME/.logout • Executed once when you log out • Located in your $HOME directory • useful content: ps –f –u $user displays the user’s active processes 205 C Shell Prompts • C shell has two prompts: – Primary prompt: % – Secondary prompt: ? • Primary prompt (%) – Can be reset – Use the command named “set prompt” to customize your command prompt. Example: ux% set prompt = “$USER > “ z036473 > New prompt 206Copyright Department of Computer Science,Northern Illinois University, 2004 01-103
  • 104. Introduction to UNIX C Shell Prompts • Secondary prompt – Appears when writing on-line script at the command prompt % foreach student (z036473 z036474) ? mail $student < memo ? ^d (type ctrl-d or type end) % – Once the command is entered, and [Return] is pressed, you cannot go back to previous lines. – C shell history does not save commands typed at the secondary prompt. 207 Variables A shell variable is a location in memory where values can be stored Variables Pre-defined User-defined Special Shell Environment String Arithmetic Built-in Pathname 208Copyright Department of Computer Science,Northern Illinois University, 2004 01-104
  • 105. Introduction to UNIX Pre-defined Variables • Two types: – Pre-defined shell variables – Pre-defined environment variables • Pre-defined shell variables are used to configure the shell. – Example: history – which contains the size of the history list. • Pre-defined environmental variables are used to configure the shell environment. – Example: HOME – which contains the full pathname of your home directory. 209 Some Predefined Shell Variables Name Contents cwd The current working directory history The size of the history list ignoreeof Prevents the shell from terminating when pressing Control-D. Use the “logout” or “exit” command. noclobber Prevents existing files from being overridden by output redirection (>), and non-existent files from being appended by append (>>) prompt The shell primary prompt savehist The number of commands to save in the history file “$HOME/.history” status The exit code of the last command 210Copyright Department of Computer Science,Northern Illinois University, 2004 01-105
  • 106. Introduction to UNIX Predefined Environment Variables Name Meaning HOME The full pathname of your home directory PATH A List of directories to search for commands MAIL The full pathname of your mailbox USER Your user id SHELL The full pathname of your login shell TERM The type of your terminal 211 Command Search Path setenv PATH /usr/local/bin:/usr/bin:/usr/ucb:$home/bin: dev etc home usr tty null skel mp ux bin local ucb z036473 bin bin csci330 .cshrc .login .logout 212Copyright Department of Computer Science,Northern Illinois University, 2004 01-106
  • 107. Introduction to UNIX Creating Environment Variables Syntax: setenv varname value • Create C shell environment variable and initialize it to a value • If varname does not exist, it is created; otherwise, it is overwritten • An Environment variable always holds exactly one value • Example: % setenv TERM vt100 % echo $TERM vt100 % setenv EDITOR pico 213 User-Defined Variables • Created by the user • Syntax: set varname=string • Create a shell variable and initialize it to a string of characters • varname: – 1-20 characters long – Letters, digits, and underscore – First character cannot be a digit – Should not be the same as one of the pre-defined variables 214Copyright Department of Computer Science,Northern Illinois University, 2004 01-107
  • 108. Introduction to UNIX Storing Values in Variables Command Result Comment set x = 123 x contains “123” x contains a number, but it is stored as string of digits set x = Hello x contains “Hello” Storing a character string set name = Jane Black x contains “Jane” Only “Jane” is stored set name = “Jane Black” x contains When a string contains spaces “Jane Black” it must be quoted set x = “Go Dons!” x contains When a string contains spaces “Go Dons!” and special characters, it must be quoted 215 Accessing the Values of a Variable • The name of the variable must be preceded by a dollar sign • Variable values can be used anywhere in a string • Examples: % set count = 7 % echo $count is greater than 6 and less than 8 7 is greater than 6 and less than 8 % echo The value of count is $count as expected The value of count is 7 as expected % echo My lucky number is $count My lucky number is 7 216Copyright Department of Computer Science,Northern Illinois University, 2004 01-108
  • 109. Introduction to UNIX Accessing the Values of a Variable • Variable values can also be stored in another variable • Example: % set count = 5 % set number = $count % echo count contains: $count count contains: 5 % echo number contains: $number number contains: 5 217 Unsetting a Variables • We can clear a variable by assigning a null value to it - examples: set z = “ “; set y = • Another method is to use the “unset” command • Example: % set x = 1 % echo "(x contains:" $x")" (x contains: 1) % unset x % echo "(x contains:" $x")" x: Undefined variable 218Copyright Department of Computer Science,Northern Illinois University, 2004 01-109
  • 110. Introduction to UNIX Command Substitution • A command surrounded by backticks (` `) is replaced by its standard output • Any newline in the output are replaced by spaces Demo 219 Command Sequence (;) • Allows you to enter a series of commands all at once • Commands are separated by a semicolon (;) • Useful for typing an entire sequence of commands at once Example: To execute these commands in sequence: date, pwd, and ls % date;pwd;ls Mon Jul 2 11:07:10 CDT 2003 /home/ux/krush/330/Summer-03/Quizzes message.rickert quiz1-sum03 quiz2.sum03 220Copyright Department of Computer Science,Northern Illinois University, 2004 01-110
  • 111. Introduction to UNIX C Shell History • C shell keeps a record of previously entered cmds so that they can later be: – Re-executed – Edited • Entered cmds are stored in the current C shell’s history buffer ($HOME/.history) • Commands are saved – Per session – Per user 221 C Shell History • Each previously run command gets a sequential event number • To view the history buffer: – Syntax: history [-rh] [count] • If no options are supplied, list all – Useful options: -r displays history list in reverse order -h inhibits the display of event numbers 222Copyright Department of Computer Science,Northern Illinois University, 2004 01-111
  • 112. Introduction to UNIX C Shell History • You can re-execute history events: 1) By the event number % !5 1) By the number relative to current event % !-3 1) By the text it contains % !ls 223 C Shell History How big a list ? 1) On the command line, type: % set history = 20 (per session only) 2) Include in your $HOME/.logout file: set savehist = 30 At logout, C shell saves the last 30 cmds you executed in the $HOME/.history file 224Copyright Department of Computer Science,Northern Illinois University, 2004 01-112
  • 113. Introduction to UNIX C Shell History Two ways to display event number as part of your cmd prompt: 1) On the command line, type: % set prompt = “! %” (per session only) 1 % echo csci 330 csci 330 2%_ 2) Include in your $HOME/.cshrc file: set prompt = “! %” (per each login) 225 C Shell History Command Action !! Re-execute the last command !N Invoke event N (integer) !str Invoke most recent event beginning with str (string) !?str? Invoke most recent event containing str ^str1^str2 Substitute str2 for str1 in the previous cmd For more listing of history commands, see Course Notes pp. 13-7 226Copyright Department of Computer Science,Northern Illinois University, 2004 01-113
  • 114. Introduction to UNIX C Shell History: Words in Event • Can use any word from any event on the history list • C shell numbers the words in each command starting at 0 • Example: grep ‘^[0-9]{5}’ | sort –t: | cut –d: -f1 Word: 0 1 2 3 4 5 6 7 8 • Can easily identify “words” since space separate them 227 C Shell History: Words in Event • Can refer to individual words with :n ( n is the word number) • Example: 100 % ls –l assign1 assign2 assign3 101 % chmod a-w !ls:2-$ $ refers to the last chmod a-w assign1 assign2 assign3 word 102 % ls –l !ls:$ ls -l assign3 -r-------- 1 krush csci 0 Feb 14 12:35 assign3 • The modifier * refers to words 1 through last (I.e. everything except the command name) 103 % rm !100:2* rm assign1 assign2 assign3 228Copyright Department of Computer Science,Northern Illinois University, 2004 01-114
  • 115. Introduction to UNIX C Shell History Demo 229 Command Substitution • A command surrounded by backticks (` `) is replaced by its standard output • Any newline in the output are replaced by spaces Demo 230Copyright Department of Computer Science,Northern Illinois University, 2004 01-115
  • 116. Introduction to UNIX C Shell Aliases • Important feature of C shell • Allows you to assign a name, or alias to command(s) • Can use alias like any other command • Can rename existing commands • Can reference shell variables • Aliases defined at the parent shell’s command line are not inherited by subshells 231 C Shell Aliases • C shell user-defined abbreviation for a command • Useful if a command syntax is difficult to remember • Define aliases at the command line or in “$HOME/.cshrc” file • Alternative: create a file name “.myalias” in your $HOME and include all alias definitions. – Include this line in your “$HOME/.cshrc” file: source $HOME/.myalias 232Copyright Department of Computer Science,Northern Illinois University, 2004 01-116
  • 117. Introduction to UNIX Creating Aliases Use the command: alias Syntax: alias name ‘command-list’ User-defined •The command(s) that will be nickname for the executed when the alias is invoked command •Multiple commands are separated by a semicolon •Metacharacters are surrounded by single quotes 233 Passing Arguments into Aliases • C shell assumes all arguments come at the end of the alias definition, unless you specified otherwise • Two common argument destinations: !* words 1 through the last word of the current command/event !^ word 1 of the current command/event • Recall: grep ‘^[0-9]{5}’ demand | sort –t: | cut –d: -f1 Word: 0 1 2 3 4 5 6 7 8 9 234Copyright Department of Computer Science,Northern Illinois University, 2004 01-117
  • 118. Introduction to UNIX Example: Creating an Alias % alias cd ‘cd !*; set prompt = “$cwd >”’ Note: • prevents the history mechanism from evaluating the !* • !* represents the argument from the most recent current command in the history list (i.e. words/fields 1 through the last word of the recent command) • !* allows you to pass any number of arguments to the alias 235 Issue a Command to the C Shell • What happens when you issue a command to the C shell? • After history substitution, examine each word that could be a command in the following sequential order: 1. Alias Substitution 2. Built-in command (commands inside the C shell; e.g. echo) 3. Commands in your search path ($PATH) 236Copyright Department of Computer Science,Northern Illinois University, 2004 01-118
  • 119. Introduction to UNIX In-Class Exercise: aliases 1. Write an alias named “h” to list the history of previously executed commands. alias h history 2. Write an alias named “c” to clear the contents of the screen. alias c clear 3. Write an alias named “rm” to prompt the user for a delete confirmation message before removing file(s). alias rm ‘rm –i !*’ 237 In-Class Exercise: aliases 4. Write an alias named “cx” to give the owner the execute permission on more or more file/directory. alias cx ‘chmod u+x !*’ 5. Write an alias named “loc” to locate/find any file in your directory hierarchy. alias loc ‘find $home –name “!:1” –print’ 6. Write an alias named “ldir” to display long listing of only directory names (i.e. not ordinary files). alias ldir ‘ls –l !* | grep “^d”’ alias fdir ‘find !:1 –type d –print’ 238Copyright Department of Computer Science,Northern Illinois University, 2004 01-119
  • 120. Introduction to UNIX Output • The output statement of the C shell is the “echo” command • Syntax: echo [option] arg1 arg2 …argN • Its arguments can be strings or variables • Example: % set time = “2:00pm.” % echo "It is now $time." It is now 2:00pm. % echo It is now $time. It is now 2:00pm. The variable cannot be % echo It is now $time. embedded in single It is now $time. quotes. 239 Output Redirection (>) • Syntax: command > file • Sends output of command to file, instead of to terminal Calls the disk usage command for the • Examples: current directory and redirects the % du > status output to a file called ‘status’ ( ) indicates command groups. Use it % (date; du) > status to combine the output of multiple commands. In this example, we place time and date in front of the disk usage 240Copyright Department of Computer Science,Northern Illinois University, 2004 01-120
  • 121. Introduction to UNIX Examples: Output Redirection (>) % date; du > usage-status Output of ‘date’ goes to? __________________ Output of ‘du’ goes to?___________________ % date > usage-status; du > usage-status The file “usage-status” contains ? ___________________________________ 241 Input Redirection (<) • Syntax: Command < file • Command will read (take input) from file, instead of from terminal • Example: tr “[A-Z]” “[a-z]” < report.input 242Copyright Department of Computer Science,Northern Illinois University, 2004 01-121
  • 122. Introduction to UNIX Examples: Output / Input • Redirecting input and output: % tr “[A-Z]” “[a-z]” < report.input > report.output • Output of one command becomes input of next: % ls > temp.txt; wc < temp.txt • Eliminate the middleman: pipe % ls | wc 243 Appending Output • Syntax: command >> file • Places the stdout of command (or command group) at the end of file • If file does not exist, C shell creates it given that the variable “noclobber” is not set • Examples: % date >> usage-status Build the file ‘usage-status’ from % ls –l >> usage-status the output of the ‘date’, ‘ls’, and % du –s >> usage-status ‘du’ commands % date > usage-status 244Copyright Department of Computer Science,Northern Illinois University, 2004 01-122
  • 123. Introduction to UNIX Using “set noclobber” % who > current-users % date >> usage-status % set noclobber – Does not allow output redirection if the file already exist – Does not allow appending if the file does not already exist % who > current-users current-users: File exists % date >> usage-statis Specified incorrect filename! statis: No such file or directory 245 Override the Extra Protection • Add ! To the redirection • ! Means “do what I mean!” • Example: % set noclobber % who >! current-users Allows redirection even if ‘current-users’ exists % date >>! usage-statis Creates the file named “usage-statis” even if it does not already exist 246Copyright Department of Computer Science,Northern Illinois University, 2004 01-123
  • 124. Introduction to UNIX Handling Standard Errors • Sometimes a command has special output to inform you of problems; for example: % gcc gets.c > compile.out “gets.c”:8: syntax error • Syntax: command >& file – Redirect stdout and stderr to file • Syntax: command >>& file – Append stdout and stderr to file 247 Examples: Redirect/Append Stderr % gcc gets.c >& compile.out % cat compile.out “gets.c”:8: syntax error % (date; gcc gets.c –o gets) >>& compile-log Add to the file ‘compile-log’ a running record with the date and time of the compilation of the file gets.c 248Copyright Department of Computer Science,Northern Illinois University, 2004 01-124
  • 125. Introduction to UNIX Separating Stdout and Stderr % cat calendar logfile >& save – Saves both stdout and stderr in file ‘save’ % cat save July 11 Planning Meeting stdout July 18Budget Meeting Cat: can’t open logfile stderr % (cat calendar logfile > save) >& errfile – Redirects stdout to ‘save’ – Redirects stderr to ‘errfile’ – Order is important, > goes inside ( ), and >& goes outside 249 Sending Stdout and Stderr Through a Pipe • Syntax: command1 |& command2 Sends stdout and stderr of command1 to be stdin to command2 • Example: % gcc gets.c |& more – Sends stdout and stderr to ‘more’ for convenient viewing % gcc gets.c |& lpr –Plpcsl – Produces a hardcopy of the messages 250Copyright Department of Computer Science,Northern Illinois University, 2004 01-125
  • 126. Introduction to UNIX Summary: Redirections and Pipe Command Syntax Meaning command < file Redirect input from file to command command > file Redirect output from command to file command >& file Redirect output and errors to file command >> file Redirect output of command and appends it to file command1 | command2 Take/pipe output of command1 as input to command2 command1 |& command2 Take/pipe output and errors of command1 as input to command2 251Copyright Department of Computer Science,Northern Illinois University, 2004 01-126