Sed – tips and Tricks          1  Logan Palanisamy
Agenda                          2 Basics Bio Break Intermediate Concepts Q&A
What is sed                               3 non-interactive non-screen oriented Line oriented Input file can be any si...
sed syntax                            4 sed [options] cmd in_file(s) sed [options] cmd in_file(s) [> out_file] Standard...
Simple examples                                           5Example                       Explanationsed s/pat1/pat2/g file...
Different sed options/switches                                      6Range                   Remarks-e                    ...
The "-f" option - example                            7 cat mysed.txt   /pat1/ s/this/that/g   /pat2/ s/before/after/ sed...
With and without "-s" option -                           Comparison                                        8Lines   Lines ...
Address Specification                                     9Range               Remarks10                  Just the line 10...
Address Specification with negation                                   10Range               Remarks10!                 All...
Regular Expressions                                    11Meta character Meaning.             Matches any single character ...
Extended Regular Expressions                                     12Meta character Meaning|               alternation. e.g....
Regular Expressions – Examples                                     13Example                       Meaning.{10,}          ...
Substitution – Format and Options                           14 [address[!]]s/pat1/pat2/[options] Options: g – global, w ...
Substitution - Examples                                        15Example                       Explanationsed s/pat1/pat2/...
Substitution – Examples contd                                          16Example                         Explanationsed /p...
Substitution – Examples contd.                                           17Example                         Explanationsed ...
Grouping and Back Referencing                           18 Parts of strings in the Search/Left-hand side can be  grouped ...
Substitution with Grouping and Back               Referencing. Examples                                          19Command...
Substitution with Grouping and Back               Referencing. Examples                                       20Command   ...
Inserting text with "i" and "a"                                21Example                  Meaningsed 4i                 In...
Changing text                                         22Example                          Meaningsed 4c                    ...
Deleting lines                                                   23Command                         Resultsed 10d in_file  ...
Printing lines                                                 24Command                       Resultsed –n 10,/pat1/p in_...
A note on the –r option                                         25Command                    Resultsed -n /([0-9]{3})[0-  ...
Printing with and without "-n"                         option - Comparison                                     26Command  ...
Inserting/reading from a file                                          27Command                        Resultsed 10r my_f...
Writing selectively                                              28Command                          Resultsed 1,10w nf in_...
Transliterating (like ‘tr’)                                      29Command                   Resulty/source/dest/         ...
Quitting from sed                                         30Command                       Resultsed 15q in_file           ...
Grouping multiple commands with { and }                          31sed –n /pat1/,/pat2/ {   s/pat3/pat4/g   s/pat5/pat6/g ...
One liners                                        32Command                       Resultsed p in_file               Duplic...
Additional concepts           not covered                           33 labels, branching to them h, H: copy/append patte...
References                          34 sed & awk by Dale Dougherty & Arnold Robins http://www.grymoire.com/Unix/Sed.html...
Q&A                            35 http://twiki.corp.yahoo.com/view/Main/LpalaniYahoo Devel-sed@yahoo-inc.com
Unanswered questions                           36 How to simulate tail with sed? How to substitute the nth to mth occurr...
Upcoming SlideShare
Loading in …5
×

Sed tips and_tricks

851 views

Published on

sed, known as the “streams editor”, is a powerful tool for text manipulation on UNIX systems. Unlike vim, sed can operate on files of limitless size. The session will cover some of the intermediate concepts such as grouping and back referencing, regular expressions, replacing the nth occurrence of a pattern, operation on every nth line and so on. We will spice up the session with a few “sed one-liners/idioms”

Published in: Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
851
On SlideShare
0
From Embeds
0
Number of Embeds
19
Actions
Shares
0
Downloads
60
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide
  • In /pat1/,+3 if pat1 again appears within the next three lines, it will be ignored.
  • Sed tips and_tricks

    1. 1. Sed – tips and Tricks 1 Logan Palanisamy
    2. 2. Agenda 2 Basics Bio Break Intermediate Concepts Q&A
    3. 3. What is sed 3 non-interactive non-screen oriented Line oriented Input file can be any size Input file not affected
    4. 4. sed syntax 4 sed [options] cmd in_file(s) sed [options] cmd in_file(s) [> out_file] Standard input: sed [options] cmd < in_file [> out_file] Pipelined input: command | sed [options] cmd [> out_file]
    5. 5. Simple examples 5Example Explanationsed s/pat1/pat2/g file Substitute all occurrences of pat1 with pat2sed /pat1/d file delete lines containing pat1 from filesed /pat1/w newfile file save lines containing pat1 to newfilesed –n 30,40p file Print lines 30 to 40sed 10q file Print the top 10 linessed –e s/pat1/pat2/ –e Substitute pat1 with pat2, and pat3 with pat4s/pat3/pat4/ filesed Substitute pat1 with pat2, and pat3 with pat4s/pat1/pat2/;s/pat3/pat4/ file
    6. 6. Different sed options/switches 6Range Remarks-e Used when multiple commands are used on the command line-n Suppress automatic printing of pattern space-f script-file The commands in the script-file get executed-r Use extended regular expressions in the script. With this option, characters such as (, ), {, }, | become meta characters, and dont have to be escaped.-s Consider files as separate rather than as a single continuous long stream-i in-place editing of the input file
    7. 7. The "-f" option - example 7 cat mysed.txt /pat1/ s/this/that/g /pat2/ s/before/after/ sed –f mysed.txt in_file apostrophes not used
    8. 8. With and without "-s" option - Comparison 8Lines Lines sed –n 1,10p f1, f2 sed –ns 1,10p f1, f2in f1 in f24 5 9 lines 4 lines from f1, 5 lines from f26 6 6 lines from f1, 4 lines from f2 6 lines from f1, 6 lines from f212 10 10 lines from f1. No lines from 10 lines from f1, 10 lines from f2 f2
    9. 9. Address Specification 9Range Remarks10 Just the line 101,10 Lines between 1 and 1010,$ Line 10 to end of file10,+3 Line 10 and 3 lines below (lines 10, 11, 12 and 13)10~3 every third line after line 1010, ~3 Line 10 and the next multiple of 3 (lines 10,11 and 12)/pat1/ All lines containing pat1/pat1/,+3 Lines containing pat1 and three lines following it/pat1/,~3 Lines containing pat1 and up to the multiple of 3/pat1/, 20 Lines between the line containing pat1 and line 20 if pat1 appears before line 20. Otherwise, just the line containing pat1/pat1/, /pat2/ lines between the line containing pat1 and line containing pat2
    10. 10. Address Specification with negation 10Range Remarks10! All lines except 10 (! is the negation indicator)1,10! Lines from 11 to end of the file10,$! Lines 1 to 910,+3! All lines except 10, 11, 12 and 1310~3! All lines except line 10 and every third line after that10, ~3! All lines except lines 10, 11, and 12/pat1/! All lines not containing pat1/pat1/,+3! All line except lines containing pat1 and three lines following it/pat1/,~3! All lines containing pat1 and up to the multiple of 3
    11. 11. Regular Expressions 11Meta character Meaning. Matches any single character except newline* Matches zero or more of the character preceding it e.g.: bugs*, table.*^ Denotes the beginning of the line. ^A denotes lines starting with A$ Denotes the end of the line. :$ denotes lines ending with : Escape character (., *, [, , etc)[] matches one or more characters within the brackets. e.g. [aeiou], [a-z], [a-zA-Z], [0-9], [:alpha:], [a-z?,!][^] matches any characters others than the ones inside brackets. eg. ^[^13579] denotes all lines not starting with odd numbers, [^02468]$ denotes all lines not ending with even numbers<, > Matches characters at the beginning or end of words
    12. 12. Extended Regular Expressions 12Meta character Meaning| alternation. e.g.: ho(use|me), the(y|m), (they|them)+ one or more occurrences of previous character. a+ is same as aa*)? zero or one occurrences of previous character.{n} exactly n repetitions of the previous char or group{n,} n or more repetitions of the previous char or group{,m} zero to m repetitions on the previous char or group{n, m} n to m repetitions of previous char or group(....) Used for groupingsed –r ... the "-r" option may have to be used on some version of sed for extended regular expressions to work e.g.: sed –r /(pat1|pat2) s/pat3/pat4/ sed –rn /(pat1|pat2){3,}/ p
    13. 13. Regular Expressions – Examples 13Example Meaning.{10,} 10 or more characters. Curly braces have to escaped[0-9]{3}-[0-9]{2}-[0-9]{4} Social Security number([1-9]{3})[1-9]{3}-[0-9]{4} Phone number (xxx)yyy-zzzz[0-9]{2,3}.[0-9]{1,3}.[0- IP address format9]{1,3}.[0-9]{1,3}[0-9]{3}[ ]*[0-9]{3} Postal code in India[0-9]{5}(-[0-9]{4})? US ZIP Code + 4
    14. 14. Substitution – Format and Options 14 [address[!]]s/pat1/pat2/[options] Options: g – global, w – write, i – ignore case, p – print, n – nth occurrence : or ; +, @ or any other character including space could also be used as the delimiter. Useful when / is part of the search or replacement string.
    15. 15. Substitution - Examples 15Example Explanationsed s/pat1/pat2/ fn Substitute the FIRST occurrence of pat1 with pat2sed s/pat1/pat2/g fn Substitute ALL occurrences of pat1 with pat2sed s/pat1/pat2/3 fn Substitute the third occurrence of pat1 with pat2sed s/pat1/pat2/3g fn Substitute all but the first two occurrences of pat1 with pat2sed s/pat1/pat2/gi fn Substitute ALL occurrences of pat1 with pat2 ignoring the casesed s/pat1/pat2/giw Write to new_files lines containing pat1new_file fn substituting with pat2sed –n s/pat1/pat2/gp fn Print lines containing pat1 substituting with pat2
    16. 16. Substitution – Examples contd 16Example Explanationsed /pat1/ s/pat2/pat3/g fn Substitute all occurrences of pat2 with pat3 on lines containing pat1sed /pat1/! s/pat2/pat3/g Substitute all occurrences of pat2 with pat3 onfn lines NOT containing pat1sed /pat1/,/pat2/ Substitute all occurrences of pat3 with pat4 ons/pat3/pat4/g fn lines between pat1 and pat2 (inclusive)sed 1,100 s/pat1/pat2/gi fn Substitute ALL occurrences of pat1 with pat2 ignoring the case between lines 1 and 100sed /pat1/,/pat2/! Substitute all occurrences of pat3 with pat4 ons/pat3/pat4/g fn lines NOT between pat1 and pat2 (inclusive)sed 1,100! s/pat1/pat2/3i fn Substitute the third occurrence of pat1 with pat2 ignoring the case from 101 to end of filesed 2~5 s/pat1/pat2/w Write every 5th line starting with the 2nd line,new_file fn substituting pat1 with pat2
    17. 17. Substitution – Examples contd. 17Example Explanationsed Substitute all occurrences of either pat1 or pat2s/(pat1|pat2)/pat3/g fn with pat3sed –r s/(pat1|pat2)/pat3/g Same as above. With the –r option, parenthesis andfn alternation characters dont have to be escapedsed –r Substitute all occurrences of either abcpat1 ors/abc(pat1|pat2)/pat3/g fn abcpat2 with pat3sed s/<pat1/pat3/g fn Substitute all occurrences pat1 that begin a word with pat3.sed s/pat1>/pat3/g fn Substitute all occurrences pat1 that is end of a word with pat3sed s/<pat1>/pat3/g fn Substitute all occurrences of the word pat1 with pat3. Note: The angular brackets (< and >) have to be escaped even with the –r option.sed s/[a-z]/u&/g fn Substitute all lower case letters to upper case letterssed s/./&&/g fn Double each character
    18. 18. Grouping and Back Referencing 18 Parts of strings in the Search/Left-hand side can be grouped and referenced in the Replacement/Right- hand side Up to nine groups possible (1, 2, ..9) Groups can be nested or referenced back on the Search side Same group can be referenced any number of times
    19. 19. Substitution with Grouping and Back Referencing. Examples 19Command Explanationsed s/^(.*):(.*)/2:1/ fn Swap two fields delimited with :. "column A:column B" becomes "column B:column A"sed –r s/^(.*):(.*)/2:1/ fn Same as above. With the –r option, the parentheses dont have to be escaped.sed –r s/^([^:]*):([^:]*)/2:1/ Same as above. Exchanges only the first twofn columns even if there are more.sed –r s/^(This (.*) nested)/2 Group 1 contains everything between "This ..1/ fn nested". Group 2 contains just the characters between "This" and "nested".
    20. 20. Substitution with Grouping and Back Referencing. Examples 20Command Explanationsed –r Print lines with six or seven character long(w)(w)(w)w?321/p fn palindromessed –r Convert six char palindrome strings tos/(.)(.)(.)321/123123/g repetitive strings. Note: Any embedded or fn trailing six spaces also will matchsed -r :a;s/(^|[^0-9.])([0- Add comma as thousands separator (Think9]+)([0-9]{3})/12,3/g;ta how easy it would have if we were to do thisnumbers.txt from the left side)sed –r s/(.*)/11/ fn Concatenate the string at the line levelsed –r s:(.*):11: fn Same as above, but using : as the delimiter between search and replacement strings
    21. 21. Inserting text with "i" and "a" 21Example Meaningsed 4i Inserts two lines before line 4my text 1my text 2 in_filesed /pat1/ i Inserts two lines before every line that containsmy text 1 pat1my text 2 in_filesed 4a Inserts two lines after line 4my text 1my text 2 in_filesed $a Inserts two lines after the end of the filemy text 1my text 2 in_file
    22. 22. Changing text 22Example Meaningsed 4c Replace line 4 with the two new linesmy text 1my text 2 in_filesed /pat1/ c Replace all lines containing pat1 with the twomy text 1 new linesmy text 2 in_filesed /pat1/, /pat2/ c Replace all lines between pat1 and pat2 with themy text 1 two new linesmy text 2 in_fileChange affects the whole line.Substitute just the matching words or strings on the line
    23. 23. Deleting lines 23Command Resultsed 10d in_file Delete the 10th linesed 1,10d in_file Delete lines between 1 and 10sed 10,$d in_file Delete lines from 10 to end of filesed 10,+3d in_file Delete line 10 and 3 lines below (lines 10, 11, 12 and 13)sed 10~3d in_file Delete every third line after line 10sed 10, ~3d in_file Delete line 10 and up to the next multiple of 3 (lines 10,11 and 12)sed /pat1/d in_file Delete all lines containing pat1sed /pat1/,+3d in_file Delete all lines containing pat1 and three lines following itsed /pat1/,~3d in_file Delete lines containing pat1 and up to next the multiple of 3sed /pat1/!d in_file Delete all lines NOT containing pat1sed –r /[0-9]{8,}/d in_file Delete lines containing 8 or more digits
    24. 24. Printing lines 24Command Resultsed –n 10,/pat1/p in_file Print lines from 10 to next line containing pat1sed –n 10,+3p in_file Print line 10 and 3 lines below (lines 10, 11, 12 and 13)sed –n 10~3p in_file Print every third line after line 10sed –n 10, ~3p in_file Print line 10 and up to the next multiple of 3 (lines 10,11 and 12)sed –n /pat1/p in_file Print all lines containing pat1sed –n /pat1/,+3p in_file Print all lines containing pat1 and three lines following itsed –n /pat1/,~3p in_file Print lines containing pat1 and up to next the multiple of 3sed –n /pat1/,/pat2/p Repetitively print lines between the ranges containingin_file pat1 and pat2sed –n /pat1/!p in_file Print all lines NOT containing pat1sed -n /([0-9]{3})[0- Print lines containing phones numbers.9]{3}-[0-9]{4}/pin_file
    25. 25. A note on the –r option 25Command Resultsed -n /([0-9]{3})[0- Print lines containing phones numbers like (408)806-9]{3}-[0-9]{4}/p 8330, (408)349-3699in_filesed -nr /([0- Same as above with the –r option. Note the escaping of9]{3})[0-9]{3}-[0- parentheses. Without escaping, ( and ) become meta9]{4}/p in_file characters, not part of the search string.
    26. 26. Printing with and without "-n" option - Comparison 26Command Commentssed 1,10p fn Lines 1 to 10 are printed twice. Rest of the lines are printed oncesed –n 1,10p fn Lines 1 to 10 are printed just once.sed s/pat1/pat2/p fn Lines containing pat1 are printed twice after substituting pat1 with pat2. Other lines are printed oncesed –n s/pat1/pat2/p Only lines containing pat1 are printed afterfn substituting pat1 with pat2
    27. 27. Inserting/reading from a file 27Command Resultsed 10r my_file in_file Insert my_file after the 10th linesed 1,10r my_file in_file Insert my_file after each line between 1 and 10sed $r my_file in_file Append my_file at the endsed 10,+3r my_file in_file Insert my_file after every line between10 and 13.sed 10~3r my_file in_file Insert my_file every third line after line 10sed /pat1/,/pat2/r my_file Insert my_file after every line between pat1 and pat2in_file
    28. 28. Writing selectively 28Command Resultsed 1,10w nf in_file Write lines 1 to 10 to new file "nf"sed /^pat1/w nf in_file Write all lines beginning with pat1 to nfsed /pat1$/w nf in_file Write lines that end with pat1sed /<pat1>/w nf in_file Write lines that contain the WORD pat1 to nfsed –e 1~2w odd_lines Write odd numbered lines to odd_lines, and even-e 2~2w even_lines` numbered files to even_linesin_filesed /pat1/, /pat2/w nf Write all lines between pat1 and pat2 to nfin_filesed –r /[0-9]{3}-[0-9]{2}-[0- Write lines containing Social Security Number nnn-nn-nnnn to the9]{4}w ssn in_file file ssn. Note the use of –r option. Otherwise, { and } have to be escaped.sed –r /(.)(.)(.)321/w Write lines containing palindromes to palin.txtpalin.txt in_file
    29. 29. Transliterating (like ‘tr’) 29Command Resulty/source/dest/ One to one character by character substitution. source and dest strings have to be of the same lengthsed y/13579/aeiou/ fn replace 1 by a, 3 by e, and so on
    30. 30. Quitting from sed 30Command Resultsed 15q in_file Quit after the 15th line. Print 14 linessed /<[Ee]nd>/q in_file Quit after encountering the word End or end
    31. 31. Grouping multiple commands with { and } 31sed –n /pat1/,/pat2/ { s/pat3/pat4/g s/pat5/pat6/g w new_file p } inp_file
    32. 32. One liners 32Command Resultsed p in_file Duplicate all the lines one below the othersed /^$/ d in_file Delete the blank lines from the input filesed ‘s/./&:/80’ Add a colon after the 80th character. (replace the eightieth occurrence of the pattern with itself and a colon)sed –r ‘s/^(.*) 1/1/ fn Replace duplicate stringssed G in_file Add a blank line after everyline.find . –type f –name Replace all occurrences of TableA with TableB in allmy_files.*.sql –exec sed –i my SQL scripts‘s/TableA/TableB/g’ {} +More oneliners http://www.catonmat.net/blog/sed-one-liners- explained-part-one/
    33. 33. Additional concepts not covered 33 labels, branching to them h, H: copy/append patternspace to holdspace g, G: copy/append holdspace to patternspace x: exchange the contents of the hold and pattern spaces
    34. 34. References 34 sed & awk by Dale Dougherty & Arnold Robins http://www.grymoire.com/Unix/Sed.html http://sed.sourceforge.net/sedfaq3.html http://en.wikipedia.org/wiki/Sed http://groups.yahoo.com/group/sed-users/ http://sed.sourceforge.net/sed1line.txt
    35. 35. Q&A 35 http://twiki.corp.yahoo.com/view/Main/LpalaniYahoo Devel-sed@yahoo-inc.com
    36. 36. Unanswered questions 36 How to simulate tail with sed? How to substitute the nth to mth occurrences of pat1 with pat2? How to substitute the last N occurrences or the nth occurrence from the end? How to identify palindrome of any length?

    ×