Sed & awk the dynamic duo

Harness the power of

Sed & Awk
Everything a PHP developer should know about sed & awk

Edition: PHPBenelux, jan 29, 2011, Antwerp

http://en.wikipedia.org/wiki/File:Slender_Loris.jpg Sed & Awk - http://joind.in/2489

First, who am I?

Sed & Awk - http://joind.in/2489

First, who am I?
Joshua Thijssen (32)


First, who am I?

Senior Software Engineer currently working
at Enrise (4worx)


First, who am I?

at Enrise (4worx)

Development in PHP, Python, Perl, C, java,
assembly.


First, who am I?

at Enrise (4worx)

assembly.

Certiﬁed MySQL DBE, MySQL DBA,
LPIC-1, LPIC-2, Zend PHP5, Zend PHP5.3,
Zend Framework, Ubuntu professional.


First, who am I?

at Enrise (4worx)

assembly.


Blogs: http://www.adayinthelifeof.nl


First, who am I?

at Enrise (4worx)

assembly.


http://www.enrise.com/blog


First, who am I?

at Enrise (4worx)

assembly.



Email: joshua@enrise.com


First, who am I?

at Enrise (4worx)

assembly.



Email: joshua@enrise.com
Twitter: @jaytaph


Comfort zones

http://www.ﬂickr.com/photos/gingerblokey/3597880429/sizes/l/in/photostream/ Sed & Awk - http://joind.in/2489

Comfort zones

http://www.gnux-consultant.fr/service/ﬁleadmin/templates/template_fr_1_FILES/elephant-php.gif Sed & Awk - http://joind.in/2489

Comfort zones

You are a PHP programmer...


Comfort zones

You are a PHP programmer...

...so you program in PHP


Comfort zones

You are doing this already:


Comfort zones


• SQL (where, order, limit, join)


Comfort zones


• Frameworks (Zend, Symfony, etc)


Comfort zones


• Frameworks (Zend, Symfony, etc)
• JQuery, Dojo, etc


Comfort zones
But why learn new stuff when you can do it in PHP?


Comfort zones

✓ Might be easier to use...


Comfort zones

✓ Might be faster to write...


Comfort zones

✓ Might be better suited for the job...


Comfort zones

✓ Might be better suited for the job...
✓ More efﬁcient


Comfort zones

!

Comfort zones

I don’t want to tell you HOW to use Sed & Awk.

!

Comfort zones


I want to tell you that for certain jobs, tools
like Sed & Awk are much better suited than
PHP.

!

Comfort zones


I want to tell you that for certain jobs, tools
like Sed & Awk are much better suited than
PHP.

! Know the capabilities of your tools and you
become a better developer...


Comfort zones

Why Sed & Awk?


Comfort zones

Why Sed & Awk?
✓Useful for data manipulation


Comfort zones

Why Sed & Awk?
✓They work well together


Comfort zones

Why Sed & Awk?
✓Both have a similar processing method


Comfort zones

Why Sed & Awk?
✓Both rely heavily on regular expressions


Comfort zones

Why Sed & Awk?
✓Both rely heavily on regular expressions
✓Nobody really harvest their power

Part I : SED

http://www.ﬂickr.com/photos/joachim_s_mueller/4298348196/ Sed & Awk - http://joind.in/2489

SED


SED

• is a Stream EDitor


SED

• applies rules based on a stream of data
(per line)


SED

• applies rules based on a stream of data
(per line)
• there is no turning back into the stream
(going forward only)


Why use SED?
Useful for:


Why use SED?
Useful for:
• Changing IP addresses or other data through many ﬁles.
mutation of large datasets


Why use SED?
Useful for:

• complex ﬁndin&certain blocks of code/data (for instance,
Only change data
replace
CSV, TXT, SQL ﬁles, docblocks etc)


Why use SED?
Useful for:

• complex findin&certain blocks of code/data (for instance,
Only change data
replace
CSV, TXT, SQL files, docblocks etc)

• complexthe next 10 lines after each 404 code read from an
Only print
retrieval of data
apache log file or print all docblocks and function headers


When use SED?
Use sed when:

Don’t use sed when:


When use SED?
Use sed when:
• When you need to change hundreds or
thousands of ﬁles
• “Complex” mutations
• Fast “one liners” in scripts
Don’t use sed when:
• When you need to change one or two items
• When you need aggregation or variables

SED
Most common example:

sed ‘s/foo/bar/g’ old > new

changes ‘foo’ into ‘bar’ throughout the
ﬁle ‘old’ and places output into ﬁle ‘new’

SED

$ cat foo.txt
foo bar foo
bar foo bar
foo bar foo
bar bar foo
foo foo bar
$ sed 's/foo/bar/g' foo.txt
bar bar bar
bar bar bar
bar bar bar
bar bar bar
bar bar bar


SED
Another common example:

sed ‘s/foo//g’ old > new

deletes ‘foo’ throughout the ﬁle
‘old’ and places output into ﬁle ‘new’

SED

$ cat foo.txt
foo bar foo
bar foo bar
foo bar foo
bar bar foo
foo foo bar
$ sed 's/foo//g' foo.txt
bar
bar bar
bar
bar bar
bar


SED
A bit more advanced:

sed ‘s/foo/FOO/2’ old > new

changes the second ‘foo’ on each line into ‘FOO’

SED

$ cat foo.txt
foo bar foo
bar foo bar
foo bar foo
bar bar foo
foo foo bar
$ sed 's/foo/FOO/2' foo.txt
foo bar FOO
bar foo bar
foo bar FOO
bar bar foo
foo FOO bar


SED
Sed can use address ranges:

sed ‘1,3 s/foo/bar/g’ file

changes all ‘foo’s to ‘bar’s on lines 1 to 3


SED

$ cat foo.txt
foo bar foo
bar foo bar
foo bar foo
bar bar foo
foo foo bar
$ sed '1,3 s/foo/bar/g' foo.txt
bar bar bar
bar bar bar
bar bar bar
bar bar foo
foo foo bar


SED
But you can also use a regex:

sed ‘1,/^$/ s/foo/bar/g’ file

changes all ‘foo’s to ‘bar’s on
lines 1 to the ﬁrst empty line

SED
$ cat foo.txt
foo bar foo
bar foo bar

foo bar foo
bar bar foo
foo foo bar
$ sed '1,/^$/ s/foo/bar/g' foo.txt
bar bar bar
bar bar bar

foo bar foo
bar bar foo
foo foo bar


SED
$ cat foo.txt
foo bar foo
bar foo bar

foo bar foo
bar bar foo
foo foo bar
$ sed '/^$/,$ s/foo/bar/g' foo.txt
foo bar foo
bar foo bar

bar bar bar
bar bar bar
bar bar bar


SED
A ! negates the address range:

sed ‘1,3 ! s/foo/bar/g’ file

changes all ‘foo’s to ‘bar’s on
every line except lines 1 to 3

SED

$ cat foo.txt
foo bar foo
bar foo bar
foo bar foo
bar bar foo
foo foo bar
$ sed '1,3 ! s/foo/bar/g' foo.txt
foo bar foo
bar foo bar
foo bar foo
bar bar bar
bar bar bar


SED
Multiple commands per range:

sed ‘1,3 { s/foo/bar/g ; s/.*/Line &/ ; }’ file

sed ‘1,3 {
s/foo/bar/g
s/.*/Line &/
}’ file

for line 1 to 3: change foo’s into bar’s
and prepend ‘Line’ to the line

SED

$ cat foo.txt
foo bar foo
bar foo bar
foo bar foo
bar bar foo
foo foo bar
$ sed '1,3 { s/foo/bar/g ; s/.*/Line &/ ; }' foo.txt
Line bar bar bar
Line bar bar bar
Line bar bar bar
bar bar foo
foo foo bar


SED
Multiple ranges:
sed ‘1,3 s/foo/bar/g
5,7 s/bar/foo/g
s/(.*)/Line 1/’ file

on line 1 to 3: change foo’s into bar’s
on line 5 to 7: change bar’s into foo’s
on all lines: add ‘Line’ in front of the line

SED

$ cat foo.txt
foo bar foo
bar foo bar
foo bar foo
bar bar foo
foo foo bar
$ sed '1,3 s/foo/bar/g ; 5,7 s/bar/foo/g ; s/.*/Line: &/' foo.txt
Line: bar bar bar
Line: bar bar bar
Line: bar bar bar
Line: bar bar foo
Line: foo foo foo


SED

sed -n ‘
/^cut/ q
1,3 { s/foo/bar/g ; p ; }
4,$ { s/bar/foo/g ; p ; }
’ file

-n means ‘don’t print’ lines


SED

sed -n ‘
/^cut/ q
1,3 { s/foo/bar/g ; p ; }
4,$ { s/bar/foo/g ; p ; }
’ file

if a line starts with ‘cut’, end processing


SED

sed -n ‘
/^cut/ q
1,3 { s/foo/bar/g ; p ; }
4,$ { s/bar/foo/g ; p ; }
’ file

line 1 to 3 will replace ‘foo’ to ‘bar’ and
print the line to output

SED

sed -n ‘
/^cut/ q
1,3 { s/foo/bar/g ; p ; }
4,$ { s/bar/foo/g ; p ; }
’ file

line 4 to the end will replace ‘bar’ to
‘foo’ and print the line to output

SED
$ cat file.txt
foo bar foo bar
bar bar foo foo
foo bar foo bar
bar bar foo foo
foo bar foo bar
foo bar foo bar
bar bar foo foo
bar bar foo foo
cut
this line is not added
$ sed -n '/^cut/ q ; 1,3 { s/foo/bar/g ; p ; } ; 4,$ { s/bar/foo/g ; p ; }' file.txt
bar bar bar bar
bar bar bar bar
bar bar bar bar
foo foo foo foo
foo foo foo foo
foo foo foo foo
foo foo foo foo
foo foo foo foo


SED commands
a append ‘text’ to output
s substitute data
y transform data (like ‘tr’)
d delete line (don’t print)
n (print) and goto next line
N (print) and add next line to pattern space
p print pattern space
q quit processing
r copy contents of ﬁle to pattern space
# comment
= prints current line number

Commenting out
[production]
...
bootstrap.path = APPLICATION_PATH "/Bootstrap.php"
bootstrap.class = "Bootstrap"
resources.frontController.controllerDirectory = APPLICATION_PATH "/controllers"

[staging : production]

[testing : production]
phpSettings.display_startup_errors = 1
phpSettings.display_errors = 1

[development : production]
...

sed '/^[testing/,/^[/ s/.*/# --- &/' application.ini


Commenting out

sed '/^[testing/,/^[/ s/.*/# --- &/' application.ini

[production]
...


# --- [testing : production]
# --- phpSettings.display_startup_errors = 1
# --- phpSettings.display_errors = 1
# ---
# ---[development : production]
...


Commenting out
sed '/^[testing/,/^[/ {
/^[/ b
/^$/ b
s/.*/# --- &/
}' application.ini
[production]
...


[testing : production]
# --- phpSettings.display_startup_errors = 1
# --- phpSettings.display_errors = 1

[development : production]
...


SED ﬂow control

b <label> Unconditionally branch to <label>
: <label> Set <label>
t <label> Conditionally branch to <label>


SED ﬂow control
solfege.txt sed script
do sed '
re :loop
mi /$/ N
fa s/n */ /
so t loop
la ' solfege.txt
si
do
Pattern buffer

Output


SED ﬂow control
do sed '
re :loop
mi /$/ N
fa s/n */ /
so t loop
la ' solfege.txt
si
do
Pattern buffer
do

Output


SED ﬂow control
do sed '
re :loop
mi /$/ N
fa s/n */ /
so t loop
la ' solfege.txt
si
do
Pattern buffer
re

Output
do


SED ﬂow control
do sed '
re :loop
mi /$/ N
fa s/n */ /
so t loop
la ' solfege.txt
si
do
Pattern buffer
re
mi

Output
do


SED ﬂow control
do sed '
re :loop
mi /$/ N
fa s/n */ /
so t loop
la ' solfege.txt
si
do
Pattern buffer
re mi

Output
do


SED ﬂow control
do sed '
re :loop
mi /$/ N
fa s/n */ /
so t loop
la ' solfege.txt
si
do
Pattern buffer
re mi
fa

Output
do


SED ﬂow control
do sed '
re :loop
mi /$/ N
fa s/n */ /
so t loop
la ' solfege.txt
si
do
Pattern buffer
re mi fa

Output
do


SED ﬂow control
do sed '
re :loop
mi /$/ N
fa s/n */ /
so t loop
la ' solfege.txt
si
do
Pattern buffer
so

Output
do
re mi fa


Sed not powerful?
Delete all lines containing ‘regex’
sed '/regex/d' filename

Remove all additional white spaces
sed 's/ *$//' filename

Reverse all lines in a ﬁle (makes use of the hold buffer)
sed -n '1 ! G ; h ; $ p' filename


Sed not powerful?
Meet Sedtris:
A fully functional Tetris clone written in SED

http://uuner.doslash.org/forfun/

So use sed when:


So use sed when:

• Repetitive work on many ﬁles


So use sed when:

• “complex” mutations


So use sed when:

• “complex” mutations
• Fast “oneliners” in scripts etc..


Part 2 : AWK

http://www.ﬂickr.com/photos/joachim_s_mueller/138409464/in/photostream/ Sed & Awk - http://joind.in/2489

AWK


AWK

• AWK is a full-ﬂedged programming
language.


AWK

language.
• There is NO way I can teach you AWK in
+- 20 minutes


AWK

language.
• There is NO way I can teach you AWK in
+- 20 minutes
• But i’ll try...


AWK
• Alfred V. Aho, Peter J.Weinberger, Brain W.
Kernighan


AWK
Kernighan
• Written in 1977 at AT&T Bell Laboratories


AWK
Kernighan
• Multiple versions: AWK, NAWK, GAWK,
MAWK and more...


AWK
Kernighan
• Multiple versions: AWK, NAWK, GAWK,
MAWK and more...
• Pattern-directed scanning and processing
language...


AWK

• [condition] { actions }


AWK

• [condition] { actions }
• 2 special “patterns” : BEGIN and END


Simple AWK
$ cat solfege.txt
do
re
mi
fa
sol
la
ti
do
$ awk '
BEGIN { print "start" }
/o/ { print "I just saw an o in " $0 }
END { print "the end" }' solfege.txt
start
I just saw an o in do
I just saw an o in sol
I just saw an o in do
the end


Apache logfile (combined)
Awk processes through “records” and “fields”
72.30.161.230 - - [18/Jan/2011:20:28:09 +0100] "GET /robots.txt HTTP/1.0" 200 387
"-" "Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/
slurp)"
72.30.161.230 - - [18/Jan/2011:20:28:10 +0100] "GET / HTTP/1.0" 200 7235 "-"
"Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/
slurp)"

you can control the record and field separators


72.30.161.230
-
-
[18/Jan/2011:20:28:09
+0100]
"GET
/robots.txt
HTTP/1.0"
200
387
"-"
"Mozilla/5.0
(compatible;
Yahoo!
Slurp;
http://help.yahoo.com/help/us/ysearch/slurp)"

72.30.161.230 - - [18/Jan/2011:20:28:10 +0100] "GET / HTTP/1.0" 200 7235 "-" "Mozilla/5.0
(compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)"

Field separator “space”


72.30.161.230 - - [18/Jan/2011:20:28:09 +0100]
GET /robots.txt HTTP/1.0
200 387
-

Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)

72.30.161.230 - - [18/Jan/2011:20:28:10 +0100] "GET / HTTP/1.0" 200 7235 "-" "Mozilla/5.0
(compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)"

Field separator “double quote” (“)

Some global AWK knowledge



• You can set the ﬁeld and record separator



• $FS=”|”; $RS=”t”



• $FS=”|”; $RS=”t”
• $0 holds the complete record (line)



• $FS=”|”; $RS=”t”
• $1 holds first field, $2 second field etc...



• $FS=”|”; $RS=”t”
• $NF holds number of ﬁelds in record



• $FS=”|”; $RS=”t”
• $NF holds number of ﬁelds in record
• $NR holds CURRENT record


Print the “user agents” from the logﬁle

$ awk -F" '{ print $6 }' apache.log
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322) NS8/0.9.6
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322) NS8/0.9.6
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)


and count them (through external tools)

$ awk -F" '{ print $6 }' apache.log | sort | uniq -c
2 Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322) NS8/0.9.6
1 Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
7 Mozilla/5.0 (compatible; Yahoo! Slurp; http://help.yahoo.com/help/us/ysearch/slurp)


Print the total bytes send out per status code

$ awk -F '{ totals[$9] += $10; }
END { for (i in totals) { printf "%d : %d bytesn", i, totals[i]; } }' apache.log
200 : 26197250 bytes
206 : 180578 bytes
301 : 31072 bytes
302 : 2991 bytes
304 : 44715 bytes
404 : 82866 bytes
500 : 361783 bytes


who triggered a 4xx code

$ awk -F '$9 ~ /4[0-9][0-9]/ { FS="""; $0=$0; print $6; FS=" " }' apache.log
Googlebot/2.1 (+http://www.googlebot.com/bot.html)
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.2) Gecko/20100115 Firefox/3.6
libwww-perl/5.805
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.6) Gecko/2009011913 Firefox/3.0.6
Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.6) Gecko/2009011913 Firefox/3.0.6
Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; en-US; rv:1.8b4) Gecko/20050908 Firefox/1.4
Mozilla/5.0 (Macintosh; U; PPC Mac OS X Mach-O; en-US; rv:1.8b4) Gecko/20050908 Firefox/1.4
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1)



Awk one liner compared to PHP:
$ awk -F '{ totals[$9] += $10; } END { for (i in totals) { printf "%d : %d bytesn", i, totals[i]; } }' apache.log

<?php

$hash = array();
foreach( file( './apache.log' ) as $line ) {
list( , , , , , , , , $status, $bytes ) = explode( ' ', $line );

if( !isset( $hash[$status] ) ) {
$hash[$status] = 0;
}

$hash[$status] += $bytes;
}
print_r($hash);

Not a whole lot different, but already more
complex and this was just a simple example...

credits to @RichardJ #pfz channel Sed & Awk - http://joind.in/2489


Sed one liner compared to PHP:

sed ‘/^.o/d’ file

<?php
    $stdin = fopen("php://stdin", "r");
    while (!feof($stdin)) {
        $line = fgets($stdin);
        if (preg_match("/^.o/", $line)) continue;
        print $line;
    }
?>

Much more work....

credits to @RichardJ #pfz channel Sed & Awk - http://joind.in/2489

Practical uses for a (PHP) developer

• parse php-errors files, syslog files, apache’s
http access logs.
• Conversion of files you get from your
customers, who always assume you can do
magic with a gazzillion GB’s of (unsorted)
data (and now you can).


In conclusion: sed & awk



• are powerful for simple one-liners but can
also be used for complex programs



• integrates perfectly with other (unix) tools
like uniq, sort, cut, ﬁnd, grep, cat, etc...



• integrates perfectly with other (unix) tools
like uniq, sort, cut, ﬁnd, grep, cat, etc...
• are a great way to automate complex and/
or repetitive (editing) tasks


In conclusion

http://ﬁles.sharenator.com/slender_loris_Worlds_strangest_looking_animals-s300x451-2279-580.jpg Sed & Awk - http://joind.in/2489

In conclusion

• Look outside your
comfort zone for other
(better) tools.


In conclusion

• Look outside your
comfort zone for other
(better) tools.
• Can you think of
examples where you
would use Sed or Awk
(instead of php?)


Questions?

http://static.squidoo.com/resize/squidoo_images/250/draft_lens2806042
module29556622photo_1240639056fry-panique-questions.jpg Sed & Awk - http://joind.in/2489

Read more on Sed & Awk

Sed:
http://www.gnu.org/software/sed/manual/html_node/index.html
http://www.grymoire.com/Unix/Sed.html
http://www.panix.com/~elﬂord/unix/sed.html
http://www.linuxtopia.org/online_books/linux_tool_guides/the_sed_faq/index.html

Awk:
http://www.gnu.org/software/gawk/
http://www.grymoire.com/Unix/Awk.html


Thank you for your
attention!

Don’t forget to rate
my talk on joind.in

http://joind.in/2489

http://farm5.static.ﬂickr.com/4078/4790219776_2fe3c9af95_b.jpg Sed & Awk - http://joind.in/2489

Sed & awk the dynamic duo

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to Sed & awk the dynamic duo

Similar to Sed & awk the dynamic duo (20)

More from Joshua Thijssen

More from Joshua Thijssen (14)

Sed & awk the dynamic duo

Editor's Notes