• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Shooting from the hip - Perl at the command line for DBAs
 

Shooting from the hip - Perl at the command line for DBAs

on

  • 2,555 views

There are many GUI applications to deal with databases and with MySQL in particular. But for die hard command line enthusiasts, who grew up with Linux, the shell prompt is still prince. You would be ...

There are many GUI applications to deal with databases and with MySQL in particular. But for die hard command line enthusiasts, who grew up with Linux, the shell prompt is still prince. You would be surprised at the vast amount of operations that you can do from the command line, and then many would surely marvel at the amount of work that a Perl one-liner can do.
By combining shell and Perl, you get the kind of power that all DBAs dream, but few achieve.
Running commands on several databases at once, filtering users, getting detailed metadata, transferring data from the operating system to a database and the other way around, creating scripts from database data, creating SQL
commands from database data. That, and more, you can learn by attending this session. Learn the art of the command line as a shortcut to quick and efficient administration, and as help for demanding developers.

Statistics

Views

Total Views
2,555
Views on SlideShare
2,530
Embed Views
25

Actions

Likes
1
Downloads
23
Comments
0

3 Embeds 25

http://www.jugsardegna.org 21
url_unknown 3
https://twitter.com 1

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

CC Attribution-NonCommercial-NoDerivs LicenseCC Attribution-NonCommercial-NoDerivs LicenseCC Attribution-NonCommercial-NoDerivs License

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Shooting from the hip - Perl at the command line for DBAs Shooting from the hip - Perl at the command line for DBAs Presentation Transcript

    • Shooting from the hip. Perl at the command :) @ @ line for DBAs - $_ #! Giuseppe Maxia MySQL Community Team Lead @datacharmer This work is licensed under the Creative Commons Attribution-Share Alike 3.0 Unported License. Friday, 06 August 2010 1
    • about me -Giuseppe Maxia  a.k.a. The Data Charmer  MySQL Community Team Lead  Long time hacking with MySQL features  Formerly, database consultant, designer, coder.  A passion for QA  An even greater passion for open source  ... and community  Passionate blogger  http://datacharmer.blogspot.com Friday, 06 August 2010 2
    • Updated presentation slides  you will find an up to date copy of these slides at:  http://tinyurl.com/shoot-cl-2010  http://tinyurl.com/shoot-cl-full-2010 Friday, 06 August 2010 3
    • I love your feedback!  Tell it to the Twitter community  #yapc_eu "your feedback here"  Twitter me  @datacharmer "your feedback here"  Blog about it Friday, 06 August 2010 4
    • This is a story about languages Friday, 06 August 2010 5
    • you got to understand, Bernie, it's not personal. It's just that you are obsolete We use language to talk to computers Friday, 06 August 2010 6
    • 10 print "hello" We use 20 goto 10 languages to talk to computers Friday, 06 August 2010 7
    • I am a polyglot Friday, 06 August 2010 8
    • An imperfect polyglot, actually Friday, 06 August 2010 9
    • I speak Italian Friday, 06 August 2010 10
    • I speak Italian with a clear Sardinian accent Friday, 06 August 2010 10
    • I speak English Friday, 06 August 2010 11
    • I speak English with a distinct Italian accent Friday, 06 August 2010 11
    • I speak Perl Friday, 06 August 2010 12
    • I speak Perl with a hint of a C accent Friday, 06 August 2010 12
    • I speak French and Spanish Friday, 06 August 2010 13
    • I speak French and Spanish but since I use them very little, they have melted with Italian and English Friday, 06 August 2010 13
    • I speak #!/bin/bash Bash Friday, 06 August 2010 14
    • I speak #!/bin/bash Bash with a strong Perl accent Friday, 06 August 2010 14
    • I speak fluent Italian La donzelletta vien dalla campagna, in sul calar del sole, col suo fascio dell'erba e reca in mano un mazzolino di rose e viole, onde, siccome suole, ornare ella si appresta ... Friday, 06 August 2010 15
    • However, Sardinian comes handy sometimes Maissavida ti frimisti! (*) (*) May you never stop Friday, 06 August 2010 16
    • I speak fluent English When in the Course of human events, it becomes necessary for one people to dissolve the political bands which have connected them with another, and to assume among the powers of the earth, the separate and equal station to which the Laws of Nature and of Nature's God entitle them, a decent respect to the opinions of mankind requires that they should declare the causes which impel them to the separation. Friday, 06 August 2010 17
    • I speak fluent English When in the Course of human events, it becomes necessary for one people to dissolve the political bands which have connected them with another, and to assume among the powers of the earth, the separate and equal station to which the Laws of Nature and of Nature's God entitle them, a decent respect to the opinions of mankind requires that they should declare the causes which impel them to the separation. hmm. wrong flag Friday, 06 August 2010 17
    • I speak fluent English When in the Course of human events, it becomes necessary for one people to dissolve the political bands which have connected them with another, and to assume among the powers of the earth, the separate and equal station to which the Laws of Nature and of Nature's God entitle them, a decent respect to the opinions of mankind requires that they should declare the causes which impel them to the separation. Friday, 06 August 2010 17
    • But sometimes Italian is still necessary barista al fresco pasta maestro espresso tempo vendetta soprano finale piano dilettante cappuccino stucco stiletto virtuoso fiasco presto lotto pizza bravo lasagna studio spaghetti stanza Friday, 06 August 2010 18
    • And sometimes Italian is needed I tuoi perversi defunti!(*) (*) Your wicked ancestors! Friday, 06 August 2010 19
    • Although Sardinian is still effective here Ancu ti currullidi sa giustizia (*) (*) May the rule of law chase you Friday, 06 August 2010 20
    • I speak Perl fluently #!/usr/bin/perl use strict; use warnings; use File::Find; while (my $line = <stdin>){ # … Friday, 06 August 2010 21
    • Sometimes, I need help from C use Inline C; print "9 + 16 = ", add(9, 16), "n"; __END__ __C__ int add(int x, int y) { return x + y; } Friday, 06 August 2010 22
    • Most of my work at the command line is Bash $ for D in $HOME/*/; do du -sh $D; > done > /tmp/occupancy #!/bin/bash Friday, 06 August 2010 23
    • And sometimes, a pinch of Perl can help $ perl -pe 's{(dd)-(dd)-(dddd)}{$3-$1-$2}g' < rawfile.txt > betterfile.txt #!/bin/bash Friday, 06 August 2010 24
    • And then on top of all this, I need to speak SQL $ mysql -e 'SELECT COUNT(*) FROM dbname.tbl_name WHERE col_name = 1 and other_col="A" ' Friday, 06 August 2010 25
    • I wish I could say that my SQL has a Perlish accent $ mysql -e 'SELECT COUNT(*) FROM dbname.tbl_name WHERE col_name = 1 and other_col="A" ' use strict; use DBI;? Friday, 06 August 2010 26
    • I wish I could say that my SQL has a Perlish accent $ mysql -e 'SELECT COUNT(*) FROM dbname.tbl_name WHERE col_name = 1 and other_col="A" ' but it wouldn't be true Friday, 06 August 2010 27
    • I could curse in Perl $ perl -pe 's{(dd)-(dd)-(ddd d)}{$3-$1-$2}g' < rawfile.txt > betterfile.txt (*) But it would not get me far. (*) already translated Friday, 06 August 2010 28
    • Then Friday, 06 August 2010 29
    • Then I will need to mix languages. Friday, 06 August 2010 29
    • Then I will need to mix languages. This talk will show exactly that. Friday, 06 August 2010 29
    • Then I will need to mix languages. This talk will show exactly that. Mixing languages. Friday, 06 August 2010 29
    • Then I will need to mix languages. This talk will show exactly that. Mixing languages. Yes. Friday, 06 August 2010 29
    • Then I will need to mix languages. This talk will show exactly that. Mixing languages. Yes. $ at the command line Friday, 06 August 2010 29
    • What this tutorial is about  Providing tools and tips for command line users;  Learn a few tricks to make you achieve more and type less. Friday, 06 August 2010 30
    • What this tutorial is about  Providing tools and tips for command line users;  Learn a few tricks to make you achieve more and type less.  Making GUI enthusiasts green with envy; bar foo baz Friday, 06 August 2010 30
    • the command line (for rookies) BASICS 31 Friday, 06 August 2010 31
    • The shell • It's a command interpreter • There are many flavors > bash > zsh > ksh 32 Friday, 06 August 2010 32
    • MySQL command line client • called "mysql" > Takes SQL commands > sends them to the server > gets results • but it can do much more ... 33 Friday, 06 August 2010 33
    • other tools • Shell scripts • Aliases • Perl • awk • sort • …. and a lot of Unix little helper 34 Friday, 06 August 2010 34
    • other tools • Shell scripts • Aliases • Perl which could replace everything else, of course • awk • sort • …. and a lot of Unix little helper 34 Friday, 06 August 2010 34
    • in case it wasn't clear (yeah, for rookies) 35 Friday, 06 August 2010 35
    • in case it wasn't clear (yeah, for rookies) • command line actions are 35 Friday, 06 August 2010 35
    • in case it wasn't clear (yeah, for rookies) • command line actions are > commands (the name of the application to invoke) 35 Friday, 06 August 2010 35
    • in case it wasn't clear (yeah, for rookies) • command line actions are > commands (the name of the application to invoke) > parameters (optional arguments for the application) 35 Friday, 06 August 2010 35
    • in case it wasn't clear (yeah, for rookies) • command line actions are > commands (the name of the application to invoke) > parameters (optional arguments for the application) > and the ENTER key 35 Friday, 06 August 2010 35
    • in case it wasn't clear (yeah, for rookies) • command line actions are > commands (the name of the application to invoke) > parameters (optional arguments for the application) > and the ENTER key Enter 35 Friday, 06 August 2010 35
    • The Unix philosophy (1) • One tool for every feature > make one tool that does one thing well > for many features, make many tools > make them work together (see next slide) 36 Friday, 06 August 2010 36
    • The Unix philosophy (2) • To make several tool cooperate: > pipe the output of one tool to the input of another one. > for more actions, pipe more tools $ sort < filename | uniq | nl 37 Friday, 06 August 2010 37
    • Unix pipes 38 Friday, 06 August 2010 38
    • Unix pipes application < input 38 Friday, 06 August 2010 38
    • Unix pipes application < input application > output 38 Friday, 06 August 2010 38
    • Unix pipes application < input application > output application < input > output 38 Friday, 06 August 2010 38
    • Unix pipes application < input application > output application < input > output application | application 38 Friday, 06 August 2010 38
    • Unix pipes application < input application > output application < input > output application | application output input 38 Friday, 06 August 2010 38
    • Unix pipes = parallel processing 39 Friday, 06 August 2010 39
    • Unix pipes = parallel processing application | application 39 Friday, 06 August 2010 39
    • Unix pipes = parallel processing application | application 39 Friday, 06 August 2010 39
    • Unix pipes = parallel processing application | application output 39 Friday, 06 August 2010 39
    • Unix pipes = parallel processing application | application output input 39 Friday, 06 August 2010 39
    • Unix pipes = parallel processing application | application output input 39 Friday, 06 August 2010 39
    • Unix pipes = parallel processing application | application output input 39 Friday, 06 August 2010 39
    • Unix pipes = parallel processing application | application output input 39 Friday, 06 August 2010 39
    • Who gets what • shell • application • database server • pager 40 Friday, 06 August 2010 40
    • who gets what shell prompt $ mysql mysql client mysql server 41 Friday, 06 August 2010 41
    • who gets what any host mysql server shell mysql client 42 Friday, 06 August 2010 42
    • who gets what $ mysql -u username -ppassword First the application then the server 43 Friday, 06 August 2010 43
    • who gets what $ mysql -u username -ppassword -e "select VERSION()" goes to the client, then the server goes to the server 44 Friday, 06 August 2010 44
    • who gets what mysql> help mysql> pager less goes to the client only mysql> select 1; goes to the server 45 Friday, 06 August 2010 45
    • the command line readline 46 Friday, 06 August 2010 46
    • readline • Common component > shell > mysql client > many CLI applications 47 Friday, 06 August 2010 47
    • Main cursor movements Ctrl A Start/End of line Ctrl E Meta B Back/Forward one word Meta F previous/next command Ctrl L clear screen and take current line on top 48 Friday, 06 August 2010 48
    • Changing text Meta U Uppercase word Meta L Lowercase word Ctrl D delete word Ctrl K delete to end of line Ctrl - undo for more tips: $ man readline 49 Friday, 06 August 2010 49
    • Some conventions in this talk $ command at the shell mysql> command at the mysql prompt 50 Friday, 06 August 2010 50
    • the Bash shell #! bash 51 Friday, 06 August 2010 51
    • components • commands • aliases • variables • functions 52 Friday, 06 August 2010 52
    • commands $ echo "something" something 53 Friday, 06 August 2010 53
    • aliases $ alias mydate='mysql -e "select NOW()"' $ mydate +---------------------+ | now() | +---------------------+ | 2010-04-12 07:14:19 | +---------------------+ 54 Friday, 06 August 2010 54
    • aliases $ alias myself='mysql' $ myself -e "select NOW()"' +---------------------+ | now() | +---------------------+ | 2010-04-12 07:14:19 | +---------------------+ 55 Friday, 06 August 2010 55
    • variables $ MYBATCH='mysql -N -B' $ MYQUERY='select curdate()' $ $MYBATCH -e "$MYQUERY" 2010-04-12 56 Friday, 06 August 2010 56
    • nesting variables # t.sh CMD="SELECT TABLE_NAME" TBL='information_schema.tables' WHC1="TABLE_SCHEMA='mysql'" WHC2="TABLE_NAME like 'u%' " Q1="$CMD FROM $TBL" Q2="WHERE $WHC1 AND $WHC2" Q="$Q1 $Q2" echo $Q echo $Q | mysql 57 Friday, 06 August 2010 57
    • nesting variables $ sh t.sh SELECT TABLE_NAME FROM information_schema.tables WHERE TABLE_SCHEMA='mysql' AND TABLE_NAME like 'u%' TABLE_NAME user 58 Friday, 06 August 2010 58
    • the mysql command line client mysql> 59 Friday, 06 August 2010 59
    • SQL commands from a file $ mysql < filename.sql mysql> source filename.sql • executes the commands contained in filename.sql • commands can be queries understood by the server • and commands for the client 60 Friday, 06 August 2010 60
    • SQL commands at the command line $ application | mysql $ echo "SELECT 1" | mysql $ mysql -e "SELECT 1" • executes the commands from standard input • commands can be queries understood by the server • and commands for the client 61 Friday, 06 August 2010 61
    • batch mode $ echo "select curdate()" | mysql curdate() 2010-04-12 62 Friday, 06 August 2010 62
    • table mode $ echo "select curdate()" | mysql -t +------------+ | curdate() | +------------+ | 2010-04-12 | +------------+ 63 Friday, 06 August 2010 63
    • table mode $ mysql -e "select curdate()" +------------+ | curdate() | +------------+ | 2010-04-12 | +------------+ 64 Friday, 06 August 2010 64
    • Forcing batch mode mysql -B -e 'select curdate()' curdate() 2010-04-12 65 Friday, 06 August 2010 65
    • Forcing batch mode - no header mysql -N -B -e 'select curdate()' 2010-04-12 ANT! PO RT IM 66 Friday, 06 August 2010 66
    • Shell + mysql $ echo "The current server's date is `mysql -N -B -e 'select curdate()'`." $ echo "The current server's date is $(mysql -N -B -e 'select curdate ()')." 67 Friday, 06 August 2010 67
    • advanced command line techniques (((((( (: @ @ - $_ loops 68 Friday, 06 August 2010 68
    • loops basics $ for N in 1 2 3 > do > echo $N > done 1 2 3 69 Friday, 06 August 2010 69
    • loops basics $ for N in 1 2 3 ; do echo $N ; done 1 2 3 70 Friday, 06 August 2010 70
    • loops basics $ for N in $(seq 1 3) ; do echo "this is number $N" ; done this is number 1 this is number 2 this is number 3 71 Friday, 06 August 2010 71
    • loops from file $ head /usr/share/dict/words A a aa aal aalii aam Aani aardvark aardwolf 72 Friday, 06 August 2010 72
    • loops from file $ for U in $(head /usr/share/dict/words ) ; do perl -le 'print uc shift' $U ; done A A AA AAL AALII AAM AANI 73 Friday, 06 August 2010 73
    • loops from mysql results $ mysql -B -N -e 'show tables from world' City Country CountryLanguage 74 Friday, 06 August 2010 74
    • loops from mysql results $ for T in $(mysql -B -N -e 'show tables from world') ; > do echo "CREATE TABLE world2.$T LIKE world.$T;" ; done CREATE TABLE world2.City LIKE world.City; CREATE TABLE world2.Country LIKE world.Country; CREATE TABLE world2.CountryLanguage LIKE world.CountryLanguage; 75 Friday, 06 August 2010 75
    • piping loops back into mysql $ for T in $(mysql -B -N -e 'show tables from world') ; > do echo "CREATE TABLE world2.$T LIKE world.$T;" ; done > | mysql -vv 76 Friday, 06 August 2010 76
    • advanced command line techniques nested :) @ @ - $_ #! #! #! #! loops #! #! 77 Friday, 06 August 2010 77
    • doing something for every table in every database $ mysql -e "show databases" +--------------------+ | Database | +--------------------+ | information_schema | | mysql | | sakila | | world | +--------------------+ 78 Friday, 06 August 2010 78
    • doing something for every table in every database $ mysql -e "show tables from world" +-----------------+ | Tables_in_world | +-----------------+ | City | | Country | | CountryLanguage | +-----------------+ 79 Friday, 06 August 2010 79
    • pseudocode for each database do for each table do print the table name 80 Friday, 06 August 2010 80
    • actual code alias mybatch='mysql -B -N' for D in $(mybatch -e "SHOW SCHEMAS") do for T in $(mybatch -e "SHOW TABLES FROM $D") do echo "$D.$T" done done 81 Friday, 06 August 2010 81
    • advanced command line techniques $HOME options files 82 Friday, 06 August 2010 82
    • Default option files /etc/my.cnf /etc/mysql/my.cnf (Debian, Ubuntu) $BASEDIR/my.cnf $HOME/.my.cnf 83 Friday, 06 August 2010 83
    • Custom option files $ mysql --no-defaults $ mysql --defaults-file=filename $ mysql --defaults-extra-file=filename 84 Friday, 06 August 2010 84
    • Option files sections [mysqld] # option for server [client] # option for any mysql client [mysql] # options specific to 'mysql' CLI app 85 Friday, 06 August 2010 85
    • Option files example [client] user=myname password=mypassword [mysql] prompt="mysql (h) {u} [d] > " 86 Friday, 06 August 2010 86
    • Option files: caution # protect your option files! $ chmod 600 $HOME/.my.cnf $ ls -l .my.cnf -rw------- 1 [...] 10:04 .my.cnf 87 Friday, 06 August 2010 87
    • prompt mysql> prompt gm (h) {u} [d] > PROMPT set to 'gm (h) {u} [d] > ' gm (localhost) {root} [(none)] > use test Database changed gm (localhost) {root} [test] > # h -> host # u -> user # d -> database 88 Friday, 06 August 2010 88
    • advanced command line techniques (((((( (: @ @ - #! aliases 89 Friday, 06 August 2010 89
    • An useful alias $ alias mysql_as_root= 'mysql -- defaults_file=~/.my_as_root.cnf' mysql_as_root -e "select user()" +----------------+ | user() | +----------------+ | root@localhost | +----------------+ 90 Friday, 06 August 2010 90
    • An useful alias #.my_as_root.cnf' [client] user=root password=your_mysql_root_password [mysql] prompt="### [h] {u} (d) > " 91 Friday, 06 August 2010 91
    • Another useful alias alias mybatch='mysql -B -N ' mybatch -e "select version()" 5.1.45-log 92 Friday, 06 August 2010 92
    • ( ) advanced command line techniques (((((( (: @ @ - #! functions 93 Friday, 06 August 2010 93
    • functions • like aliases, but with parameters 94 Friday, 06 August 2010 94
    • sample function function grant_new { mysql_as_root -e "GRANT $1 on $2 to $3 identified by "$4" " } $ grant_new SELECT "world.*" john john_secret 95 Friday, 06 August 2010 95
    • sample function $ mysql_as_root -e 'show grants for johnG' *** 1. row *** Grants for john@%: GRANT USAGE ON *.* TO 'john'@'%' IDENTIFIED BY PASSWORD '*73A88CE13374128844E3ABB500F439A9C02C 53A3' *** 2. row *** Grants for john@%: GRANT SELECT ON `world`.* TO 'john'@'%' 96 Friday, 06 August 2010 96
    • advanced command line techniques $_ getting query details 97 Friday, 06 August 2010 97
    • which query is being executed? # wrong.sql # this script has one error use test ; drop table if exists t1 ; create table t1 (id int); insert into t1 values (a); 98 Friday, 06 August 2010 98
    • which query is being executed? $ mysql < wrong.sql ERROR 1054 (42S22) at line 4: Unknown column 'a' in 'field list' # WHERE IS THE ERROR? 99 Friday, 06 August 2010 99
    • which query is being executed? $ mysql -v < wrong.sql -------------- drop table if exists t1 -------------- -------------- create table t1 (id int) -------------- -------------- insert into t1 values (a) -------------- ERROR 1054 (42S22) at line 4: Unknown column 'a' in 'field list' 100 Friday, 06 August 2010 100
    • getting execution times $ mysql -vv -e 'select rating,count(*) from sakila.film group by rating' +--------+----------+ | rating | count(*) | +--------+----------+ | G | 178 | | PG | 194 | | PG-13 | 223 | | R | 195 | | NC-17 | 210 | +--------+----------+ 5 rows in set (0.00 sec) 101 Friday, 06 August 2010 101
    • getting execution times (batch mode) $echo 'select rating,count(*) from sakila.film group by rating' | mysql -vv -------------- select rating,count(*) from sakila.film group by rating -------------- rating count(*) G 178 PG 194 PG-13 223 R 195 NC-17 210 5 rows in set 102 Friday, 06 August 2010 102
    • getting execution times (batch mode) $ echo 'select rating,count(*) from sakila.film group by rating' | mysql -vvv -------------- select rating,count(*) from sakila.film group by rating -------------- +--------+----------+ | rating | count(*) | +--------+----------+ | G | 178 | | PG | 194 | | PG-13 | 223 | | R | 195 | | NC-17 | 210 | +--------+----------+ 103 5 rows in set (0.00 sec) Friday, 06 August 2010 103
    • advanced command line techniques getting _$ distinct results 104 Friday, 06 August 2010 104
    • getting a single value $ mysql -e "show variables like 'port'" +---------------+-------+ | Variable_name | Value | +---------------+-------+ | port | 3306 | +---------------+-------+ # Here, we only want the port number 105 Friday, 06 August 2010 105
    • getting a single value $ mysql -NB -e "show variables like 'port'" port 3306 # no header, batch mode # still two values 106 Friday, 06 August 2010 106
    • getting a single value $ mysql -NB -e "show variables like 'port'" | awk '{print $2}' 3306 $ mysql -NB -e "show variables like 'port'" | perl -lane 'print $F[1]' 3306 # filtering the wanted value 107 Friday, 06 August 2010 107
    • getting a single value: is replication working? $ mysql -e "show slave status G" | grep -i slave_io_running Slave_IO_Running: Yes # too many things. # We only want a 'yes' or 'no' 108 Friday, 06 August 2010 108
    • getting a single value: is replication working? $ mysql -e "show slave status G" | grep -i "slave_io_running" | perl -lane 'print $F[1]' Yes # yes! 109 Friday, 06 August 2010 109
    • What's perl -lane? 110 Friday, 06 August 2010 110
    • What's perl -lane? perl -lane must turn perl 110 Friday, 06 August 2010 110
    • Perl command line must-know facts (1) perl -e 'print "hellon"' # executes perl code 111 Friday, 06 August 2010 111
    • Perl command line must-know facts (2) perl -ne 'print if /^a/' # starts a loop from standard input # and executes the perl code for each # iteration perl -MO=Deparse -ne 'print if /^a/' LINE: while (defined($_ = <ARGV>)) { print $_ if /^a/; } 112 Friday, 06 August 2010 112
    • Perl command line must-know facts (3) perl -pe 's/(^aw+)/$1/' # like -ne, but prints each line $ perl -MO=Deparse -pe 's/(^aw+)/$1/' LINE: while (defined($_ = <ARGV>)) { s/(^aw+)/$1/; } continue { print $_; } 113 Friday, 06 August 2010 113
    • Perl command line must-know facts (4) perl -le 'print "hello"' # strips newlines from input and adds # them to output $ perl -e 'print "hello"' hello$ $ perl -le 'print "hello"' hello $ 114 Friday, 06 August 2010 114
    • Perl command line must-know facts (5) perl -lane 'print $F[0]' # splits input into @F array $ echo "a b c" | perl -lane 'print $F[1]' b $ echo "a b c" | perl -lane 'print $F[2]' c 115 Friday, 06 August 2010 115
    • Perl command line must-know facts (6) Perl and shell contention for $ #!/bin/bash Take the $$$ and run 116 Friday, 06 August 2010 116
    • Beware of $VARIABLES on the command line $ perl -le "$x=1; print $x" syntax error at -e line 1, near "=" Execution of -e aborted due to compilation errors. 117 Friday, 06 August 2010 117
    • double quotes Beware of $VARIABLES on theconsumed by the allow the shell to shell, before Perl interpolate command line can see it variables $ perl -le "$x=1; print $x" syntax error at -e line 1, near "=" Execution of -e aborted due to compilation errors. 117 Friday, 06 August 2010 117
    • double quotes Beware of $VARIABLES on theconsumed by the allow the shell to shell, before Perl interpolate command line can see it variables $ perl -le "$x=1; print $x" syntax error at -e line 1, near "=" Execution of -e aborted due to compilation errors. consumed by the shell, before Perl can see it 117 Friday, 06 August 2010 117
    • double quotes Beware of $VARIABLES on the consumed by the allow the shell to shell, before Perl interpolate command line can see it variables $ perl -le "$x=1; print $x" syntax error at -e line 1, near "=" Execution of -e aborted due to compilation errors. consumed by the shell, before Perl $ perl -le '$x=1; print $x' can see it 1 117 Friday, 06 August 2010 117
    • double quotes Beware of $VARIABLES on the consumed by the allow the shell to shell, before Perl interpolate command line can see it variables $ perl -le "$x=1; print $x" syntax error at -e line 1, near "=" Execution of -e aborted due to compilation errors. consumed by the shell, before Perl $ perl -le '$x=1; print $x' can see it 1 single quotes the consumed by prevent variable shell, before Perl consumption by can shell it the see117 Friday, 06 August 2010 117
    • advanced command line techniques combining column names 118 Friday, 06 August 2010 118
    • combining column names $ mysql -N -e 'desc world.City' +-------------+----------+----+-----+ | ID | int(11) | NO | PRI | | Name | char(35) | NO | | | CountryCode | char(3) | NO | | | District | char(20) | NO | | | Population | int(11) | NO | | +-------------+----------+----+-----+ 119 Friday, 06 August 2010 119
    • combining column names # Expected result: # SELECT CONCAT_WS('|', # ID,Name,CountryCode,District, # Population) from City 120 Friday, 06 August 2010 120
    • combining column names (I) echo "select concat_ws('|', $(mysql -N -e 'desc world.City' | perl -lane 'push @f, $F[0];' -e 'END{print join ",", @f}')) from world.City" select concat_ws('|', ID,Name,CountryCode,District,Populatio n) from world.City 121 Friday, 06 August 2010 121
    • combining column names (II) echo "select concat_ws('|', $( mysql -NB -e 'select group_concat (column_name) from information_schema.columns where table_name="City" and table_schema="world"')) from world.City" select concat_ws('|', ID,Name,CountryCode,District,Populatio n) from world.City 122 Friday, 06 August 2010 122
    • combining column names (IIa) echo "select concat_ws('|', $( mysql -NB -e 'select group_concat (column_name) from information_schema.columns where table_name="Country" and table_schema="world"')) from world.Country" select concat_ws('|', Code,Name,Continent,Region,SurfaceArea,Inde pYear,Population,LifeExpectancy,GNP,GNPOld, LocalName,GovernmentForm,HeadOfState,Capita l,Code2) from world.Country 123 Friday, 06 August 2010 123
    • piping the result back to mysql echo "select concat_ws('|', $( mysql -NB -e 'select group_concat (column_name) from information_schema.columns where table_name="Country" and table_schema="world"')) as line from world.Country" | mysql | head -n 4 line AFG|Afghanistan|Asia|Southern and Central Asia|652090.00|1919| 22720000|45.9|5976.00|Afganistan/Afqanestan|Islamic Emirate| Mohammad Omar|1|AF NLD|Netherlands|Europe|Western Europe|41526.00|1581|15864000| 78.3|371362.00|360478.00|Nederland|Constitutional Monarchy| Beatrix|5|NL ANT|Netherlands Antilles|North America|Caribbean|800.00|217000| 74.7|1941.00|Nederlandse Antillen|Nonmetropolitan Territory of 124 The Netherlands|Beatrix|33|AN Friday, 06 August 2010 124
    • advanced command line techniques $_ dumping data 125 Friday, 06 August 2010 125
    • advanced command line techniques $_ dumping data 125 Friday, 06 August 2010 125
    • dumping a tab delimited data set # the naive way $ mysql -e "SELECT * into outfile '/tmp/city_data' from world.City" # it works only if: # - /tmp/city_data does not exist # - /tmp is world writable 126 Friday, 06 August 2010 126
    • dumping a tab delimited data set $ mysql -B -N -e "SELECT * from world.City" > /tmp/city_data # it works always # if you omit the "-N" it also # includes a header 127 Friday, 06 August 2010 127
    • dumping a CSV $ mysql -B -e "SELECT * from world.City" | perl -F"t" -lane 'print join ",", map {s/"/""/g; /^[d.]+$/ ? $_ : qq("$_")} @F ' > /tmp/city_data 128 Friday, 06 August 2010 128
    • dumping a CSV $ mysql -B -e "SELECT * from world.City" | perl -F"t" -lane 'print join ",", splits the line by tabs, instead map {s/"/""/g; of the default whitespace /^[d.]+$/ ? $_ : qq("$_")} @F ' > /tmp/city_data 128 Friday, 06 August 2010 128
    • advanced command line techniques script it 129 Friday, 06 August 2010 129
    • use the mysql.sh script  The companion code for this tutorial  http://tinyurl.com/cli-tut-code-2010  It will save you much typing  But use it at your own risk! 130 Friday, 06 August 2010 130
    • Usage • download the script • load it $ source mysql.sh 131 Friday, 06 August 2010 131
    • mysql.sh functions • my_batch "query" • returns the result of the query. • Without header • Without box $ my_batch "SELECT * FROM test.t1" 1 aaa 2 bbb 3 ccc 132 Friday, 06 August 2010 132
    • mysql.sh functions • my_value "what" • returns a single value • e.g. $ my_value "version()" 5.1.49 $ my_value "@@server_id" 1 133 Friday, 06 August 2010 133
    • mysql.sh functions • my_var "what" • returns the value of a given variable • e.g. $ my_var port 3306 $ my_var datadir /usr/local/mysql/data 134 Friday, 06 August 2010 134
    • mysql.sh functions • my_status "what" • returns the value of a given status item • e.g. $ my_status com_select 2653 $ my_status connections 452 135 Friday, 06 August 2010 135
    • mysql.sh functions • my_dump_dataset "query" [header] • returns the values of a given query separated by tabs • if a header is requested, it will be part of the result • e.g. $ my_dump_dataset "select * from world.City" | head -n 5 1 Kabul AFG Kabol 1780000 2 Qandahar AFG Qandahar 237500 3 Herat AFG Herat 186800 4 Mazar-e-Sharif AFG Balkh 127800 136 5 Amsterdam NLD Noord-Holland 731200 Friday, 06 August 2010 136
    • mysql.sh functions • my_csv "query" [separator] • returns the values of a given query separated by commas, or by a given separator • e.g. $ my_csv 'select * from world.City' | head -n 3 "ID","Name","CountryCode","District"," Population" 1,"Kabul","AFG","Kabol",1780000 2,"Qandahar","AFG","Qandahar",237500 137 Friday, 06 August 2010 137
    • inside the command line client handling :-) @ @ $_ output 138 Friday, 06 August 2010 138
    • pager mysql> SELECT * FROM world.City; rec … rec … rec … 4000 + records unreadable 139 Friday, 06 August 2010 139
    • pager mysql> pager less PAGER set to 'less' mysql> SELECT * FROM world.City; # [ records in a manageable window ] mysql> nopager PAGER set to stdout # back to normal 140 Friday, 06 August 2010 140
    • More pager tricks mysql> pager vim - # like 'less', but you can also edit # the results and eventually save # them 141 Friday, 06 August 2010 141
    • More pager tricks mysql> pager md5sum PAGER set to 'md5sum' mysql> select * from City; 5d17ffa50d6da200dee621823ade2543 - 4079 rows in set (0.01 sec) # produce a CRC for the result set # instead of displaying it. # Useful for data comparison. # (Thanks to Baron Schwartz 142 Friday, 06 August 2010 142
    • Comparing data sets with md5sum mysql> select * from City; 5d17ffa50d6da200dee621823ade2543 - mysql> select * from City order by ID; 5d17ffa50d6da200dee621823ade2543 - mysql> select * from City order by name; 5ab980f5b69a6e65d3b8cd5a95ec94e0 - mysql> select * from City order by District; 2bf37cbe91b3ffeaebc8c4b6084b8784 - 143 Friday, 06 August 2010 143
    • Comparing data sets with md5sum mysql> select * from City; 5d17ffa50d6da200dee621823ade2543 - mysql> select * from City order by ID; 5d17ffa50d6da200dee621823ade2543 - mysql> select * from City order by name; 5ab980f5b69a6e65d3b8cd5a95ec94e0 - mysql> select * from City order by District; 2bf37cbe91b3ffeaebc8c4b6084b8784 - 143 Friday, 06 August 2010 143
    • More pager tricks mysql> pager cat > /tmp/t1.txt mysql> select "one" union select "two"; mysql> pager cat > /tmp/t2.txt mysql> select "one" union select "TWO"; mysql> nopager mysql> ! vimdiff -o /tmp/t[12].txt # see the difference between two # datasets 144 Friday, 06 August 2010 144
    • 145 Friday, 06 August 2010 145
    • logging sessions mysql> tee mylog.txt # logs everything that you see on # the screen. # your commands, the results, # warnings, error messages, and so on. mysql> notee # back to normal 146 Friday, 06 August 2010 146
    • Updated presentation slides  you will find an up to date copy of these slides at:  http://tinyurl.com/shoot-cl-2010  http://tinyurl.com/shoot-cl-full-2010 Friday, 06 August 2010 147
    • read more  http://datacharmer.blogspot.com Friday, 06 August 2010 148
    • THANKS! ? Question time Comments on Twitter: @datacharmer This work is licensed under the Creative Commons Attribution-Share Alike 3.0 Unported License. Friday, 06 August 2010 149