Shooting from the hip - Perl at the command line for DBAs

Shooting from the hip. Perl at the command :) @ @ line for DBAs - $_ #! Giuseppe Maxia MySQL Community Team Lead @datacharmer This work is licensed under the Creative Commons Attribution-Share Alike 3.0 Unported License. Friday, 06 August 2010 1

about me -Giuseppe Maxia  a.k.a. The Data Charmer  MySQL Community Team Lead  Long time hacking with MySQL features  Formerly, database consultant, designer, coder.  A passion for QA  An even greater passion for open source  ... and community  Passionate blogger  http://datacharmer.blogspot.com Friday, 06 August 2010 2

Updated presentation slides  you will find an up to date copy of these slides at:  http://tinyurl.com/shoot-cl-2010  http://tinyurl.com/shoot-cl-full-2010 Friday, 06 August 2010 3

I love your feedback!  Tell it to the Twitter community  #yapc_eu "your feedback here"  Twitter me  @datacharmer "your feedback here"  Blog about it Friday, 06 August 2010 4

This is a story about languages Friday, 06 August 2010 5

you got to understand, Bernie, it's not personal. It's just that you are obsolete We use language to talk to computers Friday, 06 August 2010 6

10 print "hello" We use 20 goto 10 languages to talk to computers Friday, 06 August 2010 7

I am a polyglot Friday, 06 August 2010 8

An imperfect polyglot, actually Friday, 06 August 2010 9

I speak Italian Friday, 06 August 2010 10

I speak Italian with a clear Sardinian accent Friday, 06 August 2010 10

I speak English Friday, 06 August 2010 11

I speak English with a distinct Italian accent Friday, 06 August 2010 11

I speak Perl Friday, 06 August 2010 12

I speak Perl with a hint of a C accent Friday, 06 August 2010 12

I speak French and Spanish Friday, 06 August 2010 13

I speak French and Spanish but since I use them very little, they have melted with Italian and English Friday, 06 August 2010 13

I speak #!/bin/bash Bash Friday, 06 August 2010 14

I speak #!/bin/bash Bash with a strong Perl accent Friday, 06 August 2010 14

I speak fluent Italian La donzelletta vien dalla campagna, in sul calar del sole, col suo fascio dell'erba e reca in mano un mazzolino di rose e viole, onde, siccome suole, ornare ella si appresta ... Friday, 06 August 2010 15

However, Sardinian comes handy sometimes Maissavida ti frimisti! (*) (*) May you never stop Friday, 06 August 2010 16

I speak fluent English When in the Course of human events, it becomes necessary for one people to dissolve the political bands which have connected them with another, and to assume among the powers of the earth, the separate and equal station to which the Laws of Nature and of Nature's God entitle them, a decent respect to the opinions of mankind requires that they should declare the causes which impel them to the separation. Friday, 06 August 2010 17

I speak fluent English When in the Course of human events, it becomes necessary for one people to dissolve the political bands which have connected them with another, and to assume among the powers of the earth, the separate and equal station to which the Laws of Nature and of Nature's God entitle them, a decent respect to the opinions of mankind requires that they should declare the causes which impel them to the separation. hmm. wrong ﬂag Friday, 06 August 2010 17

I speak fluent English When in the Course of human events, it becomes necessary for one people to dissolve the political bands which have connected them with another, and to assume among the powers of the earth, the separate and equal station to which the Laws of Nature and of Nature's God entitle them, a decent respect to the opinions of mankind requires that they should declare the causes which impel them to the separation. Friday, 06 August 2010 17

But sometimes Italian is still necessary barista al fresco pasta maestro espresso tempo vendetta soprano ﬁnale piano dilettante cappuccino stucco stiletto virtuoso ﬁasco presto lotto pizza bravo lasagna studio spaghetti stanza Friday, 06 August 2010 18

And sometimes Italian is needed I tuoi perversi defunti!(*) (*) Your wicked ancestors! Friday, 06 August 2010 19

Although Sardinian is still effective here Ancu ti currullidi sa giustizia (*) (*) May the rule of law chase you Friday, 06 August 2010 20

I speak Perl fluently #!/usr/bin/perl use strict; use warnings; use File::Find; while (my $line = <stdin>){ # … Friday, 06 August 2010 21

Sometimes, I need help from C use Inline C; print "9 + 16 = ", add(9, 16), "n"; __END__ __C__ int add(int x, int y) { return x + y; } Friday, 06 August 2010 22

Most of my work at the command line is Bash $ for D in $HOME/*/; do du -sh $D; > done > /tmp/occupancy #!/bin/bash Friday, 06 August 2010 23

And sometimes, a pinch of Perl can help $ perl -pe 's{(dd)-(dd)-(dddd)}{$3-$1-$2}g' < rawfile.txt > betterfile.txt #!/bin/bash Friday, 06 August 2010 24

And then on top of all this, I need to speak SQL $ mysql -e 'SELECT COUNT(*) FROM dbname.tbl_name WHERE col_name = 1 and other_col="A" ' Friday, 06 August 2010 25

I wish I could say that my SQL has a Perlish accent $ mysql -e 'SELECT COUNT(*) FROM dbname.tbl_name WHERE col_name = 1 and other_col="A" ' use strict; use DBI;? Friday, 06 August 2010 26

I wish I could say that my SQL has a Perlish accent $ mysql -e 'SELECT COUNT(*) FROM dbname.tbl_name WHERE col_name = 1 and other_col="A" ' but it wouldn't be true Friday, 06 August 2010 27

I could curse in Perl $ perl -pe 's{(dd)-(dd)-(ddd d)}{$3-$1-$2}g' < rawfile.txt > betterfile.txt (*) But it would not get me far. (*) already translated Friday, 06 August 2010 28

Then Friday, 06 August 2010 29

Then I will need to mix languages. Friday, 06 August 2010 29

Then I will need to mix languages. This talk will show exactly that. Friday, 06 August 2010 29

Then I will need to mix languages. This talk will show exactly that. Mixing languages. Friday, 06 August 2010 29

Then I will need to mix languages. This talk will show exactly that. Mixing languages. Yes. Friday, 06 August 2010 29

Then I will need to mix languages. This talk will show exactly that. Mixing languages. Yes. $ at the command line Friday, 06 August 2010 29

What this tutorial is about  Providing tools and tips for command line users;  Learn a few tricks to make you achieve more and type less. Friday, 06 August 2010 30

What this tutorial is about  Providing tools and tips for command line users;  Learn a few tricks to make you achieve more and type less.  Making GUI enthusiasts green with envy; bar foo baz Friday, 06 August 2010 30

the command line (for rookies) BASICS 31 Friday, 06 August 2010 31

The shell • It's a command interpreter • There are many flavors > bash > zsh > ksh 32 Friday, 06 August 2010 32

MySQL command line client • called "mysql" > Takes SQL commands > sends them to the server > gets results • but it can do much more ... 33 Friday, 06 August 2010 33

other tools • Shell scripts • Aliases • Perl • awk • sort • …. and a lot of Unix little helper 34 Friday, 06 August 2010 34

other tools • Shell scripts • Aliases • Perl which could replace everything else, of course • awk • sort • …. and a lot of Unix little helper 34 Friday, 06 August 2010 34

in case it wasn't clear (yeah, for rookies) 35 Friday, 06 August 2010 35

in case it wasn't clear (yeah, for rookies) • command line actions are 35 Friday, 06 August 2010 35

in case it wasn't clear (yeah, for rookies) • command line actions are > commands (the name of the application to invoke) 35 Friday, 06 August 2010 35

in case it wasn't clear (yeah, for rookies) • command line actions are > commands (the name of the application to invoke) > parameters (optional arguments for the application) 35 Friday, 06 August 2010 35

in case it wasn't clear (yeah, for rookies) • command line actions are > commands (the name of the application to invoke) > parameters (optional arguments for the application) > and the ENTER key 35 Friday, 06 August 2010 35

in case it wasn't clear (yeah, for rookies) • command line actions are > commands (the name of the application to invoke) > parameters (optional arguments for the application) > and the ENTER key Enter 35 Friday, 06 August 2010 35

The Unix philosophy (1) • One tool for every feature > make one tool that does one thing well > for many features, make many tools > make them work together (see next slide) 36 Friday, 06 August 2010 36

The Unix philosophy (2) • To make several tool cooperate: > pipe the output of one tool to the input of another one. > for more actions, pipe more tools $ sort < filename | uniq | nl 37 Friday, 06 August 2010 37

Unix pipes 38 Friday, 06 August 2010 38

Unix pipes application < input 38 Friday, 06 August 2010 38

Unix pipes application < input application > output 38 Friday, 06 August 2010 38

Unix pipes application < input application > output application < input > output 38 Friday, 06 August 2010 38

Unix pipes application < input application > output application < input > output application | application 38 Friday, 06 August 2010 38

Unix pipes application < input application > output application < input > output application | application output input 38 Friday, 06 August 2010 38

Unix pipes = parallel processing 39 Friday, 06 August 2010 39

Unix pipes = parallel processing application | application 39 Friday, 06 August 2010 39

Unix pipes = parallel processing application | application output 39 Friday, 06 August 2010 39

Unix pipes = parallel processing application | application output input 39 Friday, 06 August 2010 39

Who gets what • shell • application • database server • pager 40 Friday, 06 August 2010 40

who gets what shell prompt $ mysql mysql client mysql server 41 Friday, 06 August 2010 41

who gets what any host mysql server shell mysql client 42 Friday, 06 August 2010 42

who gets what $ mysql -u username -ppassword First the application then the server 43 Friday, 06 August 2010 43

who gets what $ mysql -u username -ppassword -e "select VERSION()" goes to the client, then the server goes to the server 44 Friday, 06 August 2010 44

who gets what mysql> help mysql> pager less goes to the client only mysql> select 1; goes to the server 45 Friday, 06 August 2010 45

the command line readline 46 Friday, 06 August 2010 46

readline • Common component > shell > mysql client > many CLI applications 47 Friday, 06 August 2010 47

Main cursor movements Ctrl A Start/End of line Ctrl E Meta B Back/Forward one word Meta F previous/next command Ctrl L clear screen and take current line on top 48 Friday, 06 August 2010 48

Changing text Meta U Uppercase word Meta L Lowercase word Ctrl D delete word Ctrl K delete to end of line Ctrl - undo for more tips: $ man readline 49 Friday, 06 August 2010 49

Some conventions in this talk $ command at the shell mysql> command at the mysql prompt 50 Friday, 06 August 2010 50

the Bash shell #! bash 51 Friday, 06 August 2010 51

components • commands • aliases • variables • functions 52 Friday, 06 August 2010 52

commands $ echo "something" something 53 Friday, 06 August 2010 53

aliases $ alias mydate='mysql -e "select NOW()"' $ mydate +---------------------+ | now() | +---------------------+ | 2010-04-12 07:14:19 | +---------------------+ 54 Friday, 06 August 2010 54

aliases $ alias myself='mysql' $ myself -e "select NOW()"' +---------------------+ | now() | +---------------------+ | 2010-04-12 07:14:19 | +---------------------+ 55 Friday, 06 August 2010 55

variables $ MYBATCH='mysql -N -B' $ MYQUERY='select curdate()' $ $MYBATCH -e "$MYQUERY" 2010-04-12 56 Friday, 06 August 2010 56

nesting variables # t.sh CMD="SELECT TABLE_NAME" TBL='information_schema.tables' WHC1="TABLE_SCHEMA='mysql'" WHC2="TABLE_NAME like 'u%' " Q1="$CMD FROM $TBL" Q2="WHERE $WHC1 AND $WHC2" Q="$Q1 $Q2" echo $Q echo $Q | mysql 57 Friday, 06 August 2010 57

nesting variables $ sh t.sh SELECT TABLE_NAME FROM information_schema.tables WHERE TABLE_SCHEMA='mysql' AND TABLE_NAME like 'u%' TABLE_NAME user 58 Friday, 06 August 2010 58

the mysql command line client mysql> 59 Friday, 06 August 2010 59

SQL commands from a file $ mysql < filename.sql mysql> source filename.sql • executes the commands contained in filename.sql • commands can be queries understood by the server • and commands for the client 60 Friday, 06 August 2010 60

SQL commands at the command line $ application | mysql $ echo "SELECT 1" | mysql $ mysql -e "SELECT 1" • executes the commands from standard input • commands can be queries understood by the server • and commands for the client 61 Friday, 06 August 2010 61

batch mode $ echo "select curdate()" | mysql curdate() 2010-04-12 62 Friday, 06 August 2010 62

table mode $ mysql -e "select curdate()" +------------+ | curdate() | +------------+ | 2010-04-12 | +------------+ 64 Friday, 06 August 2010 64

Forcing batch mode mysql -B -e 'select curdate()' curdate() 2010-04-12 65 Friday, 06 August 2010 65

Forcing batch mode - no header mysql -N -B -e 'select curdate()' 2010-04-12 ANT! PO RT IM 66 Friday, 06 August 2010 66

Shell + mysql $ echo "The current server's date is `mysql -N -B -e 'select curdate()'`." $ echo "The current server's date is $(mysql -N -B -e 'select curdate ()')." 67 Friday, 06 August 2010 67

advanced command line techniques (((((( (: @ @ - $_ loops 68 Friday, 06 August 2010 68

loops basics $ for N in 1 2 3 > do > echo $N > done 1 2 3 69 Friday, 06 August 2010 69

loops basics $ for N in 1 2 3 ; do echo $N ; done 1 2 3 70 Friday, 06 August 2010 70

loops basics $ for N in $(seq 1 3) ; do echo "this is number $N" ; done this is number 1 this is number 2 this is number 3 71 Friday, 06 August 2010 71

loops from file $ head /usr/share/dict/words A a aa aal aalii aam Aani aardvark aardwolf 72 Friday, 06 August 2010 72

loops from file $ for U in $(head /usr/share/dict/words ) ; do perl -le 'print uc shift' $U ; done A A AA AAL AALII AAM AANI 73 Friday, 06 August 2010 73

loops from mysql results $ mysql -B -N -e 'show tables from world' City Country CountryLanguage 74 Friday, 06 August 2010 74

loops from mysql results $ for T in $(mysql -B -N -e 'show tables from world') ; > do echo "CREATE TABLE world2.$T LIKE world.$T;" ; done CREATE TABLE world2.City LIKE world.City; CREATE TABLE world2.Country LIKE world.Country; CREATE TABLE world2.CountryLanguage LIKE world.CountryLanguage; 75 Friday, 06 August 2010 75

piping loops back into mysql $ for T in $(mysql -B -N -e 'show tables from world') ; > do echo "CREATE TABLE world2.$T LIKE world.$T;" ; done > | mysql -vv 76 Friday, 06 August 2010 76

advanced command line techniques nested :) @ @ - $_ #! #! #! #! loops #! #! 77 Friday, 06 August 2010 77

pseudocode for each database do for each table do print the table name 80 Friday, 06 August 2010 80

actual code alias mybatch='mysql -B -N' for D in $(mybatch -e "SHOW SCHEMAS") do for T in $(mybatch -e "SHOW TABLES FROM $D") do echo "$D.$T" done done 81 Friday, 06 August 2010 81

advanced command line techniques $HOME options files 82 Friday, 06 August 2010 82

Default option files /etc/my.cnf /etc/mysql/my.cnf (Debian, Ubuntu) $BASEDIR/my.cnf $HOME/.my.cnf 83 Friday, 06 August 2010 83

Custom option files $ mysql --no-defaults $ mysql --defaults-file=filename $ mysql --defaults-extra-file=filename 84 Friday, 06 August 2010 84

Option files sections [mysqld] # option for server [client] # option for any mysql client [mysql] # options specific to 'mysql' CLI app 85 Friday, 06 August 2010 85

Option files example [client] user=myname password=mypassword [mysql] prompt="mysql (h) {u} [d] > " 86 Friday, 06 August 2010 86

Option files: caution # protect your option files! $ chmod 600 $HOME/.my.cnf $ ls -l .my.cnf -rw------- 1 [...] 10:04 .my.cnf 87 Friday, 06 August 2010 87

prompt mysql> prompt gm (h) {u} [d] > PROMPT set to 'gm (h) {u} [d] > ' gm (localhost) {root} [(none)] > use test Database changed gm (localhost) {root} [test] > # h -> host # u -> user # d -> database 88 Friday, 06 August 2010 88

advanced command line techniques (((((( (: @ @ - #! aliases 89 Friday, 06 August 2010 89

An useful alias $ alias mysql_as_root= 'mysql -- defaults_file=~/.my_as_root.cnf' mysql_as_root -e "select user()" +----------------+ | user() | +----------------+ | root@localhost | +----------------+ 90 Friday, 06 August 2010 90

An useful alias #.my_as_root.cnf' [client] user=root password=your_mysql_root_password [mysql] prompt="### [h] {u} (d) > " 91 Friday, 06 August 2010 91

Another useful alias alias mybatch='mysql -B -N ' mybatch -e "select version()" 5.1.45-log 92 Friday, 06 August 2010 92

( ) advanced command line techniques (((((( (: @ @ - #! functions 93 Friday, 06 August 2010 93

functions • like aliases, but with parameters 94 Friday, 06 August 2010 94

sample function function grant_new { mysql_as_root -e "GRANT $1 on $2 to $3 identified by "$4" " } $ grant_new SELECT "world.*" john john_secret 95 Friday, 06 August 2010 95

sample function $ mysql_as_root -e 'show grants for johnG' *** 1. row *** Grants for john@%: GRANT USAGE ON *.* TO 'john'@'%' IDENTIFIED BY PASSWORD '*73A88CE13374128844E3ABB500F439A9C02C 53A3' *** 2. row *** Grants for john@%: GRANT SELECT ON `world`.* TO 'john'@'%' 96 Friday, 06 August 2010 96

advanced command line techniques $_ getting query details 97 Friday, 06 August 2010 97

which query is being executed? # wrong.sql # this script has one error use test ; drop table if exists t1 ; create table t1 (id int); insert into t1 values (a); 98 Friday, 06 August 2010 98

which query is being executed? $ mysql < wrong.sql ERROR 1054 (42S22) at line 4: Unknown column 'a' in 'field list' # WHERE IS THE ERROR? 99 Friday, 06 August 2010 99

which query is being executed? $ mysql -v < wrong.sql -------------- drop table if exists t1 -------------- -------------- create table t1 (id int) -------------- -------------- insert into t1 values (a) -------------- ERROR 1054 (42S22) at line 4: Unknown column 'a' in 'field list' 100 Friday, 06 August 2010 100

getting execution times $ mysql -vv -e 'select rating,count(*) from sakila.film group by rating' +--------+----------+ | rating | count(*) | +--------+----------+ | G | 178 | | PG | 194 | | PG-13 | 223 | | R | 195 | | NC-17 | 210 | +--------+----------+ 5 rows in set (0.00 sec) 101 Friday, 06 August 2010 101

getting execution times (batch mode) $echo 'select rating,count(*) from sakila.film group by rating' | mysql -vv -------------- select rating,count(*) from sakila.film group by rating -------------- rating count(*) G 178 PG 194 PG-13 223 R 195 NC-17 210 5 rows in set 102 Friday, 06 August 2010 102

getting execution times (batch mode) $ echo 'select rating,count(*) from sakila.film group by rating' | mysql -vvv -------------- select rating,count(*) from sakila.film group by rating -------------- +--------+----------+ | rating | count(*) | +--------+----------+ | G | 178 | | PG | 194 | | PG-13 | 223 | | R | 195 | | NC-17 | 210 | +--------+----------+ 103 5 rows in set (0.00 sec) Friday, 06 August 2010 103

advanced command line techniques getting _$ distinct results 104 Friday, 06 August 2010 104

getting a single value $ mysql -NB -e "show variables like 'port'" port 3306 # no header, batch mode # still two values 106 Friday, 06 August 2010 106

getting a single value $ mysql -NB -e "show variables like 'port'" | awk '{print $2}' 3306 $ mysql -NB -e "show variables like 'port'" | perl -lane 'print $F[1]' 3306 # filtering the wanted value 107 Friday, 06 August 2010 107

getting a single value: is replication working? $ mysql -e "show slave status G" | grep -i slave_io_running Slave_IO_Running: Yes # too many things. # We only want a 'yes' or 'no' 108 Friday, 06 August 2010 108

getting a single value: is replication working? $ mysql -e "show slave status G" | grep -i "slave_io_running" | perl -lane 'print $F[1]' Yes # yes! 109 Friday, 06 August 2010 109

What's perl -lane? 110 Friday, 06 August 2010 110

What's perl -lane? perl -lane must turn perl 110 Friday, 06 August 2010 110

Perl command line must-know facts (1) perl -e 'print "hellon"' # executes perl code 111 Friday, 06 August 2010 111

Perl command line must-know facts (2) perl -ne 'print if /â/' # starts a loop from standard input # and executes the perl code for each # iteration perl -MO=Deparse -ne 'print if /â/' LINE: while (defined($_ = <ARGV>)) { print $_ if /â/; } 112 Friday, 06 August 2010 112

Perl command line must-know facts (3) perl -pe 's/(âw+)/$1/' # like -ne, but prints each line $ perl -MO=Deparse -pe 's/(âw+)/$1/' LINE: while (defined($_ = <ARGV>)) { s/(âw+)/$1/; } continue { print $_; } 113 Friday, 06 August 2010 113

Perl command line must-know facts (4) perl -le 'print "hello"' # strips newlines from input and adds # them to output $ perl -e 'print "hello"' hello$ $ perl -le 'print "hello"' hello $ 114 Friday, 06 August 2010 114

Perl command line must-know facts (5) perl -lane 'print $F[0]' # splits input into @F array $ echo "a b c" | perl -lane 'print $F[1]' b $ echo "a b c" | perl -lane 'print $F[2]' c 115 Friday, 06 August 2010 115

Perl command line must-know facts (6) Perl and shell contention for $ #!/bin/bash Take the $$$ and run 116 Friday, 06 August 2010 116

Beware of $VARIABLES on the command line $ perl -le "$x=1; print $x" syntax error at -e line 1, near "=" Execution of -e aborted due to compilation errors. 117 Friday, 06 August 2010 117

double quotes Beware of $VARIABLES on theconsumed by the allow the shell to shell, before Perl interpolate command line can see it variables $ perl -le "$x=1; print $x" syntax error at -e line 1, near "=" Execution of -e aborted due to compilation errors. 117 Friday, 06 August 2010 117

double quotes Beware of $VARIABLES on theconsumed by the allow the shell to shell, before Perl interpolate command line can see it variables $ perl -le "$x=1; print $x" syntax error at -e line 1, near "=" Execution of -e aborted due to compilation errors. consumed by the shell, before Perl can see it 117 Friday, 06 August 2010 117

double quotes Beware of $VARIABLES on the consumed by the allow the shell to shell, before Perl interpolate command line can see it variables $ perl -le "$x=1; print $x" syntax error at -e line 1, near "=" Execution of -e aborted due to compilation errors. consumed by the shell, before Perl $ perl -le '$x=1; print $x' can see it 1 117 Friday, 06 August 2010 117

double quotes Beware of $VARIABLES on the consumed by the allow the shell to shell, before Perl interpolate command line can see it variables $ perl -le "$x=1; print $x" syntax error at -e line 1, near "=" Execution of -e aborted due to compilation errors. consumed by the shell, before Perl $ perl -le '$x=1; print $x' can see it 1 single quotes the consumed by prevent variable shell, before Perl consumption by can shell it the see117 Friday, 06 August 2010 117

advanced command line techniques combining column names 118 Friday, 06 August 2010 118

combining column names $ mysql -N -e 'desc world.City' +-------------+----------+----+-----+ | ID | int(11) | NO | PRI | | Name | char(35) | NO | | | CountryCode | char(3) | NO | | | District | char(20) | NO | | | Population | int(11) | NO | | +-------------+----------+----+-----+ 119 Friday, 06 August 2010 119

combining column names # Expected result: # SELECT CONCAT_WS('|', # ID,Name,CountryCode,District, # Population) from City 120 Friday, 06 August 2010 120

combining column names (I) echo "select concat_ws('|', $(mysql -N -e 'desc world.City' | perl -lane 'push @f, $F[0];' -e 'END{print join ",", @f}')) from world.City" select concat_ws('|', ID,Name,CountryCode,District,Populatio n) from world.City 121 Friday, 06 August 2010 121

combining column names (II) echo "select concat_ws('|', $( mysql -NB -e 'select group_concat (column_name) from information_schema.columns where table_name="City" and table_schema="world"')) from world.City" select concat_ws('|', ID,Name,CountryCode,District,Populatio n) from world.City 122 Friday, 06 August 2010 122

combining column names (IIa) echo "select concat_ws('|', $( mysql -NB -e 'select group_concat (column_name) from information_schema.columns where table_name="Country" and table_schema="world"')) from world.Country" select concat_ws('|', Code,Name,Continent,Region,SurfaceArea,Inde pYear,Population,LifeExpectancy,GNP,GNPOld, LocalName,GovernmentForm,HeadOfState,Capita l,Code2) from world.Country 123 Friday, 06 August 2010 123

piping the result back to mysql echo "select concat_ws('|', $( mysql -NB -e 'select group_concat (column_name) from information_schema.columns where table_name="Country" and table_schema="world"')) as line from world.Country" | mysql | head -n 4 line AFG|Afghanistan|Asia|Southern and Central Asia|652090.00|1919| 22720000|45.9|5976.00|Afganistan/Afqanestan|Islamic Emirate| Mohammad Omar|1|AF NLD|Netherlands|Europe|Western Europe|41526.00|1581|15864000| 78.3|371362.00|360478.00|Nederland|Constitutional Monarchy| Beatrix|5|NL ANT|Netherlands Antilles|North America|Caribbean|800.00|217000| 74.7|1941.00|Nederlandse Antillen|Nonmetropolitan Territory of 124 The Netherlands|Beatrix|33|AN Friday, 06 August 2010 124

advanced command line techniques $_ dumping data 125 Friday, 06 August 2010 125

dumping a tab delimited data set # the naive way $ mysql -e "SELECT * into outfile '/tmp/city_data' from world.City" # it works only if: # - /tmp/city_data does not exist # - /tmp is world writable 126 Friday, 06 August 2010 126

dumping a tab delimited data set $ mysql -B -N -e "SELECT * from world.City" > /tmp/city_data # it works always # if you omit the "-N" it also # includes a header 127 Friday, 06 August 2010 127

dumping a CSV $ mysql -B -e "SELECT * from world.City" | perl -F"t" -lane 'print join ",", map {s/"/""/g; /^[d.]+$/ ? $_ : qq("$_")} @F ' > /tmp/city_data 128 Friday, 06 August 2010 128

dumping a CSV $ mysql -B -e "SELECT * from world.City" | perl -F"t" -lane 'print join ",", splits the line by tabs, instead map {s/"/""/g; of the default whitespace /^[d.]+$/ ? $_ : qq("$_")} @F ' > /tmp/city_data 128 Friday, 06 August 2010 128

advanced command line techniques script it 129 Friday, 06 August 2010 129

use the mysql.sh script  The companion code for this tutorial  http://tinyurl.com/cli-tut-code-2010  It will save you much typing  But use it at your own risk! 130 Friday, 06 August 2010 130

Usage • download the script • load it $ source mysql.sh 131 Friday, 06 August 2010 131

mysql.sh functions • my_batch "query" • returns the result of the query. • Without header • Without box $ my_batch "SELECT * FROM test.t1" 1 aaa 2 bbb 3 ccc 132 Friday, 06 August 2010 132

mysql.sh functions • my_value "what" • returns a single value • e.g. $ my_value "version()" 5.1.49 $ my_value "@@server_id" 1 133 Friday, 06 August 2010 133

mysql.sh functions • my_var "what" • returns the value of a given variable • e.g. $ my_var port 3306 $ my_var datadir /usr/local/mysql/data 134 Friday, 06 August 2010 134

mysql.sh functions • my_status "what" • returns the value of a given status item • e.g. $ my_status com_select 2653 $ my_status connections 452 135 Friday, 06 August 2010 135

mysql.sh functions • my_dump_dataset "query" [header] • returns the values of a given query separated by tabs • if a header is requested, it will be part of the result • e.g. $ my_dump_dataset "select * from world.City" | head -n 5 1 Kabul AFG Kabol 1780000 2 Qandahar AFG Qandahar 237500 3 Herat AFG Herat 186800 4 Mazar-e-Sharif AFG Balkh 127800 136 5 Amsterdam NLD Noord-Holland 731200 Friday, 06 August 2010 136

mysql.sh functions • my_csv "query" [separator] • returns the values of a given query separated by commas, or by a given separator • e.g. $ my_csv 'select * from world.City' | head -n 3 "ID","Name","CountryCode","District"," Population" 1,"Kabul","AFG","Kabol",1780000 2,"Qandahar","AFG","Qandahar",237500 137 Friday, 06 August 2010 137

inside the command line client handling :-) @ @ $_ output 138 Friday, 06 August 2010 138

pager mysql> SELECT * FROM world.City; rec … rec … rec … 4000 + records unreadable 139 Friday, 06 August 2010 139

pager mysql> pager less PAGER set to 'less' mysql> SELECT * FROM world.City; # [ records in a manageable window ] mysql> nopager PAGER set to stdout # back to normal 140 Friday, 06 August 2010 140

More pager tricks mysql> pager vim - # like 'less', but you can also edit # the results and eventually save # them 141 Friday, 06 August 2010 141

More pager tricks mysql> pager md5sum PAGER set to 'md5sum' mysql> select * from City; 5d17ffa50d6da200dee621823ade2543 - 4079 rows in set (0.01 sec) # produce a CRC for the result set # instead of displaying it. # Useful for data comparison. # (Thanks to Baron Schwartz 142 Friday, 06 August 2010 142

Comparing data sets with md5sum mysql> select * from City; 5d17ffa50d6da200dee621823ade2543 - mysql> select * from City order by ID; 5d17ffa50d6da200dee621823ade2543 - mysql> select * from City order by name; 5ab980f5b69a6e65d3b8cd5a95ec94e0 - mysql> select * from City order by District; 2bf37cbe91b3ffeaebc8c4b6084b8784 - 143 Friday, 06 August 2010 143

More pager tricks mysql> pager cat > /tmp/t1.txt mysql> select "one" union select "two"; mysql> pager cat > /tmp/t2.txt mysql> select "one" union select "TWO"; mysql> nopager mysql> ! vimdiff -o /tmp/t[12].txt # see the difference between two # datasets 144 Friday, 06 August 2010 144

145 Friday, 06 August 2010 145

logging sessions mysql> tee mylog.txt # logs everything that you see on # the screen. # your commands, the results, # warnings, error messages, and so on. mysql> notee # back to normal 146 Friday, 06 August 2010 146

Updated presentation slides  you will find an up to date copy of these slides at:  http://tinyurl.com/shoot-cl-2010  http://tinyurl.com/shoot-cl-full-2010 Friday, 06 August 2010 147

read more  http://datacharmer.blogspot.com Friday, 06 August 2010 148

THANKS! ? Question time Comments on Twitter: @datacharmer This work is licensed under the Creative Commons Attribution-Share Alike 3.0 Unported License. Friday, 06 August 2010 149

Shooting from the hip - Perl at the command line for DBAs

by Giuseppe Maxia on Aug 06, 2010

Accessibility

Categories

Tags

Upload Details

Usage Rights

2 Embeds 22

Statistics

Shooting from the hip - Perl at the command line for DBAs — Presentation Transcript

http://www.jugsardegna.org	19
url_unknown	3