Shooting from the hip.
           Perl at the command




                                                                ...
about me -Giuseppe Maxia
        a.k.a. The Data Charmer
        MySQL Community Team Lead
        Long time hacking wi...
Updated presentation slides




    you will find an up to date copy of these slides at:
    http://tinyurl.com/shoot-cl...
I love your feedback!
      Tell it to the Twitter community
        #yapc_eu "your feedback here"
      Twitter me
   ...
This is a story about
            languages

Friday, 06 August 2010         5
you got to understand,
                         Bernie, it's not personal.
                           It's just that you a...
10 print "hello"
We use                   20 goto 10
languages
to talk to
computers



Friday, 06 August 2010             ...
I am a polyglot



Friday, 06 August 2010                     8
An imperfect
                 polyglot, actually


Friday, 06 August 2010                9
I speak
   Italian




Friday, 06 August 2010   10
I speak
   Italian


       with a clear
        Sardinian
         accent

Friday, 06 August 2010   10
I speak
   English




Friday, 06 August 2010   11
I speak
   English


     with a distinct
     Italian accent

Friday, 06 August 2010   11
I speak
   Perl




Friday, 06 August 2010   12
I speak
   Perl


  with a hint of a C
        accent

Friday, 06 August 2010   12
I speak
   French
   and
   Spanish




Friday, 06 August 2010   13
I speak
   French
   and
   Spanish
   but since I use
  them very little,
 they have melted
  with Italian and
       Eng...
I speak
                         #!/bin/bash
   Bash




Friday, 06 August 2010                 14
I speak
                         #!/bin/bash
   Bash


      with a strong
       Perl accent

Friday, 06 August 2010     ...
I speak fluent Italian

                         La donzelletta vien dalla
                         campagna, in sul calar...
However, Sardinian comes handy sometimes




                          Maissavida
                          ti frimisti! (...
I speak fluent English


                         When in the Course of human events, it becomes
                         ...
I speak fluent English


                         When in the Course of human events, it becomes
                         ...
I speak fluent English


                         When in the Course of human events, it becomes
                         ...
But sometimes Italian is still necessary


                                  barista al fresco pasta
                     ...
And sometimes Italian is needed




                           I tuoi perversi
                           defunti!(*)




...
Although Sardinian is still effective here




                              Ancu ti currullidi
                          ...
I speak Perl fluently

                         #!/usr/bin/perl
                         use strict;
                     ...
Sometimes, I need help from C

                         use Inline C;
                         print "9 + 16 = ", add(9, 1...
Most of my work at the command line is Bash


                         $ for D in $HOME/*/; do du -sh $D;
                ...
And sometimes, a pinch of Perl can help


                  $ perl -pe
                  's{(dd)-(dd)-(dddd)}{$3-$1-$2}g' ...
And then on top of all this, I need to speak SQL


                  $ mysql -e
                  'SELECT COUNT(*) FROM db...
I wish I could say that my SQL has a Perlish
   accent


                         $ mysql -e
                         'SEL...
I wish I could say that my SQL has a Perlish
   accent


                         $ mysql -e
                         'SEL...
I could curse in Perl




                           $ perl -pe
                           's{(dd)-(dd)-(ddd
             ...
Then




Friday, 06 August 2010   29
Then


    I will need to mix languages.




Friday, 06 August 2010              29
Then


    I will need to mix languages.


                         This talk will show exactly that.




Friday, 06 Augus...
Then


    I will need to mix languages.


                         This talk will show exactly that.


        Mixing lan...
Then


    I will need to mix languages.


                         This talk will show exactly that.


        Mixing lan...
Then


    I will need to mix languages.


                         This talk will show exactly that.


        Mixing lan...
What this tutorial is about
    Providing tools and tips for command line users;
    Learn a few tricks to make you achi...
What this tutorial is about
    Providing tools and tips for command line users;
    Learn a few tricks to make you achi...
the command line (for rookies)



                         BASICS
                                     31


Friday, 06 Aug...
The shell
      • It's a command interpreter
      • There are many flavors
           > bash
           > zsh
           ...
MySQL command line client
      • called "mysql"
           > Takes SQL commands
           > sends them to the server
   ...
other tools
      •   Shell scripts
      •   Aliases
      •   Perl
      •   awk
      •   sort
      •    …. and a lot ...
other tools
      •   Shell scripts
      •   Aliases
      •   Perl          which could replace everything else, of cour...
in case it wasn't clear (yeah, for
                                  rookies)




                                    35

...
in case it wasn't clear (yeah, for
                                  rookies)
      • command line actions are




       ...
in case it wasn't clear (yeah, for
                                  rookies)
      • command line actions are
           ...
in case it wasn't clear (yeah, for
                                  rookies)
      • command line actions are
           ...
in case it wasn't clear (yeah, for
                                  rookies)
      • command line actions are
           ...
in case it wasn't clear (yeah, for
                                  rookies)
      • command line actions are
           ...
The Unix philosophy (1)
      • One tool for every feature
           > make one tool that does one thing well
           ...
The Unix philosophy (2)
    • To make several tool cooperate:
           > pipe the output of one tool to the input of ano...
Unix pipes




                         38


Friday, 06 August 2010        38
Unix pipes

              application   <   input




                                        38


Friday, 06 August 2010 ...
Unix pipes

              application   <   input

              application   >   output




                            ...
Unix pipes

              application   <   input

              application   >   output


              application   < ...
Unix pipes

              application   <   input

              application   >   output


              application   < ...
Unix pipes

              application         <   input

              application         >   output


              appl...
Unix pipes = parallel processing




                                   39


Friday, 06 August 2010                  39
Unix pipes = parallel processing

            application   |   application




                                          ...
Unix pipes = parallel processing

            application   |   application




                                          ...
Unix pipes = parallel processing

            application   |   application

              output




                    ...
Unix pipes = parallel processing

            application   |     application

              output          input




   ...
Unix pipes = parallel processing

            application   |     application

              output          input




   ...
Unix pipes = parallel processing

            application   |     application

              output          input




   ...
Unix pipes = parallel processing

            application   |     application

              output          input




   ...
Who gets what
      •   shell
      •   application
      •   database server
      •   pager




                        ...
who gets what


            shell prompt
            $ mysql



                           mysql client




              ...
who gets what
                                        any host   mysql server




            shell
                      ...
who gets what
$ mysql -u username -ppassword




                         First the application
                         t...
who gets what
$ mysql -u username -ppassword 
  -e "select VERSION()"



                                 goes to the clie...
who gets what
mysql> help
mysql> pager less                             goes to the client only
mysql> select 1;




     ...
the command line




      readline
                         46


Friday, 06 August 2010        46
readline
      • Common component
           > shell
           > mysql client
           > many CLI applications




    ...
Main cursor movements
            Ctrl A       Start/End of line
            Ctrl E
            Meta B       Back/Forward ...
Changing text
            Meta U            Uppercase word
            Meta L            Lowercase word
            Ctrl D...
Some conventions in this talk
$ command
at the shell

mysql> command
at the mysql prompt




                             ...
the Bash shell



                     #!
                          bash
                             51


Friday, 06 Augu...
components
      •   commands
      •   aliases
      •   variables
      •   functions




                         52


...
commands
$ echo "something"
something




                         53


Friday, 06 August 2010        53
aliases
$ alias mydate='mysql -e "select NOW()"'
$ mydate
+---------------------+
| now()               |
+---------------...
aliases
$ alias myself='mysql'
$ myself -e "select NOW()"'
+---------------------+
| now()                |
+-------------...
variables
$ MYBATCH='mysql -N -B'
$ MYQUERY='select curdate()'
$ $MYBATCH -e "$MYQUERY"
2010-04-12




                   ...
nesting variables
# t.sh
 CMD="SELECT TABLE_NAME"
 TBL='information_schema.tables'
 WHC1="TABLE_SCHEMA='mysql'"
 WHC2="TAB...
nesting variables
$ sh t.sh
SELECT TABLE_NAME FROM
information_schema.tables WHERE
TABLE_SCHEMA='mysql' AND TABLE_NAME lik...
the mysql command line client



                         mysql>

                                    59


Friday, 06 Augu...
SQL commands from a file
   $ mysql < filename.sql
   mysql> source filename.sql

   • executes the commands contained in
...
SQL commands at the command line
   $ application | mysql
   $ echo "SELECT 1" | mysql
   $ mysql -e "SELECT 1"

   • exec...
batch mode
   $ echo "select curdate()" | mysql
   curdate()
   2010-04-12




                                 62


Frida...
table mode
   $ echo "select curdate()" | mysql -t
   +------------+
   | curdate() |
   +------------+
   | 2010-04-12 |
...
table mode
   $ mysql -e "select curdate()"
   +------------+
   | curdate() |
   +------------+
   | 2010-04-12 |
   +---...
Forcing batch mode
   mysql -B -e 'select curdate()'
   curdate()
   2010-04-12




                                    65...
Forcing batch mode - no header
   mysql -N -B -e 'select curdate()'
   2010-04-12




                                   A...
Shell + mysql
   $ echo "The current server's date is
   `mysql -N -B -e 'select curdate()'`."

   $ echo "The current ser...
advanced command line techniques
               ((((((
               (:
           @             @
                -


  ...
loops basics
   $      for N in 1 2 3
   >      do
   >      echo $N
   >      done
   1
   2
   3



                    ...
loops basics
   $ for N in 1 2 3 ; do echo $N ; done
   1
   2
   3




                                 70


Friday, 06 A...
loops basics
   $ for N in $(seq 1 3) ;
    do echo "this is number $N" ;
    done

   this is number 1
   this is number ...
loops from file
   $ head /usr/share/dict/words
   A
   a
   aa
   aal
   aalii
   aam
   Aani
   aardvark
   aardwolf
   ...
loops from file
   $ for U in 
   $(head /usr/share/dict/words ) ; do
   perl -le 'print uc shift' $U ; done
   A
   A
   ...
loops from mysql results
   $ mysql -B -N -e 'show tables from
   world'
   City
   Country
   CountryLanguage




       ...
loops from mysql results
   $ for T in $(mysql -B -N -e 'show
   tables from world') ;
   >    do echo "CREATE TABLE world...
piping loops back into mysql
   $ for T in $(mysql -B -N -e 'show
   tables from world') ;
   >    do echo "CREATE TABLE w...
advanced command line techniques



               nested

                                      :)
                      ...
doing something for every table
    in every database
   $ mysql -e "show databases"
   +--------------------+
   | Databa...
doing something for every table
    in every database
   $ mysql -e "show tables from world"
   +-----------------+
   | T...
pseudocode
for each database
  do
  for each table
     do
     print the table name




                            80


...
actual code
alias mybatch='mysql -B -N'
for D in $(mybatch -e "SHOW SCHEMAS")
do
   for T in 
   $(mybatch -e "SHOW TABLES...
advanced command line techniques
                           $HOME




     options
      files                        82

...
Default option files
/etc/my.cnf
/etc/mysql/my.cnf (Debian, Ubuntu)
$BASEDIR/my.cnf
$HOME/.my.cnf




                    ...
Custom option files
$ mysql --no-defaults
$ mysql --defaults-file=filename
$ mysql --defaults-extra-file=filename




    ...
Option files sections
[mysqld]
# option for server

[client]
# option for any mysql client

[mysql]
# options specific to ...
Option files example

[client]
user=myname
password=mypassword

[mysql]
prompt="mysql (h) {u} [d] > "



                 ...
Option files: caution
# protect your option files!

$ chmod 600 $HOME/.my.cnf

$ ls -l .my.cnf
-rw------- 1 [...] 10:04 .m...
prompt
mysql> prompt gm (h) {u} [d] >
PROMPT set to 'gm (h) {u} [d] > '
gm (localhost) {root} [(none)] > use test
Database...
advanced command line techniques
            ((((((
            (:

        @            @
             -




            ...
An useful alias
$ alias mysql_as_root=
'mysql --
defaults_file=~/.my_as_root.cnf'

mysql_as_root -e "select user()"
+-----...
An useful alias
#.my_as_root.cnf'
[client]
user=root
password=your_mysql_root_password

[mysql]
prompt="### [h] {u} (d) > ...
Another useful alias
alias mybatch='mysql -B -N '


mybatch -e "select version()"
5.1.45-log




                         ...
( )
    advanced command line techniques
                    ((((((
                    (:
                @            @
...
functions
      • like aliases, but with parameters




                                            94


Friday, 06 August...
sample function
function grant_new {
    mysql_as_root -e 
    "GRANT $1 on $2 to $3 identified
by "$4" "
}


$ grant_new ...
sample function
$ mysql_as_root 
  -e 'show grants for johnG'
*** 1. row ***
Grants for john@%: GRANT USAGE ON *.*
TO 'joh...
advanced command line techniques



              $_         getting
                          query
                     ...
which query is being executed?
# wrong.sql
# this script has one error
use test ;
drop table if exists t1 ;
create table t...
which query is being executed?
$ mysql < wrong.sql
ERROR 1054 (42S22) at line 4: Unknown
column 'a' in 'field list'


# WH...
which query is being executed?
$ mysql -v < wrong.sql
--------------
drop table if exists t1
--------------
--------------...
getting execution times
$ mysql -vv -e 
'select rating,count(*) from sakila.film
group by rating'
+--------+----------+
| ...
getting execution times (batch mode)
$echo 'select rating,count(*) from sakila.film group
by rating' | mysql -vv
---------...
getting execution times (batch mode)
$ echo 'select rating,count(*) from sakila.film group
by rating' | mysql -vvv
-------...
advanced command line techniques



                         getting
        _$

                         distinct
       ...
getting a single value
$ mysql -e "show variables like 'port'"
+---------------+-------+
| Variable_name | Value |
+------...
getting a single value
$ mysql -NB -e "show variables like 'port'"
 port           3306


# no header, batch mode
# still ...
getting a single value
$ mysql -NB 
  -e "show variables like 'port'" 
  | awk '{print $2}'

3306

$ mysql -NB 
  -e "show...
getting a single value: is replication
    working?
$ mysql 
  -e "show slave status G" 
 | grep -i slave_io_running

    ...
getting a single value: is replication
    working?
$ mysql 
  -e "show slave status G" 
  | grep -i "slave_io_running" 
 ...
What's perl -lane?




                         110


Friday, 06 August 2010         110
What's perl -lane?   perl -lane
                         must
                         turn perl




                     ...
Perl command line must-know facts (1)
perl -e 'print "hellon"'
# executes perl code




                              111
...
Perl command line must-know facts (2)
perl -ne 'print if /^a/'
# starts a loop from standard input
# and executes the perl...
Perl command line must-know facts (3)
perl -pe 's/(^aw+)/$1/'
# like -ne, but prints each line

$ perl -MO=Deparse -pe 's/...
Perl command line must-know facts (4)
perl -le 'print "hello"'
# strips newlines from input and adds
# them to output

$ p...
Perl command line must-know facts (5)
perl -lane 'print $F[0]'
# splits input into @F array

$ echo "a b c" | 
   perl -la...
Perl command line must-know facts (6)
Perl and shell contention for $




 #!/bin/bash

                         Take the ...
Beware of $VARIABLES on the
command line
$ perl -le "$x=1; print $x"
syntax error at -e line 1, near "="
Execution of -e a...
double quotes
Beware of $VARIABLES on theconsumed by the
                           allow the shell to
                   ...
double quotes
Beware of $VARIABLES on theconsumed by the
                           allow the shell to
                   ...
double quotes
Beware of $VARIABLES on the   consumed by the
                              allow the shell to
             ...
double quotes
Beware of $VARIABLES on the   consumed by the
                              allow the shell to
             ...
advanced command line techniques


         combining
          column
           names
                               118...
combining column names
$ mysql -N -e 'desc world.City'
+-------------+----------+----+-----+
|          ID | int(11) | NO ...
combining column names
#     Expected result:
#     SELECT CONCAT_WS('|',
#     ID,Name,CountryCode,District,
#     Popula...
combining column names (I)
echo "select concat_ws('|',
  $(mysql -N -e 'desc world.City' | 
   perl -lane 'push @f, $F[0];...
combining column names (II)
echo "select concat_ws('|', $(
  mysql -NB -e 'select group_concat
(column_name) from
informat...
combining column names (IIa)
echo "select concat_ws('|', $(
  mysql -NB -e 'select group_concat
(column_name) from
informa...
piping the result back to mysql
echo "select concat_ws('|', $(
  mysql -NB -e 'select group_concat
(column_name) from
info...
advanced command line techniques


                 $_

                         dumping
                          data
  ...
advanced command line techniques


                 $_

                         dumping
                          data
  ...
dumping a tab delimited data set
# the naive way
$ mysql 
  -e "SELECT *
  into outfile '/tmp/city_data'
from world.City"
...
dumping a tab delimited data set
$ mysql -B -N 
  -e "SELECT * from world.City" 
  > /tmp/city_data

# it works always

# ...
dumping a CSV
$ mysql -B 
  -e "SELECT * from world.City" 
 | perl -F"t" -lane 
  'print join ",",
   map {s/"/""/g;
     ...
dumping a CSV
$ mysql -B 
  -e "SELECT * from world.City" 
 | perl -F"t" -lane 
  'print join ",",    splits the line by t...
advanced command line techniques




      script
        it
                               129


Friday, 06 August 2010  ...
use the mysql.sh script
            The companion code for this tutorial
            http://tinyurl.com/cli-tut-code-201...
Usage
      • download the script
      • load it

         $ source mysql.sh




                              131


Frid...
mysql.sh functions
      •   my_batch "query"
      •   returns the result of the query.
      •   Without header
      • ...
mysql.sh functions
      • my_value "what"
      • returns a single value

      • e.g.
      $ my_value "version()"
     ...
mysql.sh functions
      • my_var "what"
      • returns the value of a given variable

      • e.g.
      $ my_var port
 ...
mysql.sh functions
      • my_status "what"
      • returns the value of a given status item

      • e.g.
      $ my_stat...
mysql.sh functions
      • my_dump_dataset "query" [header]
      • returns the values of a given query separated by tabs
...
mysql.sh functions
      • my_csv "query" [separator]
      • returns the values of a given query separated by
        com...
inside the command line client

     handling


                                           :-)
                           ...
pager
mysql> SELECT * FROM world.City;

rec …
rec …
rec …

4000 + records
unreadable


                               139
...
pager
mysql> pager less
PAGER set to 'less'
mysql> SELECT * FROM world.City;

# [ records in a manageable window ]

mysql>...
More pager tricks
mysql> pager vim -
# like 'less', but you can also edit
# the results and eventually save
# them




   ...
More pager tricks
mysql> pager md5sum
PAGER set to 'md5sum'
mysql> select * from City;
5d17ffa50d6da200dee621823ade2543   ...
Comparing data sets with md5sum
mysql> select * from City;
5d17ffa50d6da200dee621823ade2543   -

mysql> select * from City...
Comparing data sets with md5sum
mysql> select * from City;
5d17ffa50d6da200dee621823ade2543   -

mysql> select * from City...
More pager tricks
mysql>                   pager cat > /tmp/t1.txt
mysql>                   select "one" union select
"two...
145


Friday, 06 August 2010         145
logging sessions
mysql> tee mylog.txt

#      logs everything that you see on
#      the screen.
#      your commands, the...
Updated presentation slides




    you will find an up to date copy of these slides at:
    http://tinyurl.com/shoot-cl...
read more
    http://datacharmer.blogspot.com




Friday, 06 August 2010                 148
THANKS!
             ?
                                  Question
                                    time
               ...
Upcoming SlideShare
Loading in...5
×

Shooting from the hip - Perl at the command line for DBAs

2,404

Published on

There are many GUI applications to deal with databases and with MySQL in particular. But for die hard command line enthusiasts, who grew up with Linux, the shell prompt is still prince. You would be surprised at the vast amount of operations that you can do from the command line, and then many would surely marvel at the amount of work that a Perl one-liner can do.
By combining shell and Perl, you get the kind of power that all DBAs dream, but few achieve.
Running commands on several databases at once, filtering users, getting detailed metadata, transferring data from the operating system to a database and the other way around, creating scripts from database data, creating SQL
commands from database data. That, and more, you can learn by attending this session. Learn the art of the command line as a shortcut to quick and efficient administration, and as help for demanding developers.

Published in: Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
2,404
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
26
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Shooting from the hip - Perl at the command line for DBAs

  1. 1. Shooting from the hip. Perl at the command :) @ @ line for DBAs - $_ #! Giuseppe Maxia MySQL Community Team Lead @datacharmer This work is licensed under the Creative Commons Attribution-Share Alike 3.0 Unported License. Friday, 06 August 2010 1
  2. 2. about me -Giuseppe Maxia  a.k.a. The Data Charmer  MySQL Community Team Lead  Long time hacking with MySQL features  Formerly, database consultant, designer, coder.  A passion for QA  An even greater passion for open source  ... and community  Passionate blogger  http://datacharmer.blogspot.com Friday, 06 August 2010 2
  3. 3. Updated presentation slides  you will find an up to date copy of these slides at:  http://tinyurl.com/shoot-cl-2010  http://tinyurl.com/shoot-cl-full-2010 Friday, 06 August 2010 3
  4. 4. I love your feedback!  Tell it to the Twitter community  #yapc_eu "your feedback here"  Twitter me  @datacharmer "your feedback here"  Blog about it Friday, 06 August 2010 4
  5. 5. This is a story about languages Friday, 06 August 2010 5
  6. 6. you got to understand, Bernie, it's not personal. It's just that you are obsolete We use language to talk to computers Friday, 06 August 2010 6
  7. 7. 10 print "hello" We use 20 goto 10 languages to talk to computers Friday, 06 August 2010 7
  8. 8. I am a polyglot Friday, 06 August 2010 8
  9. 9. An imperfect polyglot, actually Friday, 06 August 2010 9
  10. 10. I speak Italian Friday, 06 August 2010 10
  11. 11. I speak Italian with a clear Sardinian accent Friday, 06 August 2010 10
  12. 12. I speak English Friday, 06 August 2010 11
  13. 13. I speak English with a distinct Italian accent Friday, 06 August 2010 11
  14. 14. I speak Perl Friday, 06 August 2010 12
  15. 15. I speak Perl with a hint of a C accent Friday, 06 August 2010 12
  16. 16. I speak French and Spanish Friday, 06 August 2010 13
  17. 17. I speak French and Spanish but since I use them very little, they have melted with Italian and English Friday, 06 August 2010 13
  18. 18. I speak #!/bin/bash Bash Friday, 06 August 2010 14
  19. 19. I speak #!/bin/bash Bash with a strong Perl accent Friday, 06 August 2010 14
  20. 20. I speak fluent Italian La donzelletta vien dalla campagna, in sul calar del sole, col suo fascio dell'erba e reca in mano un mazzolino di rose e viole, onde, siccome suole, ornare ella si appresta ... Friday, 06 August 2010 15
  21. 21. However, Sardinian comes handy sometimes Maissavida ti frimisti! (*) (*) May you never stop Friday, 06 August 2010 16
  22. 22. I speak fluent English When in the Course of human events, it becomes necessary for one people to dissolve the political bands which have connected them with another, and to assume among the powers of the earth, the separate and equal station to which the Laws of Nature and of Nature's God entitle them, a decent respect to the opinions of mankind requires that they should declare the causes which impel them to the separation. Friday, 06 August 2010 17
  23. 23. I speak fluent English When in the Course of human events, it becomes necessary for one people to dissolve the political bands which have connected them with another, and to assume among the powers of the earth, the separate and equal station to which the Laws of Nature and of Nature's God entitle them, a decent respect to the opinions of mankind requires that they should declare the causes which impel them to the separation. hmm. wrong flag Friday, 06 August 2010 17
  24. 24. I speak fluent English When in the Course of human events, it becomes necessary for one people to dissolve the political bands which have connected them with another, and to assume among the powers of the earth, the separate and equal station to which the Laws of Nature and of Nature's God entitle them, a decent respect to the opinions of mankind requires that they should declare the causes which impel them to the separation. Friday, 06 August 2010 17
  25. 25. But sometimes Italian is still necessary barista al fresco pasta maestro espresso tempo vendetta soprano finale piano dilettante cappuccino stucco stiletto virtuoso fiasco presto lotto pizza bravo lasagna studio spaghetti stanza Friday, 06 August 2010 18
  26. 26. And sometimes Italian is needed I tuoi perversi defunti!(*) (*) Your wicked ancestors! Friday, 06 August 2010 19
  27. 27. Although Sardinian is still effective here Ancu ti currullidi sa giustizia (*) (*) May the rule of law chase you Friday, 06 August 2010 20
  28. 28. I speak Perl fluently #!/usr/bin/perl use strict; use warnings; use File::Find; while (my $line = <stdin>){ # … Friday, 06 August 2010 21
  29. 29. Sometimes, I need help from C use Inline C; print "9 + 16 = ", add(9, 16), "n"; __END__ __C__ int add(int x, int y) { return x + y; } Friday, 06 August 2010 22
  30. 30. Most of my work at the command line is Bash $ for D in $HOME/*/; do du -sh $D; > done > /tmp/occupancy #!/bin/bash Friday, 06 August 2010 23
  31. 31. And sometimes, a pinch of Perl can help $ perl -pe 's{(dd)-(dd)-(dddd)}{$3-$1-$2}g' < rawfile.txt > betterfile.txt #!/bin/bash Friday, 06 August 2010 24
  32. 32. And then on top of all this, I need to speak SQL $ mysql -e 'SELECT COUNT(*) FROM dbname.tbl_name WHERE col_name = 1 and other_col="A" ' Friday, 06 August 2010 25
  33. 33. I wish I could say that my SQL has a Perlish accent $ mysql -e 'SELECT COUNT(*) FROM dbname.tbl_name WHERE col_name = 1 and other_col="A" ' use strict; use DBI;? Friday, 06 August 2010 26
  34. 34. I wish I could say that my SQL has a Perlish accent $ mysql -e 'SELECT COUNT(*) FROM dbname.tbl_name WHERE col_name = 1 and other_col="A" ' but it wouldn't be true Friday, 06 August 2010 27
  35. 35. I could curse in Perl $ perl -pe 's{(dd)-(dd)-(ddd d)}{$3-$1-$2}g' < rawfile.txt > betterfile.txt (*) But it would not get me far. (*) already translated Friday, 06 August 2010 28
  36. 36. Then Friday, 06 August 2010 29
  37. 37. Then I will need to mix languages. Friday, 06 August 2010 29
  38. 38. Then I will need to mix languages. This talk will show exactly that. Friday, 06 August 2010 29
  39. 39. Then I will need to mix languages. This talk will show exactly that. Mixing languages. Friday, 06 August 2010 29
  40. 40. Then I will need to mix languages. This talk will show exactly that. Mixing languages. Yes. Friday, 06 August 2010 29
  41. 41. Then I will need to mix languages. This talk will show exactly that. Mixing languages. Yes. $ at the command line Friday, 06 August 2010 29
  42. 42. What this tutorial is about  Providing tools and tips for command line users;  Learn a few tricks to make you achieve more and type less. Friday, 06 August 2010 30
  43. 43. What this tutorial is about  Providing tools and tips for command line users;  Learn a few tricks to make you achieve more and type less.  Making GUI enthusiasts green with envy; bar foo baz Friday, 06 August 2010 30
  44. 44. the command line (for rookies) BASICS 31 Friday, 06 August 2010 31
  45. 45. The shell • It's a command interpreter • There are many flavors > bash > zsh > ksh 32 Friday, 06 August 2010 32
  46. 46. MySQL command line client • called "mysql" > Takes SQL commands > sends them to the server > gets results • but it can do much more ... 33 Friday, 06 August 2010 33
  47. 47. other tools • Shell scripts • Aliases • Perl • awk • sort • …. and a lot of Unix little helper 34 Friday, 06 August 2010 34
  48. 48. other tools • Shell scripts • Aliases • Perl which could replace everything else, of course • awk • sort • …. and a lot of Unix little helper 34 Friday, 06 August 2010 34
  49. 49. in case it wasn't clear (yeah, for rookies) 35 Friday, 06 August 2010 35
  50. 50. in case it wasn't clear (yeah, for rookies) • command line actions are 35 Friday, 06 August 2010 35
  51. 51. in case it wasn't clear (yeah, for rookies) • command line actions are > commands (the name of the application to invoke) 35 Friday, 06 August 2010 35
  52. 52. in case it wasn't clear (yeah, for rookies) • command line actions are > commands (the name of the application to invoke) > parameters (optional arguments for the application) 35 Friday, 06 August 2010 35
  53. 53. in case it wasn't clear (yeah, for rookies) • command line actions are > commands (the name of the application to invoke) > parameters (optional arguments for the application) > and the ENTER key 35 Friday, 06 August 2010 35
  54. 54. in case it wasn't clear (yeah, for rookies) • command line actions are > commands (the name of the application to invoke) > parameters (optional arguments for the application) > and the ENTER key Enter 35 Friday, 06 August 2010 35
  55. 55. The Unix philosophy (1) • One tool for every feature > make one tool that does one thing well > for many features, make many tools > make them work together (see next slide) 36 Friday, 06 August 2010 36
  56. 56. The Unix philosophy (2) • To make several tool cooperate: > pipe the output of one tool to the input of another one. > for more actions, pipe more tools $ sort < filename | uniq | nl 37 Friday, 06 August 2010 37
  57. 57. Unix pipes 38 Friday, 06 August 2010 38
  58. 58. Unix pipes application < input 38 Friday, 06 August 2010 38
  59. 59. Unix pipes application < input application > output 38 Friday, 06 August 2010 38
  60. 60. Unix pipes application < input application > output application < input > output 38 Friday, 06 August 2010 38
  61. 61. Unix pipes application < input application > output application < input > output application | application 38 Friday, 06 August 2010 38
  62. 62. Unix pipes application < input application > output application < input > output application | application output input 38 Friday, 06 August 2010 38
  63. 63. Unix pipes = parallel processing 39 Friday, 06 August 2010 39
  64. 64. Unix pipes = parallel processing application | application 39 Friday, 06 August 2010 39
  65. 65. Unix pipes = parallel processing application | application 39 Friday, 06 August 2010 39
  66. 66. Unix pipes = parallel processing application | application output 39 Friday, 06 August 2010 39
  67. 67. Unix pipes = parallel processing application | application output input 39 Friday, 06 August 2010 39
  68. 68. Unix pipes = parallel processing application | application output input 39 Friday, 06 August 2010 39
  69. 69. Unix pipes = parallel processing application | application output input 39 Friday, 06 August 2010 39
  70. 70. Unix pipes = parallel processing application | application output input 39 Friday, 06 August 2010 39
  71. 71. Who gets what • shell • application • database server • pager 40 Friday, 06 August 2010 40
  72. 72. who gets what shell prompt $ mysql mysql client mysql server 41 Friday, 06 August 2010 41
  73. 73. who gets what any host mysql server shell mysql client 42 Friday, 06 August 2010 42
  74. 74. who gets what $ mysql -u username -ppassword First the application then the server 43 Friday, 06 August 2010 43
  75. 75. who gets what $ mysql -u username -ppassword -e "select VERSION()" goes to the client, then the server goes to the server 44 Friday, 06 August 2010 44
  76. 76. who gets what mysql> help mysql> pager less goes to the client only mysql> select 1; goes to the server 45 Friday, 06 August 2010 45
  77. 77. the command line readline 46 Friday, 06 August 2010 46
  78. 78. readline • Common component > shell > mysql client > many CLI applications 47 Friday, 06 August 2010 47
  79. 79. Main cursor movements Ctrl A Start/End of line Ctrl E Meta B Back/Forward one word Meta F previous/next command Ctrl L clear screen and take current line on top 48 Friday, 06 August 2010 48
  80. 80. Changing text Meta U Uppercase word Meta L Lowercase word Ctrl D delete word Ctrl K delete to end of line Ctrl - undo for more tips: $ man readline 49 Friday, 06 August 2010 49
  81. 81. Some conventions in this talk $ command at the shell mysql> command at the mysql prompt 50 Friday, 06 August 2010 50
  82. 82. the Bash shell #! bash 51 Friday, 06 August 2010 51
  83. 83. components • commands • aliases • variables • functions 52 Friday, 06 August 2010 52
  84. 84. commands $ echo "something" something 53 Friday, 06 August 2010 53
  85. 85. aliases $ alias mydate='mysql -e "select NOW()"' $ mydate +---------------------+ | now() | +---------------------+ | 2010-04-12 07:14:19 | +---------------------+ 54 Friday, 06 August 2010 54
  86. 86. aliases $ alias myself='mysql' $ myself -e "select NOW()"' +---------------------+ | now() | +---------------------+ | 2010-04-12 07:14:19 | +---------------------+ 55 Friday, 06 August 2010 55
  87. 87. variables $ MYBATCH='mysql -N -B' $ MYQUERY='select curdate()' $ $MYBATCH -e "$MYQUERY" 2010-04-12 56 Friday, 06 August 2010 56
  88. 88. nesting variables # t.sh CMD="SELECT TABLE_NAME" TBL='information_schema.tables' WHC1="TABLE_SCHEMA='mysql'" WHC2="TABLE_NAME like 'u%' " Q1="$CMD FROM $TBL" Q2="WHERE $WHC1 AND $WHC2" Q="$Q1 $Q2" echo $Q echo $Q | mysql 57 Friday, 06 August 2010 57
  89. 89. nesting variables $ sh t.sh SELECT TABLE_NAME FROM information_schema.tables WHERE TABLE_SCHEMA='mysql' AND TABLE_NAME like 'u%' TABLE_NAME user 58 Friday, 06 August 2010 58
  90. 90. the mysql command line client mysql> 59 Friday, 06 August 2010 59
  91. 91. SQL commands from a file $ mysql < filename.sql mysql> source filename.sql • executes the commands contained in filename.sql • commands can be queries understood by the server • and commands for the client 60 Friday, 06 August 2010 60
  92. 92. SQL commands at the command line $ application | mysql $ echo "SELECT 1" | mysql $ mysql -e "SELECT 1" • executes the commands from standard input • commands can be queries understood by the server • and commands for the client 61 Friday, 06 August 2010 61
  93. 93. batch mode $ echo "select curdate()" | mysql curdate() 2010-04-12 62 Friday, 06 August 2010 62
  94. 94. table mode $ echo "select curdate()" | mysql -t +------------+ | curdate() | +------------+ | 2010-04-12 | +------------+ 63 Friday, 06 August 2010 63
  95. 95. table mode $ mysql -e "select curdate()" +------------+ | curdate() | +------------+ | 2010-04-12 | +------------+ 64 Friday, 06 August 2010 64
  96. 96. Forcing batch mode mysql -B -e 'select curdate()' curdate() 2010-04-12 65 Friday, 06 August 2010 65
  97. 97. Forcing batch mode - no header mysql -N -B -e 'select curdate()' 2010-04-12 ANT! PO RT IM 66 Friday, 06 August 2010 66
  98. 98. Shell + mysql $ echo "The current server's date is `mysql -N -B -e 'select curdate()'`." $ echo "The current server's date is $(mysql -N -B -e 'select curdate ()')." 67 Friday, 06 August 2010 67
  99. 99. advanced command line techniques (((((( (: @ @ - $_ loops 68 Friday, 06 August 2010 68
  100. 100. loops basics $ for N in 1 2 3 > do > echo $N > done 1 2 3 69 Friday, 06 August 2010 69
  101. 101. loops basics $ for N in 1 2 3 ; do echo $N ; done 1 2 3 70 Friday, 06 August 2010 70
  102. 102. loops basics $ for N in $(seq 1 3) ; do echo "this is number $N" ; done this is number 1 this is number 2 this is number 3 71 Friday, 06 August 2010 71
  103. 103. loops from file $ head /usr/share/dict/words A a aa aal aalii aam Aani aardvark aardwolf 72 Friday, 06 August 2010 72
  104. 104. loops from file $ for U in $(head /usr/share/dict/words ) ; do perl -le 'print uc shift' $U ; done A A AA AAL AALII AAM AANI 73 Friday, 06 August 2010 73
  105. 105. loops from mysql results $ mysql -B -N -e 'show tables from world' City Country CountryLanguage 74 Friday, 06 August 2010 74
  106. 106. loops from mysql results $ for T in $(mysql -B -N -e 'show tables from world') ; > do echo "CREATE TABLE world2.$T LIKE world.$T;" ; done CREATE TABLE world2.City LIKE world.City; CREATE TABLE world2.Country LIKE world.Country; CREATE TABLE world2.CountryLanguage LIKE world.CountryLanguage; 75 Friday, 06 August 2010 75
  107. 107. piping loops back into mysql $ for T in $(mysql -B -N -e 'show tables from world') ; > do echo "CREATE TABLE world2.$T LIKE world.$T;" ; done > | mysql -vv 76 Friday, 06 August 2010 76
  108. 108. advanced command line techniques nested :) @ @ - $_ #! #! #! #! loops #! #! 77 Friday, 06 August 2010 77
  109. 109. doing something for every table in every database $ mysql -e "show databases" +--------------------+ | Database | +--------------------+ | information_schema | | mysql | | sakila | | world | +--------------------+ 78 Friday, 06 August 2010 78
  110. 110. doing something for every table in every database $ mysql -e "show tables from world" +-----------------+ | Tables_in_world | +-----------------+ | City | | Country | | CountryLanguage | +-----------------+ 79 Friday, 06 August 2010 79
  111. 111. pseudocode for each database do for each table do print the table name 80 Friday, 06 August 2010 80
  112. 112. actual code alias mybatch='mysql -B -N' for D in $(mybatch -e "SHOW SCHEMAS") do for T in $(mybatch -e "SHOW TABLES FROM $D") do echo "$D.$T" done done 81 Friday, 06 August 2010 81
  113. 113. advanced command line techniques $HOME options files 82 Friday, 06 August 2010 82
  114. 114. Default option files /etc/my.cnf /etc/mysql/my.cnf (Debian, Ubuntu) $BASEDIR/my.cnf $HOME/.my.cnf 83 Friday, 06 August 2010 83
  115. 115. Custom option files $ mysql --no-defaults $ mysql --defaults-file=filename $ mysql --defaults-extra-file=filename 84 Friday, 06 August 2010 84
  116. 116. Option files sections [mysqld] # option for server [client] # option for any mysql client [mysql] # options specific to 'mysql' CLI app 85 Friday, 06 August 2010 85
  117. 117. Option files example [client] user=myname password=mypassword [mysql] prompt="mysql (h) {u} [d] > " 86 Friday, 06 August 2010 86
  118. 118. Option files: caution # protect your option files! $ chmod 600 $HOME/.my.cnf $ ls -l .my.cnf -rw------- 1 [...] 10:04 .my.cnf 87 Friday, 06 August 2010 87
  119. 119. prompt mysql> prompt gm (h) {u} [d] > PROMPT set to 'gm (h) {u} [d] > ' gm (localhost) {root} [(none)] > use test Database changed gm (localhost) {root} [test] > # h -> host # u -> user # d -> database 88 Friday, 06 August 2010 88
  120. 120. advanced command line techniques (((((( (: @ @ - #! aliases 89 Friday, 06 August 2010 89
  121. 121. An useful alias $ alias mysql_as_root= 'mysql -- defaults_file=~/.my_as_root.cnf' mysql_as_root -e "select user()" +----------------+ | user() | +----------------+ | root@localhost | +----------------+ 90 Friday, 06 August 2010 90
  122. 122. An useful alias #.my_as_root.cnf' [client] user=root password=your_mysql_root_password [mysql] prompt="### [h] {u} (d) > " 91 Friday, 06 August 2010 91
  123. 123. Another useful alias alias mybatch='mysql -B -N ' mybatch -e "select version()" 5.1.45-log 92 Friday, 06 August 2010 92
  124. 124. ( ) advanced command line techniques (((((( (: @ @ - #! functions 93 Friday, 06 August 2010 93
  125. 125. functions • like aliases, but with parameters 94 Friday, 06 August 2010 94
  126. 126. sample function function grant_new { mysql_as_root -e "GRANT $1 on $2 to $3 identified by "$4" " } $ grant_new SELECT "world.*" john john_secret 95 Friday, 06 August 2010 95
  127. 127. sample function $ mysql_as_root -e 'show grants for johnG' *** 1. row *** Grants for john@%: GRANT USAGE ON *.* TO 'john'@'%' IDENTIFIED BY PASSWORD '*73A88CE13374128844E3ABB500F439A9C02C 53A3' *** 2. row *** Grants for john@%: GRANT SELECT ON `world`.* TO 'john'@'%' 96 Friday, 06 August 2010 96
  128. 128. advanced command line techniques $_ getting query details 97 Friday, 06 August 2010 97
  129. 129. which query is being executed? # wrong.sql # this script has one error use test ; drop table if exists t1 ; create table t1 (id int); insert into t1 values (a); 98 Friday, 06 August 2010 98
  130. 130. which query is being executed? $ mysql < wrong.sql ERROR 1054 (42S22) at line 4: Unknown column 'a' in 'field list' # WHERE IS THE ERROR? 99 Friday, 06 August 2010 99
  131. 131. which query is being executed? $ mysql -v < wrong.sql -------------- drop table if exists t1 -------------- -------------- create table t1 (id int) -------------- -------------- insert into t1 values (a) -------------- ERROR 1054 (42S22) at line 4: Unknown column 'a' in 'field list' 100 Friday, 06 August 2010 100
  132. 132. getting execution times $ mysql -vv -e 'select rating,count(*) from sakila.film group by rating' +--------+----------+ | rating | count(*) | +--------+----------+ | G | 178 | | PG | 194 | | PG-13 | 223 | | R | 195 | | NC-17 | 210 | +--------+----------+ 5 rows in set (0.00 sec) 101 Friday, 06 August 2010 101
  133. 133. getting execution times (batch mode) $echo 'select rating,count(*) from sakila.film group by rating' | mysql -vv -------------- select rating,count(*) from sakila.film group by rating -------------- rating count(*) G 178 PG 194 PG-13 223 R 195 NC-17 210 5 rows in set 102 Friday, 06 August 2010 102
  134. 134. getting execution times (batch mode) $ echo 'select rating,count(*) from sakila.film group by rating' | mysql -vvv -------------- select rating,count(*) from sakila.film group by rating -------------- +--------+----------+ | rating | count(*) | +--------+----------+ | G | 178 | | PG | 194 | | PG-13 | 223 | | R | 195 | | NC-17 | 210 | +--------+----------+ 103 5 rows in set (0.00 sec) Friday, 06 August 2010 103
  135. 135. advanced command line techniques getting _$ distinct results 104 Friday, 06 August 2010 104
  136. 136. getting a single value $ mysql -e "show variables like 'port'" +---------------+-------+ | Variable_name | Value | +---------------+-------+ | port | 3306 | +---------------+-------+ # Here, we only want the port number 105 Friday, 06 August 2010 105
  137. 137. getting a single value $ mysql -NB -e "show variables like 'port'" port 3306 # no header, batch mode # still two values 106 Friday, 06 August 2010 106
  138. 138. getting a single value $ mysql -NB -e "show variables like 'port'" | awk '{print $2}' 3306 $ mysql -NB -e "show variables like 'port'" | perl -lane 'print $F[1]' 3306 # filtering the wanted value 107 Friday, 06 August 2010 107
  139. 139. getting a single value: is replication working? $ mysql -e "show slave status G" | grep -i slave_io_running Slave_IO_Running: Yes # too many things. # We only want a 'yes' or 'no' 108 Friday, 06 August 2010 108
  140. 140. getting a single value: is replication working? $ mysql -e "show slave status G" | grep -i "slave_io_running" | perl -lane 'print $F[1]' Yes # yes! 109 Friday, 06 August 2010 109
  141. 141. What's perl -lane? 110 Friday, 06 August 2010 110
  142. 142. What's perl -lane? perl -lane must turn perl 110 Friday, 06 August 2010 110
  143. 143. Perl command line must-know facts (1) perl -e 'print "hellon"' # executes perl code 111 Friday, 06 August 2010 111
  144. 144. Perl command line must-know facts (2) perl -ne 'print if /^a/' # starts a loop from standard input # and executes the perl code for each # iteration perl -MO=Deparse -ne 'print if /^a/' LINE: while (defined($_ = <ARGV>)) { print $_ if /^a/; } 112 Friday, 06 August 2010 112
  145. 145. Perl command line must-know facts (3) perl -pe 's/(^aw+)/$1/' # like -ne, but prints each line $ perl -MO=Deparse -pe 's/(^aw+)/$1/' LINE: while (defined($_ = <ARGV>)) { s/(^aw+)/$1/; } continue { print $_; } 113 Friday, 06 August 2010 113
  146. 146. Perl command line must-know facts (4) perl -le 'print "hello"' # strips newlines from input and adds # them to output $ perl -e 'print "hello"' hello$ $ perl -le 'print "hello"' hello $ 114 Friday, 06 August 2010 114
  147. 147. Perl command line must-know facts (5) perl -lane 'print $F[0]' # splits input into @F array $ echo "a b c" | perl -lane 'print $F[1]' b $ echo "a b c" | perl -lane 'print $F[2]' c 115 Friday, 06 August 2010 115
  148. 148. Perl command line must-know facts (6) Perl and shell contention for $ #!/bin/bash Take the $$$ and run 116 Friday, 06 August 2010 116
  149. 149. Beware of $VARIABLES on the command line $ perl -le "$x=1; print $x" syntax error at -e line 1, near "=" Execution of -e aborted due to compilation errors. 117 Friday, 06 August 2010 117
  150. 150. double quotes Beware of $VARIABLES on theconsumed by the allow the shell to shell, before Perl interpolate command line can see it variables $ perl -le "$x=1; print $x" syntax error at -e line 1, near "=" Execution of -e aborted due to compilation errors. 117 Friday, 06 August 2010 117
  151. 151. double quotes Beware of $VARIABLES on theconsumed by the allow the shell to shell, before Perl interpolate command line can see it variables $ perl -le "$x=1; print $x" syntax error at -e line 1, near "=" Execution of -e aborted due to compilation errors. consumed by the shell, before Perl can see it 117 Friday, 06 August 2010 117
  152. 152. double quotes Beware of $VARIABLES on the consumed by the allow the shell to shell, before Perl interpolate command line can see it variables $ perl -le "$x=1; print $x" syntax error at -e line 1, near "=" Execution of -e aborted due to compilation errors. consumed by the shell, before Perl $ perl -le '$x=1; print $x' can see it 1 117 Friday, 06 August 2010 117
  153. 153. double quotes Beware of $VARIABLES on the consumed by the allow the shell to shell, before Perl interpolate command line can see it variables $ perl -le "$x=1; print $x" syntax error at -e line 1, near "=" Execution of -e aborted due to compilation errors. consumed by the shell, before Perl $ perl -le '$x=1; print $x' can see it 1 single quotes the consumed by prevent variable shell, before Perl consumption by can shell it the see117 Friday, 06 August 2010 117
  154. 154. advanced command line techniques combining column names 118 Friday, 06 August 2010 118
  155. 155. combining column names $ mysql -N -e 'desc world.City' +-------------+----------+----+-----+ | ID | int(11) | NO | PRI | | Name | char(35) | NO | | | CountryCode | char(3) | NO | | | District | char(20) | NO | | | Population | int(11) | NO | | +-------------+----------+----+-----+ 119 Friday, 06 August 2010 119
  156. 156. combining column names # Expected result: # SELECT CONCAT_WS('|', # ID,Name,CountryCode,District, # Population) from City 120 Friday, 06 August 2010 120
  157. 157. combining column names (I) echo "select concat_ws('|', $(mysql -N -e 'desc world.City' | perl -lane 'push @f, $F[0];' -e 'END{print join ",", @f}')) from world.City" select concat_ws('|', ID,Name,CountryCode,District,Populatio n) from world.City 121 Friday, 06 August 2010 121
  158. 158. combining column names (II) echo "select concat_ws('|', $( mysql -NB -e 'select group_concat (column_name) from information_schema.columns where table_name="City" and table_schema="world"')) from world.City" select concat_ws('|', ID,Name,CountryCode,District,Populatio n) from world.City 122 Friday, 06 August 2010 122
  159. 159. combining column names (IIa) echo "select concat_ws('|', $( mysql -NB -e 'select group_concat (column_name) from information_schema.columns where table_name="Country" and table_schema="world"')) from world.Country" select concat_ws('|', Code,Name,Continent,Region,SurfaceArea,Inde pYear,Population,LifeExpectancy,GNP,GNPOld, LocalName,GovernmentForm,HeadOfState,Capita l,Code2) from world.Country 123 Friday, 06 August 2010 123
  160. 160. piping the result back to mysql echo "select concat_ws('|', $( mysql -NB -e 'select group_concat (column_name) from information_schema.columns where table_name="Country" and table_schema="world"')) as line from world.Country" | mysql | head -n 4 line AFG|Afghanistan|Asia|Southern and Central Asia|652090.00|1919| 22720000|45.9|5976.00|Afganistan/Afqanestan|Islamic Emirate| Mohammad Omar|1|AF NLD|Netherlands|Europe|Western Europe|41526.00|1581|15864000| 78.3|371362.00|360478.00|Nederland|Constitutional Monarchy| Beatrix|5|NL ANT|Netherlands Antilles|North America|Caribbean|800.00|217000| 74.7|1941.00|Nederlandse Antillen|Nonmetropolitan Territory of 124 The Netherlands|Beatrix|33|AN Friday, 06 August 2010 124
  161. 161. advanced command line techniques $_ dumping data 125 Friday, 06 August 2010 125
  162. 162. advanced command line techniques $_ dumping data 125 Friday, 06 August 2010 125
  163. 163. dumping a tab delimited data set # the naive way $ mysql -e "SELECT * into outfile '/tmp/city_data' from world.City" # it works only if: # - /tmp/city_data does not exist # - /tmp is world writable 126 Friday, 06 August 2010 126
  164. 164. dumping a tab delimited data set $ mysql -B -N -e "SELECT * from world.City" > /tmp/city_data # it works always # if you omit the "-N" it also # includes a header 127 Friday, 06 August 2010 127
  165. 165. dumping a CSV $ mysql -B -e "SELECT * from world.City" | perl -F"t" -lane 'print join ",", map {s/"/""/g; /^[d.]+$/ ? $_ : qq("$_")} @F ' > /tmp/city_data 128 Friday, 06 August 2010 128
  166. 166. dumping a CSV $ mysql -B -e "SELECT * from world.City" | perl -F"t" -lane 'print join ",", splits the line by tabs, instead map {s/"/""/g; of the default whitespace /^[d.]+$/ ? $_ : qq("$_")} @F ' > /tmp/city_data 128 Friday, 06 August 2010 128
  167. 167. advanced command line techniques script it 129 Friday, 06 August 2010 129
  168. 168. use the mysql.sh script  The companion code for this tutorial  http://tinyurl.com/cli-tut-code-2010  It will save you much typing  But use it at your own risk! 130 Friday, 06 August 2010 130
  169. 169. Usage • download the script • load it $ source mysql.sh 131 Friday, 06 August 2010 131
  170. 170. mysql.sh functions • my_batch "query" • returns the result of the query. • Without header • Without box $ my_batch "SELECT * FROM test.t1" 1 aaa 2 bbb 3 ccc 132 Friday, 06 August 2010 132
  171. 171. mysql.sh functions • my_value "what" • returns a single value • e.g. $ my_value "version()" 5.1.49 $ my_value "@@server_id" 1 133 Friday, 06 August 2010 133
  172. 172. mysql.sh functions • my_var "what" • returns the value of a given variable • e.g. $ my_var port 3306 $ my_var datadir /usr/local/mysql/data 134 Friday, 06 August 2010 134
  173. 173. mysql.sh functions • my_status "what" • returns the value of a given status item • e.g. $ my_status com_select 2653 $ my_status connections 452 135 Friday, 06 August 2010 135
  174. 174. mysql.sh functions • my_dump_dataset "query" [header] • returns the values of a given query separated by tabs • if a header is requested, it will be part of the result • e.g. $ my_dump_dataset "select * from world.City" | head -n 5 1 Kabul AFG Kabol 1780000 2 Qandahar AFG Qandahar 237500 3 Herat AFG Herat 186800 4 Mazar-e-Sharif AFG Balkh 127800 136 5 Amsterdam NLD Noord-Holland 731200 Friday, 06 August 2010 136
  175. 175. mysql.sh functions • my_csv "query" [separator] • returns the values of a given query separated by commas, or by a given separator • e.g. $ my_csv 'select * from world.City' | head -n 3 "ID","Name","CountryCode","District"," Population" 1,"Kabul","AFG","Kabol",1780000 2,"Qandahar","AFG","Qandahar",237500 137 Friday, 06 August 2010 137
  176. 176. inside the command line client handling :-) @ @ $_ output 138 Friday, 06 August 2010 138
  177. 177. pager mysql> SELECT * FROM world.City; rec … rec … rec … 4000 + records unreadable 139 Friday, 06 August 2010 139
  178. 178. pager mysql> pager less PAGER set to 'less' mysql> SELECT * FROM world.City; # [ records in a manageable window ] mysql> nopager PAGER set to stdout # back to normal 140 Friday, 06 August 2010 140
  179. 179. More pager tricks mysql> pager vim - # like 'less', but you can also edit # the results and eventually save # them 141 Friday, 06 August 2010 141
  180. 180. More pager tricks mysql> pager md5sum PAGER set to 'md5sum' mysql> select * from City; 5d17ffa50d6da200dee621823ade2543 - 4079 rows in set (0.01 sec) # produce a CRC for the result set # instead of displaying it. # Useful for data comparison. # (Thanks to Baron Schwartz 142 Friday, 06 August 2010 142
  181. 181. Comparing data sets with md5sum mysql> select * from City; 5d17ffa50d6da200dee621823ade2543 - mysql> select * from City order by ID; 5d17ffa50d6da200dee621823ade2543 - mysql> select * from City order by name; 5ab980f5b69a6e65d3b8cd5a95ec94e0 - mysql> select * from City order by District; 2bf37cbe91b3ffeaebc8c4b6084b8784 - 143 Friday, 06 August 2010 143
  182. 182. Comparing data sets with md5sum mysql> select * from City; 5d17ffa50d6da200dee621823ade2543 - mysql> select * from City order by ID; 5d17ffa50d6da200dee621823ade2543 - mysql> select * from City order by name; 5ab980f5b69a6e65d3b8cd5a95ec94e0 - mysql> select * from City order by District; 2bf37cbe91b3ffeaebc8c4b6084b8784 - 143 Friday, 06 August 2010 143
  183. 183. More pager tricks mysql> pager cat > /tmp/t1.txt mysql> select "one" union select "two"; mysql> pager cat > /tmp/t2.txt mysql> select "one" union select "TWO"; mysql> nopager mysql> ! vimdiff -o /tmp/t[12].txt # see the difference between two # datasets 144 Friday, 06 August 2010 144
  184. 184. 145 Friday, 06 August 2010 145
  185. 185. logging sessions mysql> tee mylog.txt # logs everything that you see on # the screen. # your commands, the results, # warnings, error messages, and so on. mysql> notee # back to normal 146 Friday, 06 August 2010 146
  186. 186. Updated presentation slides  you will find an up to date copy of these slides at:  http://tinyurl.com/shoot-cl-2010  http://tinyurl.com/shoot-cl-full-2010 Friday, 06 August 2010 147
  187. 187. read more  http://datacharmer.blogspot.com Friday, 06 August 2010 148
  188. 188. THANKS! ? Question time Comments on Twitter: @datacharmer This work is licensed under the Creative Commons Attribution-Share Alike 3.0 Unported License. Friday, 06 August 2010 149
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×