Don't be a GUI rookie forever - Be in command! (line)

Don’t be a rookie forever.
Be in command! (line)

Giuseppe Maxia
@datacharmer
$
This work is licensed under the Creative Commons Attribution-Share Alike 3.0 Unported License.
Sunday, August 21, 11

about me -Giuseppe Maxia
§ a.k.a. The Data Charmer
§ QA Director, Continuent, Inc
§ Long time hacking with database features
§ Formerly, database consultant, designer, coder.
§ A passion for QA
§ An even greater passion for open source
§ ... and community
§ Passionate blogger
§ http://datacharmer.blogspot.com


What this tutorial is about
§ Providing tools and tips for command line users;
§ Learn a few tricks to make you achieve more and type
less.


What this tutorial is about
§ Providing tools and tips for command line users;
§ Learn a few tricks to make you achieve more and type
less.
§ Making GUI enthusiasts green with envy;
bar

foo
baz


This tutorial
§ A quick guide at the command line usage
§ A practical case for database admins
§ but ...
§ Useful for all


Problem
§ One document contains one date in 'mm/dd/yyyy'
format
§ Need to convert the date into the 'yyyy-mm-dd' format
(or dd.mm.yyyy)

GUI USER SOLUTION
§ open the document with a word processor
§ change the date

§ Command line USER SOLUTION
§ Probably the same


Problem
§ One document contains many dates in 'mm/dd/yyyy'
format
§ Need to convert the dates into the 'yyyy-mm-dd'
format (or dd.mm.yyyy)

GUI USER SOLUTION
§ open the document with an advanced word processor
§ hope it supports regular expressions

§ Command line USER SOLUTION
§ Use vi or emacs. Do a search-and-replace operation
§ Or use the next page solution

Problem
§ Many documents contain many dates in 'mm/dd/yyyy'
format
§ Need to convert the dates in all documents into the
'yyyy-mm-dd' format (or dd.mm.yyyy)

GUI USER SOLUTION
§ Realizes that it will take days to do this manually
§ Scream

§ Command Line USER SOLUTION
§ uses sed or perl to do the replacement in one line


the command line

?
WHY
8


An educational story
1985-90
1991
1993
1994
1995-97
1998-99
2000-2001
2001-2006
2007-2009
2010-
9


1985-90 • MS-DOS
1991
1993
1994
1995-97
1998-99
2000-2001
2001-2006
2007-2009
2010-
9


1985-90 • MS-DOS
1991 • Windows 3.0
1993
1994
1995-97
1998-99
2000-2001
2001-2006
2007-2009
2010-
9


1985-90 • MS-DOS
1991 • Windows 3.0
1993 • Windows 3.1
1994
1995-97
1998-99
2000-2001
2001-2006
2007-2009
2010-
9


1985-90 • MS-DOS
1991 • Windows 3.0
1993 • Windows 3.1
1994 • Windows NT
1995-97
1998-99
2000-2001
2001-2006
2007-2009
2010-
9


1985-90 • MS-DOS
1991 • Windows 3.0
1993 • Windows 3.1
1994 • Windows NT
1995-97 • Windows 95
1998-99
2000-2001
2001-2006
2007-2009
2010-
9


1985-90 • MS-DOS
1991 • Windows 3.0
1993 • Windows 3.1
1994 • Windows NT
1995-97 • Windows 95
1998-99 • Windows 98
2000-2001
2001-2006
2007-2009
2010-
9


1985-90 • MS-DOS
1991 • Windows 3.0
1993 • Windows 3.1
1994 • Windows NT
1995-97 • Windows 95
1998-99 • Windows 98
2000-2001 • Windows 2000
2001-2006
2007-2009
2010-
9


1985-90 • MS-DOS
1991 • Windows 3.0
1993 • Windows 3.1
1994 • Windows NT
1995-97 • Windows 95
1998-99 • Windows 98
2000-2001 • Windows 2000
2001-2006 • Windows XP
2007-2009
2010-
9


1985-90 • MS-DOS
1991 • Windows 3.0
1993 • Windows 3.1
1994 • Windows NT
1995-97 • Windows 95
1998-99 • Windows 98
2000-2001 • Windows 2000
2007-2009 • Windows Vista
2010-
9


1985-90 • MS-DOS
1991 • Windows 3.0
1993 • Windows 3.1
1994 • Windows NT
1995-97 • Windows 95
1998-99 • Windows 98
2000-2001 • Windows 2000
2010- • Windows 7
9


1985-90 • MS-DOS • Unix shell
1991 • Windows 3.0
1993 • Windows 3.1 • Linux
1994 • Windows NT
1995-97 • Windows 95
1998-99 • Windows 98
2000-2001 • Windows 2000
2001-2006 • Windows XP • Mac OSX
2010- • Windows 7
9


Why the command line
• portability
• efficiency
• fun

10


the command line (for rookies)

||||
NO
PANIC 11


Command line panic (for rookies)

$_

Command line panic (for rookies)

$_
NOW WHAT?

the command line (for rookies)

BASICS
13


The shell
• It's a command interpreter
• There are many flavors
> bash
> zsh
> ksh

14


MySQL command line client
• called "mysql"
> Takes SQL commands
> sends them to the server
> gets results

• but it can do much more ...

15


other tools
• Shell scripts
• Aliases
• Perl
• awk
• sort
• …. and a lot of Unix little helper

16


other tools
• Shell scripts
• Aliases
• Perl which could replace everything else, of course
• awk
• sort
• …. and a lot of Unix little helper

16


in case it wasn't clear (yeah, for
rookies)

17


rookies)
• command line actions are

17


rookies)
> commands (the name of the application to invoke)

17


rookies)
> parameters (optional arguments for the application)

17


rookies)
> and the ENTER key

17


rookies)
> and the ENTER key

Enter

17


The Unix philosophy (1)
• One tool for every feature
> make one tool that does one thing well
> for many features, make many tools
> make them work together (see next slide)

18


The Unix philosophy (2)
• To make several tool cooperate:
> pipe the output of one tool to the input of another one.
> for more actions, pipe more tools

$ sort < filename | uniq | nl

19


Unix pipes

20


Unix pipes

application < input

20


Unix pipes

application < input

application > output

20


Unix pipes

application < input


application < input > output

20


Unix pipes

application < input



application | application

20


Unix pipes

application < input




output input
20


Unix pipes = parallel processing

21




21




output

21




output input

21


Who gets what
• shell
• application
• database server
• pager

22


who gets what

shell prompt
$ mysql

mysql client

mysql server
23


who gets what
any host mysql server

shell
mysql client

24


who gets what
$ mysql -u username -ppassword

First the application
then the server

25


who gets what
$ mysql -u username -ppassword
-e "select VERSION()"

goes to the client, then the
server

goes to the server

26


who gets what
mysql> help
mysql> pager less goes to the client only
mysql> select 1;

goes to the server

27


the command line

readline
28


readline
• Common component
> shell
> mysql client
> many CLI applications

29


Main cursor movements
Ctrl A Start/End of line
Ctrl E
Meta B Back/Forward one word
Meta F
previous/next command

Ctrl L clear screen and take
current line on top
Ctrl R Search history
30


Changing text
Meta U Uppercase word
Meta L Lowercase word
Ctrl D delete word
Ctrl K delete to end of line
Ctrl - undo

for more tips:
$ man readline 31


Some conventions in this talk
$ command
at the shell

mysql> command
at the mysql prompt

32


The command line friends

#!
handy
tools
33

your best friends at the command line
• man
• ls
• cd
• find
• grep
• date
• echo
• pwd
• mv
34
• cp

man {command_name}
• Tells you how to use a command

man ls

35


ls [directory]
• lists contents of current [or given] directory

ls

ls /some/path

ls *.txt

36


cd [directory]
• change directory to given path
• takes to $HOME directory if no path is given

cd /some/path

cd ..

37


find {directory} [conditions]
• finds files in the given directory that satisfy conditions

find $HOME/docs -name "*mouse*"

38


grep {pattern} filename
• finds pattern matching rows in filename

grep Munich *.txt

grep "Munich|Berlin" *.txt

grep "[0-9][0-9]:[0-9][0-9]" a.txt

39


date [layout]
• displays the current date and time, with optional layout

date
Tue Aug 16 08:33:04 CEST 2011

date '+%Y-%m-%d'
2011-08-16

40


echo "message"
• displays the given message and/or variables

echo "hi"
hi

GREETINGS='Hello, there!'
echo $GREETINGS
Hello, there!

41


pwd
• displays the current path (Print Working Directory

pwd
/home/gmax/test_dir

42


mv old_file_name new_file_name
• moves a file from one place to another
• renames a file

mv ugly.txt /some/other/path
mv *.txt /storage/texts

mv ugly.txt beautiful.txt

43


cp file_name destination
• copies a file from one place to another
• creates a copy of a file

cp ugly.txt /some/other/path
cp *.txt /storage/texts

cp ugly.txt beautiful.txt

44


other important friends at the
command line
• rm
• mkdir
• rmdir
• cat
• sed
• strings
• wc
• head
• tail
• more/less 45

• sort

rm file_name
• removes a file

rm ugly.txt
rm *.txt

### DANGER!!!!
rm -r *
rm -rf *

46


mkdir/rmdir directory
• creates or removes a directory

mkdir mydir

rmdir mydir

47


cat
• shows the contents of a file

cat addresses.txt

48


sed [option] file
• performs editing on a file

sed -e 's/Ship/Airplane/' travel.txt

49


strings filename
• extracts printable text from a file

strings /bin/ls

50


wc filename
• counts words (and characters and lines) in a file

wc .bashrc
72 164 2842 .bashrc
# (lines, words, characters)

51


head/tail filename
• display the first or the last lines of a file

head long_file.txt
head -n 5 long_file.txt

tail long_file.txt
tail -n 1 long_file.txt

52


more/less filename
• display a file with scrollable pages

# old: inefficient
more filename.txt

# new: preferred
less filename.txt

53


sort filename
• displays the ordered contents of a filename

sort unordered.txt

sort unordered.txt > sorted.txt

54


#!
mixing
languages
55


This is a story about
languages


you got to understand,
Bernie, it's not personal.
It's just that you are
obsolete
We use
language
to talk to
computers


10 print "hello"
We use 20 goto 10
languages
to talk to
computers


I am a polyglot


An imperfect
polyglot, actually


I speak
Italian


I speak
Italian

with a clear
Sardinian
accent


I speak
English


I speak
English

with a distinct
Italian accent


I speak
Perl


I speak
Perl

with a hint of a C
accent


I speak
French
and
Spanish


I speak
French
and
Spanish
but since I use
them very little,
they have melted
with Italian and
English

I speak
#!/bin/bash
Bash


I speak
#!/bin/bash
Bash

with a strong
Perl accent


I speak fluent Italian

La donzelletta vien dalla
campagna, in sul calar del
sole, col suo fascio dell'erba
e reca in mano un mazzolino
di rose e viole, onde, siccome
suole, ornare ella si appresta ...


However, Sardinian comes handy sometimes

Maissavida
ti frimisti! (*)

(*) May you never stop

I speak fluent English

When in the Course of human events, it becomes
necessary for one people to dissolve the political bands
which have connected them with another, and to assume
among the powers of the earth, the separate and equal
station to which the Laws of Nature and of Nature's God
entitle them, a decent respect to the opinions of mankind
requires that they should declare the causes which impel
them to the separation.


I speak fluent English

When in the Course of human events, it becomes
necessary for one people to dissolve the political bands
which have connected them with another, and to assume
among the powers of the earth, the separate and equal
station to which the Laws of Nature and of Nature's God
entitle them, a decent respect to the opinions of mankind
requires that they should declare the causes which impel
them to the separation.

hmm.
wrong
ﬂag


But sometimes Italian is still necessary

barista al fresco pasta
maestro espresso tempo
vendetta soprano ﬁnale piano
dilettante cappuccino stucco
stiletto virtuoso ﬁasco presto lotto
pizza bravo lasagna
studio spaghetti
stanza


And sometimes Italian is needed

I tuoi perversi
defunti!(*)

(*) Your wicked ancestors!

Although Sardinian is still effective here

Ancu ti currullidi
sa giustizia (*)

(*) May the rule of law chase you

I speak Perl fluently

#!/usr/bin/perl
use strict;
use warnings;
use File::Find;

while (my $line = <stdin>){
# …


Sometimes, I need help from C

use Inline C;
print "9 + 16 = ", add(9, 16), "n";

__END__
__C__
int add(int x, int y) {
return x + y;
}


Most of my work at the command line is Bash

$ for D in $HOME/*/; do du -sh $D;
> done > /tmp/occupancy

#!/bin/bash

And sometimes, a pinch of Perl can help

$ perl -pe
's{(dd)-(dd)-(dddd)}{$3-$1-$2}g'
< rawfile.txt > betterfile.txt

#!/bin/bash

And then on top of all this, I need to speak SQL

$ mysql -e
'SELECT COUNT(*) FROM dbname.tbl_name
WHERE col_name = 1 and other_col="A" '


I wish I could say that my SQL has a Perlish
accent

$ mysql -e

use strict;
use DBI;?


I wish I could say that my SQL has a Perlish
accent

$ mysql -e

but it wouldn't be true


I could curse in Perl

$ perl -pe
's{(dd)-(dd)-(ddd
d)}{$3-$1-$2}g'
< rawfile.txt >
betterfile.txt (*)

But it would not
get me far.
(*) already translated

Then


Then

I will need to mix languages.


Then


This talk will show exactly that.


Then



Mixing languages.


Then



Mixing languages.
Yes.


Then



Mixing languages.
Yes.

$ at the command line

the Bash shell

#!
bash
81


components
• commands
• aliases
• variables
• functions

82


commands
$ echo "something"
something

83


aliases
$ alias mydate='mysql -e "select NOW()"'
$ mydate
+---------------------+
| now() |
+---------------------+
| 2010-04-12 07:14:19 |
+---------------------+

84


aliases
$ alias myself='mysql'
$ myself -e "select NOW()"'
+---------------------+
| now() |
+---------------------+
| 2010-04-12 07:14:19 |
+---------------------+

85


variables
$ MYBATCH='mysql -N -B'
$ MYQUERY='select curdate()'
$ $MYBATCH -e "$MYQUERY"
2010-04-12

86


nesting variables
# t.sh
CMD="SELECT TABLE_NAME"
TBL='information_schema.tables'
WHC1="TABLE_SCHEMA='mysql'"
WHC2="TABLE_NAME like 'u%' "
Q1="$CMD FROM $TBL"
Q2="WHERE $WHC1 AND $WHC2"
Q="$Q1 $Q2"
echo $Q
echo $Q | mysql
87


nesting variables
$ sh t.sh
SELECT TABLE_NAME FROM
information_schema.tables WHERE
TABLE_SCHEMA='mysql' AND TABLE_NAME like
'u%'
TABLE_NAME
user

88


the mysql command line client

mysql>

89


SQL commands from a file
$ mysql < filename.sql
mysql> source filename.sql

• executes the commands contained in
filename.sql
• commands can be queries understood
by the server
• and commands for the client

90


SQL commands at the command line
$ application | mysql
$ echo "SELECT 1" | mysql
$ mysql -e "SELECT 1"

• executes the commands from standard
input
• commands can be queries understood
by the server
• and commands for the client

91


batch mode
$ echo "select curdate()" | mysql
curdate()
2010-04-12

92


table mode
$ echo "select curdate()" | mysql -t
+------------+
| curdate() |
+------------+
| 2010-04-12 |
+------------+

93


table mode
$ mysql -e "select curdate()"
+------------+
| curdate() |
+------------+
| 2010-04-12 |
+------------+

94


Forcing batch mode
mysql -B -e 'select curdate()'
curdate()
2010-04-12

95


Forcing batch mode - no header
mysql -N -B -e 'select curdate()'
2010-04-12

ANT!
PO RT
IM

96


Shell + mysql
$ echo "The current server's date is
`mysql -N -B -e 'select curdate()'`."

$ echo "The current server's date is
$(mysql -N -B -e 'select
curdate()')."

97


advanced command line techniques
((((((
(:
@ @
-

$_

loops
98


loops basics
$ for N in 1 2 3
> do
> echo $N
> done
1
2
3

99


loops basics
$ for N in 1 2 3 ; do echo $N ; done
1
2
3

100


loops basics
$ for N in $(seq 1 3) ;
do echo "this is number $N" ;
done

this is number 1
this is number 2
this is number 3

101


loops from file
$ head /usr/share/dict/words
A
a
aa
aal
aalii
aam
Aani
aardvark
aardwolf
102


loops from file
$ for U in
$(head /usr/share/dict/words ) ; do
perl -le 'print uc shift' $U ; done
A
A
AA
AAL
AALII
AAM
AANI
103


loops from mysql results
$ mysql -B -N -e 'show tables from
world'
City
Country
CountryLanguage

104


loops from mysql results
$ for T in $(mysql -B -N -e 'show
tables from world') ;
> do echo "CREATE TABLE world2.$T
LIKE world.$T;" ; done

CREATE TABLE world2.City LIKE
world.City;
CREATE TABLE world2.Country LIKE
world.Country;
CREATE TABLE world2.CountryLanguage
LIKE world.CountryLanguage;
105


piping loops back into mysql
$ for T in $(mysql -B -N -e 'show
tables from world') ;
> do echo "CREATE TABLE world2.$T
LIKE world.$T;" ; done
> | mysql -vv

106



nested

:)
@ @

-
$_

#!
#!
#!
#!

loops
#!
#!

107


pseudocode
for each database
do
for each table
do
print the table name

110


actual code
alias mybatch='mysql -B -N'
for D in $(mybatch -e "SHOW SCHEMAS")
do
for T in
$(mybatch -e "SHOW TABLES FROM $D")
do
echo "$D.$T"
done
done

111


$HOME

options
files 112


Default option files
/etc/my.cnf
/etc/mysql/my.cnf (Debian, Ubuntu)
$BASEDIR/my.cnf
$HOME/.my.cnf

113


Custom option files
$ mysql --no-defaults
$ mysql --defaults-file=filename
$ mysql --defaults-extra-file=filename

114


Option files sections
[mysqld]
# option for server

[client]
# option for any mysql client

[mysql]
# options specific to 'mysql' CLI app

115


Option files example

[client]
user=myname
password=mypassword

[mysql]
prompt="mysql (h) {u} [d] > "

116


Option files: caution
# protect your option files!

$ chmod 600 $HOME/.my.cnf

$ ls -l .my.cnf
-rw------- 1 [...] 10:04 .my.cnf

117


prompt
mysql> prompt gm (h) {u} [d] >
PROMPT set to 'gm (h) {u} [d] > '
gm (localhost) {root} [(none)] > use test
Database changed
gm (localhost) {root} [test] >

# h -> host
# u -> user
# d -> database

118


((((((
(:

@ @
-

#!

aliases
119


An useful alias
$ alias mysql_as_root=
'mysql --
defaults_file=~/.my_as_root.cnf'

mysql_as_root -e "select user()"
+----------------+
| user() |
+----------------+
| root@localhost |
+----------------+
120


An useful alias
#.my_as_root.cnf'
[client]
user=root
password=your_mysql_root_password

[mysql]
prompt="### [h] {u} (d) > "

121


Another useful alias
alias mybatch='mysql -B -N '

mybatch -e "select version()"
5.1.45-log

122


( )
((((((
(:
@ @
-

#!

functions
123


functions
• like aliases, but with parameters

124


sample function
function grant_new {
mysql_as_root -e
"GRANT $1 on $2 to $3 identified
by "$4" "
}

$ grant_new SELECT "world.*" john
john_secret

125


sample function
$ mysql_as_root
-e 'show grants for johnG'
*** 1. row ***
Grants for john@%: GRANT USAGE ON *.*
TO 'john'@'%' IDENTIFIED BY PASSWORD
'*73A88CE13374128844E3ABB500F439A9C02C
53A3'
*** 2. row ***
Grants for john@%: GRANT SELECT ON
`world`.* TO 'john'@'%'

126



$_ getting
query
details127


which query is being executed?
# wrong.sql
# this script has one error
use test ;
drop table if exists t1 ;
create table t1 (id int);
insert into t1 values (a);

128


$ mysql < wrong.sql
ERROR 1054 (42S22) at line 4: Unknown
column 'a' in 'field list'

# WHERE IS THE ERROR?

129


$ mysql -v < wrong.sql
--------------
drop table if exists t1
--------------
--------------
create table t1 (id int)
--------------
--------------
insert into t1 values (a)
--------------
ERROR 1054 (42S22) at line 4: Unknown column
'a' in 'field list'
130


getting execution times
$ mysql -vv -e
'select rating,count(*) from sakila.film
group by rating'
+--------+----------+
| rating | count(*) |
+--------+----------+
| G | 178 |
| PG | 194 |
| PG-13 | 223 |
| R | 195 |
| NC-17 | 210 |
+--------+----------+
5 rows in set (0.00 sec) 131


getting execution times (batch mode)
$echo 'select rating,count(*) from sakila.film group
by rating' | mysql -vv
--------------
select rating,count(*) from sakila.film group by
rating
--------------

rating count(*)
G 178
PG 194
PG-13 223
R 195
NC-17 210
5 rows in set

132


getting execution times (batch mode)
$ echo 'select rating,count(*) from sakila.film group
by rating' | mysql -vvv
--------------
select rating,count(*) from sakila.film group by
rating
--------------
+--------+----------+
| rating | count(*) |
+--------+----------+
| G | 178 |
| PG | 194 |
| PG-13 | 223 |
| R | 195 |
| NC-17 | 210 |
+--------+----------+
133
5 rows in set (0.00 sec)


getting
_$

distinct
results134


$ mysql -NB -e "show variables like 'port'"
port 3306

# no header, batch mode
# still two values

136


$ mysql -NB
-e "show variables like 'port'"
| awk '{print $2}'

3306

$ mysql -NB
-e "show variables like 'port'"
| perl -lane 'print $F[1]'

3306
# filtering the wanted value
137


getting a single value: is replication
working?
$ mysql
-e "show slave status G"
| grep -i slave_io_running

Slave_IO_Running: Yes

# too many things.
# We only want a 'yes' or 'no'

138


getting a single value: is replication
working?
$ mysql
-e "show slave status G"
| grep -i "slave_io_running"
| perl -lane 'print $F[1]'

Yes

# yes!

139


What's perl -lane?

140


What's perl -lane? perl -lane
must
turn perl

140


Perl command line must-know facts (1)
perl -e 'print "hellon"'
# executes perl code

141


perl -ne 'print if /â/'
# starts a loop from standard input
# and executes the perl code for each
# iteration

perl -MO=Deparse -ne 'print if /â/'
LINE: while (defined($_ = <ARGV>)) {
print $_ if /â/;
}

142


perl -pe 's/(âw+)/$1/'
# like -ne, but prints each line

$ perl -MO=Deparse -pe 's/(âw+)/$1/'
LINE: while (defined($_ = <ARGV>)) {
s/(âw+)/$1/;
}
continue {
print $_;
}
143


perl -le 'print "hello"'
# strips newlines from input and adds
# them to output

$ perl -e 'print "hello"'
hello$

$ perl -le 'print "hello"'
hello
$
144


perl -lane 'print $F[0]'
# splits input into @F array

$ echo "a b c" |
b

$ echo "a b c" |
c
145


Perl and shell contention for $

#!/bin/bash

Take the $$$ and run
146


Beware of $VARIABLES on the
command line
$ perl -le "$x=1; print $x"
syntax error at -e line 1, near "="
Execution of -e aborted due to
compilation errors.

147


double quotes
Beware of $VARIABLES on theconsumed by the
allow the shell to
shell, before Perl
interpolate
command line can see it
variables
compilation errors.

147


double quotes
Beware of $VARIABLES on theconsumed by the
allow the shell to
shell, before Perl
interpolate
variables
compilation errors.
consumed by the
shell, before Perl
can see it

147


double quotes
Beware of $VARIABLES on the consumed by the
allow the shell to
shell, before Perl
interpolate
variables
compilation errors.
consumed by the
shell, before Perl
$ perl -le '$x=1; print $x' can see it
1

147


double quotes
Beware of $VARIABLES on the consumed by the
allow the shell to
shell, before Perl
interpolate
variables
compilation errors.
consumed by the
shell, before Perl
$ perl -le '$x=1; print $x' can see it
1
single quotes the
consumed by
prevent variable
shell, before Perl
consumption by
can shell it
the see147


combining
column
names
148


combining column names
$ mysql -N -e 'desc world.City'
+-------------+----------+----+-----+
| ID | int(11) | NO | PRI |
| Name | char(35) | NO | |
| CountryCode | char(3) | NO | |
| District | char(20) | NO | |
| Population | int(11) | NO | |
+-------------+----------+----+-----+

149


combining column names
# Expected result:
# SELECT CONCAT_WS('|',
# ID,Name,CountryCode,District,
# Population) from City

150


combining column names (I)
echo "select concat_ws('|',
$(mysql -N -e 'desc world.City' |
perl -lane 'push @f, $F[0];'
-e 'END{print join ",", @f}'))
from world.City"
select concat_ws('|',
ID,Name,CountryCode,District,Populatio
n) from world.City

151


combining column names (II)
echo "select concat_ws('|', $(
mysql -NB -e 'select
group_concat(column_name) from
information_schema.columns where
table_name="City" and
table_schema="world"')) from
world.City"
ID,Name,CountryCode,District,Populatio
n) from world.City
152


combining column names (IIa)
table_name="Country" and
table_schema="world"')) from
world.Country"
Code,Name,Continent,Region,SurfaceArea,Inde
pYear,Population,LifeExpectancy,GNP,GNPOld,
LocalName,GovernmentForm,HeadOfState,Capita
l,Code2) from world.Country 153


piping the result back to mysql
table_name="Country" and
table_schema="world"')) as line from
world.Country" | mysql | head -n 4
line
AFG|Afghanistan|Asia|Southern and Central Asia|652090.00|1919|
22720000|45.9|5976.00|Afganistan/Afqanestan|Islamic Emirate|
Mohammad Omar|1|AF
NLD|Netherlands|Europe|Western Europe|41526.00|1581|15864000|
78.3|371362.00|360478.00|Nederland|Constitutional Monarchy|
Beatrix|5|NL
ANT|Netherlands Antilles|North America|Caribbean|800.00|217000|
74.7|1941.00|Nederlandse Antillen|Nonmetropolitan Territory of
154
The Netherlands|Beatrix|33|AN


$_

dumping
data
155


dumping a tab delimited data set
# the naive way
$ mysql
-e "SELECT *
into outfile '/tmp/city_data'
from world.City"

# it works only if:
# - /tmp/city_data does not exist
# - /tmp is world writable

156


dumping a tab delimited data set
$ mysql -B -N
-e "SELECT * from world.City"
> /tmp/city_data

# it works always

# if you omit the "-N" it also
# includes a header

157


dumping a CSV
$ mysql -B
| perl -F"t" -lane
'print join ",",
map {s/"/""/g;
/^[d.]+$/ ? $_ : qq("$_")}
@F '
> /tmp/city_data

158


dumping a CSV
$ mysql -B
| perl -F"t" -lane
'print join ",", splits the line by tabs, instead
map {s/"/""/g; of the default whitespace
/^[d.]+$/ ? $_ : qq("$_")}
@F '
> /tmp/city_data

158



script
it
159


use the mysql.sh script
§ The companion code for this tutorial
§ http://tinyurl.com/cli-tut-code-2010
§ It will save you much typing
§ But use it at your own risk!

160


Usage
• download the script
• load it

$ source mysql.sh

161


mysql.sh functions
• my_batch "query"
• returns the result of the query.
• Without header
• Without box

$ my_batch "SELECT * FROM test.t1"
1 aaa
2 bbb
3 ccc
162


mysql.sh functions
• my_value "what"
• returns a single value

• e.g.
$ my_value "version()"
5.1.49

$ my_value "@@server_id"
1
163


mysql.sh functions
• my_var "what"
• returns the value of a given variable

• e.g.
$ my_var port
3306

$ my_var datadir
/usr/local/mysql/data
164


mysql.sh functions
• my_status "what"
• returns the value of a given status item

• e.g.
$ my_status com_select
2653

$ my_status connections
452
165


mysql.sh functions
• my_dump_dataset "query" [header]
• returns the values of a given query separated by tabs
• if a header is requested, it will be part of the result
• e.g.
$ my_dump_dataset "select * from
world.City" | head -n 5
1 Kabul AFG Kabol 1780000
2 Qandahar AFG Qandahar 237500
3 Herat AFG Herat 186800
4 Mazar-e-Sharif AFG Balkh 127800
166
5 Amsterdam NLD Noord-Holland 731200

mysql.sh functions
• my_csv "query" [separator]
• returns the values of a given query separated by
commas, or by a given separator
• e.g.
$ my_csv 'select * from world.City' |
head -n 3
"ID","Name","CountryCode","District","
Population"
1,"Kabul","AFG","Kabol",1780000
2,"Qandahar","AFG","Qandahar",237500
167


inside the command line client

handling

:-)
@ @

$_

output
168


pager
mysql> SELECT * FROM world.City;

rec …
rec …
rec …

4000 + records
unreadable

169


pager
mysql> pager less
PAGER set to 'less'
mysql> SELECT * FROM world.City;

# [ records in a manageable window ]

mysql> nopager
PAGER set to stdout

# back to normal
170


More pager tricks
mysql> pager vim -
# like 'less', but you can also edit
# the results and eventually save
# them

171


More pager tricks
mysql> pager md5sum
PAGER set to 'md5sum'
mysql> select * from City;
5d17ffa50d6da200dee621823ade2543 -
4079 rows in set (0.01 sec)

# produce a CRC for the result set
# instead of displaying it.
# Useful for data comparison.
# (Thanks to Baron Schwartz
172


Comparing data sets with md5sum
mysql> select * from City;
5d17ffa50d6da200dee621823ade2543 -

mysql> select * from City order by ID;
5d17ffa50d6da200dee621823ade2543 -

mysql> select * from City order by name;
5ab980f5b69a6e65d3b8cd5a95ec94e0 -

mysql> select * from City order by
District;
2bf37cbe91b3ffeaebc8c4b6084b8784 - 173


More pager tricks
mysql> pager cat > /tmp/t1.txt
mysql> select "one" union select
"two";
mysql> pager cat > /tmp/t2.txt
mysql> select "one" union select
"TWO";
mysql> nopager
mysql> ! vimdiff -o /tmp/t[12].txt

# see the difference between two
# datasets
174


logging sessions
mysql> tee mylog.txt

# logs everything that you see on
# the screen.
# your commands, the results,
# warnings, error messages, and so on.

mysql> notee

# back to normal
176


read more
§ http://datacharmer.blogspot.com


THANKS!
?
Question
time
Comments on Twitter:
@datacharmer
This work is licensed under the Creative Commons Attribution-Share Alike 3.0 Unported License.

to do
§ add dead languages
§ comparing Latin and ancient Greek to COBOL and
Pascal
§ add new tricks
§ {x,y} expansion in bash
§ G in MySQL
§


Don't be a GUI rookie forever - Be in command! (line)

Recommended

Recommended

More Related Content

More from Giuseppe Maxia

More from Giuseppe Maxia (20)

Recently uploaded

Recently uploaded (20)

Don't be a GUI rookie forever - Be in command! (line)