Don't be a GUI rookie forever - Be in command! (line)

Don’t be a rookie forever. Be in command! (line) Giuseppe Maxia @datacharmer $ This work is licensed under the Creative Commons Attribution-Share Alike 3.0 Unported License.Sunday, August 21, 11

about me -Giuseppe Maxia § a.k.a. The Data Charmer § QA Director, Continuent, Inc § Long time hacking with database features § Formerly, database consultant, designer, coder. § A passion for QA § An even greater passion for open source § ... and community § Passionate blogger § http://datacharmer.blogspot.comSunday, August 21, 11

What this tutorial is about § Providing tools and tips for command line users; § Learn a few tricks to make you achieve more and type less.Sunday, August 21, 11

What this tutorial is about § Providing tools and tips for command line users; § Learn a few tricks to make you achieve more and type less. § Making GUI enthusiasts green with envy; bar foo bazSunday, August 21, 11

This tutorial § A quick guide at the command line usage § A practical case for database admins § but ... § Useful for allSunday, August 21, 11

Problem § One document contains one date in mm/dd/yyyy format § Need to convert the date into the yyyy-mm-dd format (or dd.mm.yyyy) GUI USER SOLUTION § open the document with a word processor § change the date § Command line USER SOLUTION § Probably the sameSunday, August 21, 11

Problem § One document contains many dates in mm/dd/yyyy format § Need to convert the dates into the yyyy-mm-dd format (or dd.mm.yyyy) GUI USER SOLUTION § open the document with an advanced word processor § hope it supports regular expressions § Command line USER SOLUTION § Use vi or emacs. Do a search-and-replace operation § Or use the next page solutionSunday, August 21, 11

Problem § Many documents contain many dates in mm/dd/yyyy format § Need to convert the dates in all documents into the yyyy-mm-dd format (or dd.mm.yyyy) GUI USER SOLUTION § Realizes that it will take days to do this manually § Scream § Command Line USER SOLUTION § uses sed or perl to do the replacement in one lineSunday, August 21, 11

the command line ? WHY 8Sunday, August 21, 11

An educational story 1985-90 1991 1993 1994 1995-97 1998-99 2000-2001 2001-2006 2007-2009 2010- 9Sunday, August 21, 11

An educational story 1985-90 • MS-DOS 1991 1993 1994 1995-97 1998-99 2000-2001 2001-2006 2007-2009 2010- 9Sunday, August 21, 11

An educational story 1985-90 • MS-DOS 1991 • Windows 3.0 1993 1994 1995-97 1998-99 2000-2001 2001-2006 2007-2009 2010- 9Sunday, August 21, 11

An educational story 1985-90 • MS-DOS 1991 • Windows 3.0 1993 • Windows 3.1 1994 1995-97 1998-99 2000-2001 2001-2006 2007-2009 2010- 9Sunday, August 21, 11

An educational story 1985-90 • MS-DOS 1991 • Windows 3.0 1993 • Windows 3.1 1994 • Windows NT 1995-97 1998-99 2000-2001 2001-2006 2007-2009 2010- 9Sunday, August 21, 11

An educational story 1985-90 • MS-DOS 1991 • Windows 3.0 1993 • Windows 3.1 1994 • Windows NT 1995-97 • Windows 95 1998-99 2000-2001 2001-2006 2007-2009 2010- 9Sunday, August 21, 11

An educational story 1985-90 • MS-DOS 1991 • Windows 3.0 1993 • Windows 3.1 1994 • Windows NT 1995-97 • Windows 95 1998-99 • Windows 98 2000-2001 2001-2006 2007-2009 2010- 9Sunday, August 21, 11

An educational story 1985-90 • MS-DOS 1991 • Windows 3.0 1993 • Windows 3.1 1994 • Windows NT 1995-97 • Windows 95 1998-99 • Windows 98 2000-2001 • Windows 2000 2001-2006 2007-2009 2010- 9Sunday, August 21, 11

An educational story 1985-90 • MS-DOS 1991 • Windows 3.0 1993 • Windows 3.1 1994 • Windows NT 1995-97 • Windows 95 1998-99 • Windows 98 2000-2001 • Windows 2000 2001-2006 • Windows XP 2007-2009 2010- 9Sunday, August 21, 11

An educational story 1985-90 • MS-DOS 1991 • Windows 3.0 1993 • Windows 3.1 1994 • Windows NT 1995-97 • Windows 95 1998-99 • Windows 98 2000-2001 • Windows 2000 2001-2006 • Windows XP 2007-2009 • Windows Vista 2010- 9Sunday, August 21, 11

An educational story 1985-90 • MS-DOS 1991 • Windows 3.0 1993 • Windows 3.1 1994 • Windows NT 1995-97 • Windows 95 1998-99 • Windows 98 2000-2001 • Windows 2000 2001-2006 • Windows XP 2007-2009 • Windows Vista 2010- • Windows 7 9Sunday, August 21, 11

An educational story 1985-90 • MS-DOS • Unix shell 1991 • Windows 3.0 1993 • Windows 3.1 • Linux 1994 • Windows NT 1995-97 • Windows 95 1998-99 • Windows 98 2000-2001 • Windows 2000 2001-2006 • Windows XP • Mac OSX 2007-2009 • Windows Vista 2010- • Windows 7 9Sunday, August 21, 11

Why the command line • portability • efficiency • fun 10Sunday, August 21, 11

the command line (for rookies) |||| NO PANIC 11Sunday, August 21, 11

Command line panic (for rookies) $_Sunday, August 21, 11

Command line panic (for rookies) $_ NOW WHAT?Sunday, August 21, 11

the command line (for rookies) BASICS 13Sunday, August 21, 11

The shell • Its a command interpreter • There are many flavors > bash > zsh > ksh 14Sunday, August 21, 11

MySQL command line client • called "mysql" > Takes SQL commands > sends them to the server > gets results • but it can do much more ... 15Sunday, August 21, 11

other tools • Shell scripts • Aliases • Perl • awk • sort • …. and a lot of Unix little helper 16Sunday, August 21, 11

other tools • Shell scripts • Aliases • Perl which could replace everything else, of course • awk • sort • …. and a lot of Unix little helper 16Sunday, August 21, 11

in case it wasnt clear (yeah, for rookies) 17Sunday, August 21, 11

in case it wasnt clear (yeah, for rookies) • command line actions are 17Sunday, August 21, 11

in case it wasnt clear (yeah, for rookies) • command line actions are > commands (the name of the application to invoke) 17Sunday, August 21, 11

in case it wasnt clear (yeah, for rookies) • command line actions are > commands (the name of the application to invoke) > parameters (optional arguments for the application) 17Sunday, August 21, 11

in case it wasnt clear (yeah, for rookies) • command line actions are > commands (the name of the application to invoke) > parameters (optional arguments for the application) > and the ENTER key 17Sunday, August 21, 11

in case it wasnt clear (yeah, for rookies) • command line actions are > commands (the name of the application to invoke) > parameters (optional arguments for the application) > and the ENTER key Enter 17Sunday, August 21, 11

The Unix philosophy (1) • One tool for every feature > make one tool that does one thing well > for many features, make many tools > make them work together (see next slide) 18Sunday, August 21, 11

The Unix philosophy (2) • To make several tool cooperate: > pipe the output of one tool to the input of another one. > for more actions, pipe more tools $ sort < filename | uniq | nl 19Sunday, August 21, 11

Unix pipes 20Sunday, August 21, 11

Unix pipes application < input 20Sunday, August 21, 11

Unix pipes application < input application > output 20Sunday, August 21, 11

Unix pipes application < input application > output application < input > output 20Sunday, August 21, 11

Unix pipes application < input application > output application < input > output application | application 20Sunday, August 21, 11

Unix pipes application < input application > output application < input > output application | application output input 20Sunday, August 21, 11

Unix pipes = parallel processing 21Sunday, August 21, 11

Unix pipes = parallel processing application | application 21Sunday, August 21, 11

Unix pipes = parallel processing application | application output 21Sunday, August 21, 11

Unix pipes = parallel processing application | application output input 21Sunday, August 21, 11

Who gets what • shell • application • database server • pager 22Sunday, August 21, 11

who gets what shell prompt $ mysql mysql client mysql server 23Sunday, August 21, 11

who gets what any host mysql server shell mysql client 24Sunday, August 21, 11

who gets what$ mysql -u username -ppassword First the application then the server 25Sunday, August 21, 11

who gets what$ mysql -u username -ppassword -e "select VERSION()" goes to the client, then the server goes to the server 26Sunday, August 21, 11

who gets whatmysql> helpmysql> pager less goes to the client onlymysql> select 1; goes to the server 27Sunday, August 21, 11

the command line readline 28Sunday, August 21, 11

readline • Common component > shell > mysql client > many CLI applications 29Sunday, August 21, 11

Main cursor movements Ctrl A Start/End of line Ctrl E Meta B Back/Forward one word Meta F previous/next command Ctrl L clear screen and take current line on top Ctrl R Search history 30Sunday, August 21, 11

Changing text Meta U Uppercase word Meta L Lowercase word Ctrl D delete word Ctrl K delete to end of line Ctrl - undo for more tips: $ man readline 31Sunday, August 21, 11

Some conventions in this talk$ commandat the shellmysql> commandat the mysql prompt 32Sunday, August 21, 11

The command line friends #! handySunday, August 21, 11 tools 33

your best friends at the command line • man • ls • cd • find • grep • date • echo • pwd • mv 34 • cpSunday, August 21, 11

man {command_name} • Tells you how to use a command man ls 35Sunday, August 21, 11

ls [directory] • lists contents of current [or given] directory ls ls /some/path ls *.txt 36Sunday, August 21, 11

cd [directory] • change directory to given path • takes to $HOME directory if no path is given cd /some/path cd .. 37Sunday, August 21, 11

find {directory} [conditions] • finds files in the given directory that satisfy conditions find $HOME/docs -name "*mouse*" 38Sunday, August 21, 11

grep {pattern} filename • finds pattern matching rows in filename grep Munich *.txt grep "Munich|Berlin" *.txt grep "[0-9][0-9]:[0-9][0-9]" a.txt 39Sunday, August 21, 11

date [layout] • displays the current date and time, with optional layout date Tue Aug 16 08:33:04 CEST 2011 date +%Y-%m-%d 2011-08-16 40Sunday, August 21, 11

echo "message" • displays the given message and/or variables echo "hi" hi GREETINGS=Hello, there! echo $GREETINGS Hello, there! 41Sunday, August 21, 11

pwd • displays the current path (Print Working Directory pwd /home/gmax/test_dir 42Sunday, August 21, 11

mv old_file_name new_file_name • moves a file from one place to another • renames a file mv ugly.txt /some/other/path mv *.txt /storage/texts mv ugly.txt beautiful.txt 43Sunday, August 21, 11

cp file_name destination • copies a file from one place to another • creates a copy of a file cp ugly.txt /some/other/path cp *.txt /storage/texts cp ugly.txt beautiful.txt 44Sunday, August 21, 11

other important friends at the command line • rm • mkdir • rmdir • cat • sed • strings • wc • head • tail • more/less 45 • sortSunday, August 21, 11

rm file_name • removes a file rm ugly.txt rm *.txt ### DANGER!!!! rm -r * rm -rf * 46Sunday, August 21, 11

mkdir/rmdir directory • creates or removes a directory mkdir mydir rmdir mydir 47Sunday, August 21, 11

cat • shows the contents of a file cat addresses.txt 48Sunday, August 21, 11

sed [option] file • performs editing on a file sed -e s/Ship/Airplane/ travel.txt 49Sunday, August 21, 11

strings filename • extracts printable text from a file strings /bin/ls 50Sunday, August 21, 11

wc filename • counts words (and characters and lines) in a file wc .bashrc 72 164 2842 .bashrc # (lines, words, characters) 51Sunday, August 21, 11

head/tail filename • display the first or the last lines of a file head long_file.txt head -n 5 long_file.txt tail long_file.txt tail -n 1 long_file.txt 52Sunday, August 21, 11

more/less filename • display a file with scrollable pages # old: inefficient more filename.txt # new: preferred less filename.txt 53Sunday, August 21, 11

sort filename • displays the ordered contents of a filename sort unordered.txt sort unordered.txt > sorted.txt 54Sunday, August 21, 11

#! mixing languages 55Sunday, August 21, 11

This is a story about languagesSunday, August 21, 11

you got to understand, Bernie, its not personal. Its just that you are obsoleteWe uselanguageto talk tocomputersSunday, August 21, 11

10 print "hello"We use 20 goto 10languagesto talk tocomputersSunday, August 21, 11

I am a polyglotSunday, August 21, 11

An imperfect polyglot, actuallySunday, August 21, 11

I speak ItalianSunday, August 21, 11

I speak Italian with a clear Sardinian accentSunday, August 21, 11

I speak EnglishSunday, August 21, 11

I speak English with a distinct Italian accentSunday, August 21, 11

I speak PerlSunday, August 21, 11

I speak Perl with a hint of a C accentSunday, August 21, 11

I speak French and SpanishSunday, August 21, 11

I speak French and Spanish but since I use them very little, they have melted with Italian and EnglishSunday, August 21, 11

I speak #!/bin/bash BashSunday, August 21, 11

I speak #!/bin/bash Bash with a strong Perl accentSunday, August 21, 11

I speak fluent Italian La donzelletta vien dalla campagna, in sul calar del sole, col suo fascio dellerba e reca in mano un mazzolino di rose e viole, onde, siccome suole, ornare ella si appresta ...Sunday, August 21, 11

However, Sardinian comes handy sometimes Maissavida ti frimisti! (*) (*) May you never stopSunday, August 21, 11

I speak fluent English When in the Course of human events, it becomes necessary for one people to dissolve the political bands which have connected them with another, and to assume among the powers of the earth, the separate and equal station to which the Laws of Nature and of Natures God entitle them, a decent respect to the opinions of mankind requires that they should declare the causes which impel them to the separation.Sunday, August 21, 11

I speak fluent English When in the Course of human events, it becomes necessary for one people to dissolve the political bands which have connected them with another, and to assume among the powers of the earth, the separate and equal station to which the Laws of Nature and of Natures God entitle them, a decent respect to the opinions of mankind requires that they should declare the causes which impel them to the separation. hmm. wrong ﬂagSunday, August 21, 11

I speak fluent English When in the Course of human events, it becomes necessary for one people to dissolve the political bands which have connected them with another, and to assume among the powers of the earth, the separate and equal station to which the Laws of Nature and of Natures God entitle them, a decent respect to the opinions of mankind requires that they should declare the causes which impel them to the separation.Sunday, August 21, 11

But sometimes Italian is still necessary barista al fresco pasta maestro espresso tempo vendetta soprano ﬁnale piano dilettante cappuccino stucco stiletto virtuoso ﬁasco presto lotto pizza bravo lasagna studio spaghetti stanzaSunday, August 21, 11

And sometimes Italian is needed I tuoi perversi defunti!(*) (*) Your wicked ancestors!Sunday, August 21, 11

Although Sardinian is still effective here Ancu ti currullidi sa giustizia (*) (*) May the rule of law chase youSunday, August 21, 11

I speak Perl fluently #!/usr/bin/perl use strict; use warnings; use File::Find; while (my $line = <stdin>){ # …Sunday, August 21, 11

Sometimes, I need help from C use Inline C; print "9 + 16 = ", add(9, 16), "n"; __END__ __C__ int add(int x, int y) { return x + y; }Sunday, August 21, 11

Most of my work at the command line is Bash $ for D in $HOME/*/; do du -sh $D; > done > /tmp/occupancy #!/bin/bashSunday, August 21, 11

And sometimes, a pinch of Perl can help $ perl -pe s{(dd)-(dd)-(dddd)}{$3-$1-$2}g < rawfile.txt > betterfile.txt #!/bin/bashSunday, August 21, 11

And then on top of all this, I need to speak SQL $ mysql -e SELECT COUNT(*) FROM dbname.tbl_name WHERE col_name = 1 and other_col="A" Sunday, August 21, 11

I wish I could say that my SQL has a Perlish accent $ mysql -e SELECT COUNT(*) FROM dbname.tbl_name WHERE col_name = 1 and other_col="A" use strict; use DBI;?Sunday, August 21, 11

I wish I could say that my SQL has a Perlish accent $ mysql -e SELECT COUNT(*) FROM dbname.tbl_name WHERE col_name = 1 and other_col="A" but it wouldnt be trueSunday, August 21, 11

I could curse in Perl $ perl -pe s{(dd)-(dd)-(ddd d)}{$3-$1-$2}g < rawfile.txt > betterfile.txt (*) But it would not get me far. (*) already translatedSunday, August 21, 11

ThenSunday, August 21, 11

Then I will need to mix languages.Sunday, August 21, 11

Then I will need to mix languages. This talk will show exactly that.Sunday, August 21, 11

Then I will need to mix languages. This talk will show exactly that. Mixing languages.Sunday, August 21, 11

Then I will need to mix languages. This talk will show exactly that. Mixing languages. Yes.Sunday, August 21, 11

Then I will need to mix languages. This talk will show exactly that. Mixing languages. Yes. $ at the command lineSunday, August 21, 11

the Bash shell #! bash 81Sunday, August 21, 11

components • commands • aliases • variables • functions 82Sunday, August 21, 11

commands$ echo "something"something 83Sunday, August 21, 11

aliases$ alias mydate=mysql -e "select NOW()"$ mydate+---------------------+| now() |+---------------------+| 2010-04-12 07:14:19 |+---------------------+ 84Sunday, August 21, 11

aliases$ alias myself=mysql$ myself -e "select NOW()"+---------------------+| now() |+---------------------+| 2010-04-12 07:14:19 |+---------------------+ 85Sunday, August 21, 11

variables$ MYBATCH=mysql -N -B$ MYQUERY=select curdate()$ $MYBATCH -e "$MYQUERY"2010-04-12 86Sunday, August 21, 11

nesting variables# t.sh CMD="SELECT TABLE_NAME" TBL=information_schema.tables WHC1="TABLE_SCHEMA=mysql" WHC2="TABLE_NAME like u% " Q1="$CMD FROM $TBL" Q2="WHERE $WHC1 AND $WHC2" Q="$Q1 $Q2" echo $Q echo $Q | mysql 87Sunday, August 21, 11

nesting variables$ sh t.shSELECT TABLE_NAME FROMinformation_schema.tables WHERETABLE_SCHEMA=mysql AND TABLE_NAME likeu%TABLE_NAMEuser 88Sunday, August 21, 11

the mysql command line client mysql> 89Sunday, August 21, 11

SQL commands from a file $ mysql < filename.sql mysql> source filename.sql • executes the commands contained in filename.sql • commands can be queries understood by the server • and commands for the client 90Sunday, August 21, 11

SQL commands at the command line $ application | mysql $ echo "SELECT 1" | mysql $ mysql -e "SELECT 1" • executes the commands from standard input • commands can be queries understood by the server • and commands for the client 91Sunday, August 21, 11

batch mode $ echo "select curdate()" | mysql curdate() 2010-04-12 92Sunday, August 21, 11

table mode $ mysql -e "select curdate()" +------------+ | curdate() | +------------+ | 2010-04-12 | +------------+ 94Sunday, August 21, 11

Forcing batch mode mysql -B -e select curdate() curdate() 2010-04-12 95Sunday, August 21, 11

Forcing batch mode - no header mysql -N -B -e select curdate() 2010-04-12 ANT! PO RT IM 96Sunday, August 21, 11

Shell + mysql $ echo "The current servers date is `mysql -N -B -e select curdate()`." $ echo "The current servers date is $(mysql -N -B -e select curdate())." 97Sunday, August 21, 11

advanced command line techniques (((((( (: @ @ - $_ loops 98Sunday, August 21, 11

loops basics $ for N in 1 2 3 > do > echo $N > done 1 2 3 99Sunday, August 21, 11

loops basics $ for N in 1 2 3 ; do echo $N ; done 1 2 3 100Sunday, August 21, 11

loops basics $ for N in $(seq 1 3) ; do echo "this is number $N" ; done this is number 1 this is number 2 this is number 3 101Sunday, August 21, 11

loops from file $ head /usr/share/dict/words A a aa aal aalii aam Aani aardvark aardwolf 102Sunday, August 21, 11

loops from file $ for U in $(head /usr/share/dict/words ) ; do perl -le print uc shift $U ; done A A AA AAL AALII AAM AANI 103Sunday, August 21, 11

loops from mysql results $ mysql -B -N -e show tables from world City Country CountryLanguage 104Sunday, August 21, 11

loops from mysql results $ for T in $(mysql -B -N -e show tables from world) ; > do echo "CREATE TABLE world2.$T LIKE world.$T;" ; done CREATE TABLE world2.City LIKE world.City; CREATE TABLE world2.Country LIKE world.Country; CREATE TABLE world2.CountryLanguage LIKE world.CountryLanguage; 105Sunday, August 21, 11

piping loops back into mysql $ for T in $(mysql -B -N -e show tables from world) ; > do echo "CREATE TABLE world2.$T LIKE world.$T;" ; done > | mysql -vv 106Sunday, August 21, 11

advanced command line techniques nested :) @ @ - $_ #! #! #! #! loops #! #! 107Sunday, August 21, 11

pseudocodefor each database do for each table do print the table name 110Sunday, August 21, 11

actual codealias mybatch=mysql -B -Nfor D in $(mybatch -e "SHOW SCHEMAS")do for T in $(mybatch -e "SHOW TABLES FROM $D") do echo "$D.$T" donedone 111Sunday, August 21, 11

advanced command line techniques $HOME options files 112Sunday, August 21, 11

Default option files/etc/my.cnf/etc/mysql/my.cnf (Debian, Ubuntu)$BASEDIR/my.cnf$HOME/.my.cnf 113Sunday, August 21, 11

Custom option files$ mysql --no-defaults$ mysql --defaults-file=filename$ mysql --defaults-extra-file=filename 114Sunday, August 21, 11

Option files sections[mysqld]# option for server[client]# option for any mysql client[mysql]# options specific to mysql CLI app 115Sunday, August 21, 11

Option files example[client]user=mynamepassword=mypassword[mysql]prompt="mysql (h) {u} [d] > " 116Sunday, August 21, 11

Option files: caution# protect your option files!$ chmod 600 $HOME/.my.cnf$ ls -l .my.cnf-rw------- 1 [...] 10:04 .my.cnf 117Sunday, August 21, 11

promptmysql> prompt gm (h) {u} [d] >PROMPT set to gm (h) {u} [d] > gm (localhost) {root} [(none)] > use testDatabase changedgm (localhost) {root} [test] ># h -> host# u -> user# d -> database 118Sunday, August 21, 11

advanced command line techniques (((((( (: @ @ - #! aliases 119Sunday, August 21, 11

An useful alias$ alias mysql_as_root=mysql --defaults_file=~/.my_as_root.cnfmysql_as_root -e "select user()"+----------------+| user() |+----------------+| root@localhost |+----------------+ 120Sunday, August 21, 11

An useful alias#.my_as_root.cnf[client]user=rootpassword=your_mysql_root_password[mysql]prompt="### [h] {u} (d) > " 121Sunday, August 21, 11

Another useful aliasalias mybatch=mysql -B -N mybatch -e "select version()"5.1.45-log 122Sunday, August 21, 11

( ) advanced command line techniques (((((( (: @ @ - #! functions 123Sunday, August 21, 11

functions • like aliases, but with parameters 124Sunday, August 21, 11

sample functionfunction grant_new { mysql_as_root -e "GRANT $1 on $2 to $3 identifiedby "$4" "}$ grant_new SELECT "world.*" john john_secret 125Sunday, August 21, 11

sample function$ mysql_as_root -e show grants for johnG*** 1. row ***Grants for john@%: GRANT USAGE ON *.*TO john@% IDENTIFIED BY PASSWORD*73A88CE13374128844E3ABB500F439A9C02C53A3*** 2. row ***Grants for john@%: GRANT SELECT ON`world`.* TO john@% 126Sunday, August 21, 11

advanced command line techniques $_ getting query details127Sunday, August 21, 11

which query is being executed?# wrong.sql# this script has one erroruse test ;drop table if exists t1 ;create table t1 (id int);insert into t1 values (a); 128Sunday, August 21, 11

which query is being executed?$ mysql < wrong.sqlERROR 1054 (42S22) at line 4: Unknowncolumn a in field list# WHERE IS THE ERROR? 129Sunday, August 21, 11

which query is being executed?$ mysql -v < wrong.sql--------------drop table if exists t1----------------------------create table t1 (id int)----------------------------insert into t1 values (a)--------------ERROR 1054 (42S22) at line 4: Unknown columna in field list 130Sunday, August 21, 11

getting execution times$ mysql -vv -e select rating,count(*) from sakila.filmgroup by rating+--------+----------+| rating | count(*) |+--------+----------+| G | 178 || PG | 194 || PG-13 | 223 || R | 195 || NC-17 | 210 |+--------+----------+5 rows in set (0.00 sec) 131Sunday, August 21, 11

getting execution times (batch mode)$echo select rating,count(*) from sakila.film groupby rating | mysql -vv--------------select rating,count(*) from sakila.film group byrating--------------rating count(*)G 178PG 194PG-13 223R 195NC-17 2105 rows in set 132Sunday, August 21, 11

getting execution times (batch mode)$ echo select rating,count(*) from sakila.film groupby rating | mysql -vvv--------------select rating,count(*) from sakila.film group byrating--------------+--------+----------+| rating | count(*) |+--------+----------+| G | 178 || PG | 194 || PG-13 | 223 || R | 195 || NC-17 | 210 |+--------+----------+ 1335 rows in set (0.00 sec)Sunday, August 21, 11

advanced command line techniques getting _$ distinct results134Sunday, August 21, 11

getting a single value$ mysql -NB -e "show variables like port" port 3306# no header, batch mode# still two values 136Sunday, August 21, 11

getting a single value$ mysql -NB -e "show variables like port" | awk {print $2}3306$ mysql -NB -e "show variables like port" | perl -lane print $F[1]3306# filtering the wanted value 137Sunday, August 21, 11

getting a single value: is replication working?$ mysql -e "show slave status G" | grep -i slave_io_running Slave_IO_Running: Yes# too many things.# We only want a yes or no 138Sunday, August 21, 11

getting a single value: is replication working?$ mysql -e "show slave status G" | grep -i "slave_io_running" | perl -lane print $F[1]Yes# yes! 139Sunday, August 21, 11

Whats perl -lane? 140Sunday, August 21, 11

Whats perl -lane? perl -lane must turn perl 140Sunday, August 21, 11

Perl command line must-know facts (1)perl -e print "hellon"# executes perl code 141Sunday, August 21, 11

Perl command line must-know facts (2)perl -ne print if /â/# starts a loop from standard input# and executes the perl code for each# iterationperl -MO=Deparse -ne print if /â/LINE: while (defined($_ = <ARGV>)) { print $_ if /â/;} 142Sunday, August 21, 11

Perl command line must-know facts (3)perl -pe s/(âw+)/$1/# like -ne, but prints each line$ perl -MO=Deparse -pe s/(âw+)/$1/LINE: while (defined($_ = <ARGV>)) { s/(âw+)/$1/;}continue { print $_;} 143Sunday, August 21, 11

Perl command line must-know facts (4)perl -le print "hello"# strips newlines from input and adds# them to output$ perl -e print "hello"hello$$ perl -le print "hello"hello$ 144Sunday, August 21, 11

Perl command line must-know facts (5)perl -lane print $F[0]# splits input into @F array$ echo "a b c" | perl -lane print $F[1]b$ echo "a b c" | perl -lane print $F[2]c 145Sunday, August 21, 11

Perl command line must-know facts (6)Perl and shell contention for $ #!/bin/bash Take the $$$ and run 146Sunday, August 21, 11

Beware of $VARIABLES on thecommand line$ perl -le "$x=1; print $x"syntax error at -e line 1, near "="Execution of -e aborted due tocompilation errors. 147Sunday, August 21, 11

double quotesBeware of $VARIABLES on theconsumed by the allow the shell to shell, before Perl interpolatecommand line can see it variables$ perl -le "$x=1; print $x"syntax error at -e line 1, near "="Execution of -e aborted due tocompilation errors. 147Sunday, August 21, 11

double quotesBeware of $VARIABLES on theconsumed by the allow the shell to shell, before Perl interpolatecommand line can see it variables$ perl -le "$x=1; print $x"syntax error at -e line 1, near "="Execution of -e aborted due tocompilation errors. consumed by the shell, before Perl can see it 147Sunday, August 21, 11

double quotesBeware of $VARIABLES on the consumed by the allow the shell to shell, before Perl interpolatecommand line can see it variables$ perl -le "$x=1; print $x"syntax error at -e line 1, near "="Execution of -e aborted due tocompilation errors. consumed by the shell, before Perl$ perl -le $x=1; print $x can see it1 147Sunday, August 21, 11

double quotesBeware of $VARIABLES on the consumed by the allow the shell to shell, before Perl interpolatecommand line can see it variables$ perl -le "$x=1; print $x"syntax error at -e line 1, near "="Execution of -e aborted due tocompilation errors. consumed by the shell, before Perl$ perl -le $x=1; print $x can see it1 single quotes the consumed by prevent variable shell, before Perl consumption by can shell it the see147Sunday, August 21, 11

advanced command line techniques combining column names 148Sunday, August 21, 11

combining column names$ mysql -N -e desc world.City+-------------+----------+----+-----+| ID | int(11) | NO | PRI || Name | char(35) | NO | || CountryCode | char(3) | NO | || District | char(20) | NO | || Population | int(11) | NO | |+-------------+----------+----+-----+ 149Sunday, August 21, 11

combining column names# Expected result:# SELECT CONCAT_WS(|,# ID,Name,CountryCode,District,# Population) from City 150Sunday, August 21, 11

combining column names (I)echo "select concat_ws(|, $(mysql -N -e desc world.City | perl -lane push @f, $F[0]; -e END{print join ",", @f}))from world.City"select concat_ws(|,ID,Name,CountryCode,District,Population) from world.City 151Sunday, August 21, 11

combining column names (II)echo "select concat_ws(|, $( mysql -NB -e selectgroup_concat(column_name) frominformation_schema.columns wheretable_name="City" andtable_schema="world")) fromworld.City"select concat_ws(|,ID,Name,CountryCode,District,Population) from world.City 152Sunday, August 21, 11

combining column names (IIa)echo "select concat_ws(|, $( mysql -NB -e selectgroup_concat(column_name) frominformation_schema.columns wheretable_name="Country" andtable_schema="world")) fromworld.Country"select concat_ws(|,Code,Name,Continent,Region,SurfaceArea,IndepYear,Population,LifeExpectancy,GNP,GNPOld,LocalName,GovernmentForm,HeadOfState,Capital,Code2) from world.Country 153Sunday, August 21, 11

piping the result back to mysqlecho "select concat_ws(|, $( mysql -NB -e selectgroup_concat(column_name) frominformation_schema.columns wheretable_name="Country" andtable_schema="world")) as line fromworld.Country" | mysql | head -n 4lineAFG|Afghanistan|Asia|Southern and Central Asia|652090.00|1919|22720000|45.9|5976.00|Afganistan/Afqanestan|Islamic Emirate|Mohammad Omar|1|AFNLD|Netherlands|Europe|Western Europe|41526.00|1581|15864000|78.3|371362.00|360478.00|Nederland|Constitutional Monarchy|Beatrix|5|NLANT|Netherlands Antilles|North America|Caribbean|800.00|217000|74.7|1941.00|Nederlandse Antillen|Nonmetropolitan Territory of 154The Netherlands|Beatrix|33|ANSunday, August 21, 11

advanced command line techniques $_ dumping data 155Sunday, August 21, 11

dumping a tab delimited data set# the naive way$ mysql -e "SELECT * into outfile /tmp/city_datafrom world.City"# it works only if:# - /tmp/city_data does not exist# - /tmp is world writable 156Sunday, August 21, 11

dumping a tab delimited data set$ mysql -B -N -e "SELECT * from world.City" > /tmp/city_data# it works always# if you omit the "-N" it also# includes a header 157Sunday, August 21, 11

dumping a CSV$ mysql -B -e "SELECT * from world.City" | perl -F"t" -lane print join ",", map {s/"/""/g; /^[d.]+$/ ? $_ : qq("$_")} @F > /tmp/city_data 158Sunday, August 21, 11

dumping a CSV$ mysql -B -e "SELECT * from world.City" | perl -F"t" -lane print join ",", splits the line by tabs, instead map {s/"/""/g; of the default whitespace /^[d.]+$/ ? $_ : qq("$_")} @F > /tmp/city_data 158Sunday, August 21, 11

advanced command line techniques script it 159Sunday, August 21, 11

use the mysql.sh script § The companion code for this tutorial § http://tinyurl.com/cli-tut-code-2010 § It will save you much typing § But use it at your own risk! 160Sunday, August 21, 11

Usage • download the script • load it $ source mysql.sh 161Sunday, August 21, 11

mysql.sh functions • my_batch "query" • returns the result of the query. • Without header • Without box $ my_batch "SELECT * FROM test.t1" 1 aaa 2 bbb 3 ccc 162Sunday, August 21, 11

mysql.sh functions • my_value "what" • returns a single value • e.g. $ my_value "version()" 5.1.49 $ my_value "@@server_id" 1 163Sunday, August 21, 11

mysql.sh functions • my_var "what" • returns the value of a given variable • e.g. $ my_var port 3306 $ my_var datadir /usr/local/mysql/data 164Sunday, August 21, 11

mysql.sh functions • my_status "what" • returns the value of a given status item • e.g. $ my_status com_select 2653 $ my_status connections 452 165Sunday, August 21, 11

mysql.sh functions • my_dump_dataset "query" [header] • returns the values of a given query separated by tabs • if a header is requested, it will be part of the result • e.g. $ my_dump_dataset "select * from world.City" | head -n 5 1 Kabul AFG Kabol 1780000 2 Qandahar AFG Qandahar 237500 3 Herat AFG Herat 186800 4 Mazar-e-Sharif AFG Balkh 127800 166 5 Amsterdam NLD Noord-Holland 731200Sunday, August 21, 11

mysql.sh functions • my_csv "query" [separator] • returns the values of a given query separated by commas, or by a given separator • e.g.$ my_csv select * from world.City |head -n 3"ID","Name","CountryCode","District","Population"1,"Kabul","AFG","Kabol",17800002,"Qandahar","AFG","Qandahar",237500 167Sunday, August 21, 11

inside the command line client handling :-) @ @ $_ output 168Sunday, August 21, 11

pagermysql> SELECT * FROM world.City;rec …rec …rec …4000 + recordsunreadable 169Sunday, August 21, 11

pagermysql> pager lessPAGER set to lessmysql> SELECT * FROM world.City;# [ records in a manageable window ]mysql> nopagerPAGER set to stdout# back to normal 170Sunday, August 21, 11

More pager tricksmysql> pager vim -# like less, but you can also edit# the results and eventually save# them 171Sunday, August 21, 11

More pager tricksmysql> pager md5sumPAGER set to md5summysql> select * from City;5d17ffa50d6da200dee621823ade2543 -4079 rows in set (0.01 sec)# produce a CRC for the result set# instead of displaying it.# Useful for data comparison.# (Thanks to Baron Schwartz 172Sunday, August 21, 11

Comparing data sets with md5summysql> select * from City;5d17ffa50d6da200dee621823ade2543 -mysql> select * from City order by ID;5d17ffa50d6da200dee621823ade2543 -mysql> select * from City order by name;5ab980f5b69a6e65d3b8cd5a95ec94e0 -mysql> select * from City order byDistrict;2bf37cbe91b3ffeaebc8c4b6084b8784 - 173Sunday, August 21, 11

More pager tricksmysql> pager cat > /tmp/t1.txtmysql> select "one" union select"two";mysql> pager cat > /tmp/t2.txtmysql> select "one" union select"TWO";mysql> nopagermysql> ! vimdiff -o /tmp/t[12].txt# see the difference between two# datasets 174Sunday, August 21, 11

175Sunday, August 21, 11

logging sessionsmysql> tee mylog.txt# logs everything that you see on# the screen.# your commands, the results,# warnings, error messages, and so on.mysql> notee# back to normal 176Sunday, August 21, 11

read more § http://datacharmer.blogspot.comSunday, August 21, 11

THANKS! ? Question time Comments on Twitter: @datacharmer This work is licensed under the Creative Commons Attribution-Share Alike 3.0 Unported License.Sunday, August 21, 11

to do § add dead languages § comparing Latin and ancient Greek to COBOL and Pascal § add new tricks § {x,y} expansion in bash § G in MySQL §Sunday, August 21, 11

Don't be a GUI rookie forever - Be in command! (line)

by Giuseppe Maxia

Accessibility

Categories

Upload Details

Usage Rights

2 Embeds 798

Statistics

Don’t be a GUI rookie forever - Be in command! (line) Presentation Transcript

http://mcgray.com.ua	797
http://twitter.com	1