This document presents an agenda for becoming a "console cowboy" by learning to be more productive using the terminal and bash shell. It covers the basic terminal tools, bash usage and configuration, utilities like grep, sed and awk, scripting with variables, conditionals and loops, and tools for developers like Homebrew, Git, Xcode and xcpretty. The goal is to stop using the mouse and automate work by writing scripts to harness the power of the Unix command line.
Shapes for Sharing between Graph Data Spacesย - and Epistemic Querying of RDF-...
ย
Unleash your inner console cowboy
1. Unleash your inner
console cowboy
Kenneth Geisshirt
kg@realm.io
Realm Inc.
@realm
http://github.com/Realm/
http://realm.io/
Ivan Constantin, https://www.๏ฌickr.com/photos/ivan70s/
2. Todayโs goal
โข Present bash as a productivity tool
โข stop using the mouse ๐
โข Write scripts to automate your work
โข Begin to use advanced tools in your daily work
Become a console cowboy
3. Agenda
โข The terminal and the
shell
โข Basic usage of bash
โข Living on the
command-line
โข Useful utilities
โข Scripting
โข Home brew
โข Tools for developers
โข git
โข Xcode
5. Which terminal?
โข iTerm2 is much better
โข Easier to change tab (โ left
+ right, CTRL+TAB)
โข Change Desktop navigation
to โฅ left + right
โข Add CTRL left + right to
iTerm2 preferences
โข Keeps SSH connection alive
โข http://iterm2.com/
6. Which shell?
OS X comes with many shells
โค bash, csh, ksh, sh, tcsh, and zsh
Parรฉe, https://www.๏ฌickr.com/photos/pareeerica/
Since 10.3, bash has been the default shell
OS X 10.11.1 carries bash 3.2.57 (2014-11-07)
Stephen R. Bourne (Bell lab) introduced the
shell to UNIX in 1977
Home brew has many great bash related packages
7. Redirection
UNIX idioms
โข a tool should do one thing
but do it well
โข text is the universal data
format
}Output of one utility is input for the next
Bash implements redirection:
โข stdout to ๏ฌle: >
โข stdin from ๏ฌle <
โข append stdout to ๏ฌle: >>
โข stderr to stdout: 2>&1
Examples:
echo โHelloโ > hello
cat < hello
echo โWorldโ >> hello
clang notfound.m > error 2>&1
8. Pipes
โข Introduced to UNIX by Douglas
McIlroy in 1973
โข Pipes are the glue in UNIX
component based programming (aka
shell scripting)
โข Powerful idiom for stream processing
โข The character | is used by all known
shells
Examples
lsof | grep ^Keynote | wc -l
ifconfig | grep -e ^[a-z] | cut -f1 -d:
"Pipeline" by TyIzaeL - Licensed under Public domain via Wikimedia Commons
http://commons.wikimedia.org/wiki/File:Pipeline.svg#mediaviewer/File:Pipeline.svg
9. Con๏ฌguration
โข $HOME/.bash_profile and $HOME/.bashrc
are your personal con๏ฌguration
โข alias - useful for often used options
โข Setting prompt (PS1) and search path (PATH)
โข Reload con๏ฌguration: source ~/.bash_profile
10. .bash_pro๏ฌle
# Ignore a few commands in history
export HISTIGNORE="pwd:ls:ls -l:cd"
# don't put duplicate lines in the history. See bash(1) for more options
# don't overwrite GNU Midnight Commander's setting of `ignorespace'.
HISTCONTROL=$HISTCONTROL${HISTCONTROL+:}ignoredups
# Bash completion
if [ -f $(brew --prefix)/etc/bash_completion ]; then
. $(brew --prefix)/etc/bash_completion
fi
# Prompt - including git
PS1='u@h:w$(__git_ps1 " (%s)") $ โ
# Color ls etc.
alias ls="ls -G"
alias ll="ls -l"
# https://xkcd.com/149/
alias fuck='sudo $(history -p !!)' # rerun as root
11. Keyboard short-cuts
โข Bash uses Emacs bindings by default ๐
(help bind for details)
Movement
โข CTRL-a beginning of line
โข CTRL-e end of line
โข CTRL-โ One word left
โข CTRL-โ One word right
Cut-n-paste
โข CTRL-space mark
โข ๐ Delete word
โข CTRL-d Delete character
โข CTRL-_ undo
โข CTRL-k delete until end
of line
โข CTRL-y yank from kill-
ring
Remap CTRL-โ/โ
12. History
โข Bash stores your command history
โข history - show latest commands
โข !! - run last command
โข !number - run command number again
โข CTRL-r - search in history
โข Exclude commands from history:
โข export HISTIGNORE=โpwd:ls:ls -l:cdโ
โข Exclude duplicates from history:
โข export HISTCONTROL:ignoredups
13. Completion
โข Use TAB to let Bash complete as much as possible
โข Use TAB+TAB to show possible completions
โข Bash has programmable completion โ you can
specify what Bash does
โข Large collections of completion recipes exist (home
brew is your friend)
14. Living on the command-line
โข cd - - go back to previous folder
โข file file - guess ๏ฌle content (magic numbers)
โข lsof - list open ๏ฌles
โข ps (aux or -ef) and top - show processes
โข Simple watch:
while true ; do clear ; command ; sleep
n ; done
15. OS X speci๏ฌc commands
โข open file - Starts registered program and open ๏ฌle
โข say โHello worldโ - speech synthesis (download
extra voices/languages in System preferences)
โข ls | pbcopy - copy stdin to paste board
โข pbpaste - paste to stdout
โข dns-sd -B _ssh._tcp - show Bonjour enabled SSH
hosts
16. Useful utilities
Find ๏ฌles: find . -name โ*.oโ -delete
Patterns: grep -r list *
Cut ๏ฌeld: cut -f1,3 -d: /etc/passwd
Word count: wc -l *.cpp
Transform: tr โ โ โ_โ < README.org
Sort lines: sort -t: -n -r -k 4 /etc/passwd
Last lines: tail /etc/passwd
First lines: head /etc/passwd
17. sed - the stream editor
โข sed is used to edit ๏ฌles non-
interactively
โข Option -E gives an editing (regular)
expression
โข s/FISH/HORSE/g - substitute
โข /FISH/d - delete lines
joinash, https://www.๏ฌickr.com/photos/joinash/
Option -i is tricky:
โข GNU sed has optional extension
โข BSD sed requires extension (โโ is useful)
18. awk - a processing tool
โข awk is a programming
language by itself
โข Matching lines are
processed
โข line is split in ๏ฌelds
(spaces are default)
Patterns:
BEGIN - before opening ๏ฌle
END - after closing ๏ฌle
A. Aho, P. Weinberger, B. Kernighan
cat foo | awk 'BEGIN {total=0 } END { print total } { total+=$1 }'
20. Bash for programmers
โข Bash is a complete programming language
โข Shell scripts grow and become ugly ๐
โข Execution:
โข sh script.sh
โข chmod +x script.sh; ./script.sh
โข Interpreted language โ slow
21. Basic syntax
โข White spaces: space and
tab
โข Comments: # and to end-
of-line
โข Statements: either end-
of-line of ; (semicolon)
โข Variables and functions:
Letters, digits and
underscore
#!/bin/bash
# Monte Carlo calculation of pi
NSTEPS=500
NHITS=0
i=0
while [ $i -lt $NSTEPS ]; do
x=$(echo $RANDOM/32767 | bc -l)
y=$(echo $RANDOM/32767 | bc -l)
d=$(echo "sqrt($x*$x+$y*$y) < 1.0" | bc -l)
if [ $d -eq 1 ]; then
NHITS=$(($NHITS + 1))
fi
i=$(($i + 1))
done
PI=$(echo "4.0*$NHITS/$NSTEPS" | bc -l)
echo "PI = $PI"
22. Variablesโข Case-sensitive names
โข No declarations, no types
โข Strings: โโฆโ are substituted; โโฆโ are not
โข Assignment (=): no spaces!
โข $(โฆ) assignment from stdout
including spaces
โข I often use awk โ{print $1}โ to
remove spaces
โข $((โฆ)) arithmetic
โข $varname - value of variable varname
Built-in variables:
โข $# is the number of argument
โข $1, $2, โฆ are the arguments
โข $$ is the process ID
โข $? is exit code of last command
24. Branches
โข Simple branching with if then
else fi
โข Enclose condition with []
โข elif is possible, too
โข Use case in esac when you
can many cases and single
condition
String operators:
-z is empty?
-d is directory?
-f is ๏ฌle?
== equal to
!= not equal to
Integer operators:
-eq equal to
-lt less than
-ne not equal to
-gt greater than
25. branches.sh
#!/bin/bash
if [ -z "$1" ]; then
name="Arthur"
else
name="$1"
fi
if [ "$name" != "Arthur" ]; then
echo "Not Arthur"
else
echo "Hello Arthur"
fi
answer=$((6*7))
if [ $answer -eq 42 ]; then
echo "Life, the universe, and everything"
fi
26. branches.sh - conโt
case "$name" in
"Arthur")
echo "Welcome onboard"
;;
"Trillian")
echo "You know Arthur"
;;
*)
echo "Who are you?"
;;
esac
27. Loops
โข Simple loops: for โฆ in โฆ ; do โฆ
done
โข The seq utility can generate list of
numbers
โข Conditional loops: while โฆ ; do โฆ
done
โข Line-by-line: while read line ;
do โฆ done
One-liner (similar to watch)
while [ true ]; do
clear;
echo $RANDOM;
sleep 1;
done
28. loops.sh#!/bin/bash
# Multiplication table
for i in $(seq 1 10); do
echo "$i $((3*$i))"
done
# All .sh files
for f in $(ls *.sh); do
echo "$f $(head -1 $f | cut -c3-) $(wc -l $f | awk '{print $1}')"
done
# read self line-by-line
i=1
cat $0 | while read line ; do
nchars=$(echo "$line" | wc -c | awk '{print $1}')
echo "$i $nchars"
i=$(($i+1))
done | sort -n -k 2
29. Functions
โข Functions can increase readability of your scripts
โข arguments are $1, $2, โฆ
โข local variables can be used
โข return an integer and get it as $?
โข Use global variable to return a string ๐
31. Tips and tricks
โข Use set -e to exit early
โข or use || exit 1
โข set -O pipefail and you can get the exit code of the
๏ฌrst failing program in a pipe
โข xcpretty never fails but xcodebuild might
โข Use tee to write to stdout and ๏ฌle
โข To trace (debugging): set -x or sh -x
32. Tips and tricks
โข Always use โ$varโ when dealing with ๏ฌle names (and strings)
โข str=โfish horseโ; for i in $str; do echo $i; done
โข str=โfish horseโ; for i in โ$strโ; do echo $i; done
โข Call mkdir -p when creating folders
โข Create temp. ๏ฌles with mktemp /tmp/$$.XXXXXX
โข Using variable to modify behaviour of script:
โข FLAGS=โ-O3 -libc++=stdlibc++โ build.sh
โข Subshells: (cd foo && rm -f bar)
34. Home brew
โข Home brew provides calories for
console cowboys
โข You donโt have to be root to install
โข Software is installed in /usr/
local/Cellar, and symlinked to /
usr/local/bin
โข Brew cask is for binary distribution
โข http://brew.sh and http://
caskroom.io
Greg Peverill-Conti, https://www.๏ฌickr.com/photos/gregpc/
Examples:
brew search bash
brew info bash
brew install bash
brew update
35. Tools for developers
โข Apple provides some basic tools
โข nm - display symbol table
โข c++filt - Prettify C++ and Java names
โข otool -L - display which shared libraries are
required
โข libtool - create libraries
โข lipo - manipulate fat/universal binaries
zzpza, https://www.๏ฌickr.com/photos/zzpza/
Examples:
nm book.o | c++filt
otool -L RealmInspector
37. Xcode
โข You can build Xcode projects at the
command-line
xcodebuild -scheme OreDevPlain -
configuration Release -sdk
iphonesimulator
โข Targets: clean, build, test
โข You can add shell scripts to build phases
38. xcpretty
โข The output of xcodebuild can be hard to read
โข xcpretty makes it prettier
โข Installation:
โข sudo gem install xcpretty
โข Usage:
โข xbuildcode โฆ | xcpretty
40. Further information
โข Classical Shell Scripting. R. Arnolds and N.H.F.
Beebe. OโReilly Media, 2005.
โข The sed FAQ: http://sed.sourceforge.net/
sedfaq.html
โข Advanced Bash-Scripting Guide: http://
www.tldp.org/LDP/abs/html/
http://realm
.io
W
e
are
hiring