If you have a problem, if no one else can help... and if you can find them, maybe you can hire the bAsh-TEAM!


Published on

Ever had a problem?
No one could help you?
Please call the BASH Team!

This presentation starts with some preliminary thoughts about software integration.

Shell scripting means knowing your system and the applications you have available that can do the dirty stuff for you.

BASH is all about gluing... so what are the ways we can use to control resources and integrate software through scripting?

Some set of "advanced" features of BASH are explained and I'm sure you'll find them very handy to enhance your scripts. The BASH recipes can be used for common daily tasks of end users or administrators.

"Hacking" web sites can also be done using BASH and some command line tools. We'll talk a bit about it.

Learn also with real examples explained, namely:
- fetch music streams from web URL's to disk, from a popular Portuguese radio station
- emulate a user/replicate online banking tasks and extract values automatically, with a real bank example (by means of "reverse engineering" web pages)
- send SMS from command line, using a mobile operator's site (by means of "reverse engineering" web pages)

This is not intended to be a BASH tutorial for beginners, neither a simple plain tutorial. It's more a kind of BASH "hacking" guide.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

If you have a problem, if no one else can help... and if you can find them, maybe you can hire the bAsh-TEAM!

  1. 1. If you have a problem, if no one else can help... and if you can find them, maybe you can hire the bAsh-TEAM<br />A different BASH shell scripting tutorial<br />03.12.2009 @ Codebits 2009<br />Thisdocumentisintellectualpropertyof PT Inovação, SA andshouldnotbeused without express written permission.<br />
  2. 2. Schedule/Agenda<br />A little about me<br />Preliminary thoughts<br />Pragmatic programming & Gluing stuff<br />BASH<br />Some characteristics<br />Tricky snippets explained<br />Introduction to “Hacking” web sites (tools, how-to)<br />Some examples explained<br />Extra stuff (if we have time, I’ll guess we’ll have)<br />References/Further reading<br />03-12-2009<br />2<br />
  3. 3. Warm-up<br />Random crap…<br />03-12-2009<br />3<br />
  4. 4. A little about me<br />Co-founder of Radioactive Design portuguese demogroup in the 90’s<br />Experience in “hacking” mostly in Linux, networking and protocols (messaging protocols@Layer 7, TCP@Layer 4)<br />I love problems and to find their sources!<br />Doing work for a “telco” R&D (PT Inovação), developing and managing projects and sometimes doing research <br />Languages<br />Java (mostly J2SE)<br />Perl<br />Ruby / Ruby-on-Rails<br />BASH<br />Some C and if you dig a little you’ll find also Pascal and even Visual Basic <br />03-12-2009<br />4<br />
  5. 5. Preliminary thoughts<br />Pragmatic programming<br />Don’t try to reinvent the wheel, at least every week <br />Be pragmatic:<br />If you can reuse a solution, think about it!<br />If you’re going to implement it, and if it really doesn’t matter what technology you should be using… then use the fastest/pragmatic approach; don’t implement it in your preferred language just because you like more that one<br />03-12-2009<br />5<br />
  6. 6. Gluing stuff<br />Make it all work together<br />Some call it gluing, other prefer “integration”… I just say make it all work<br />Integration can be made using well defined API’s or some “out-of-band glue”<br />External Scripting analyzing logs, produced files, etc…<br />Know your system!<br />Be aware of the tools available in your OS, they will make your life a lot easier!<br />03-12-2009<br />6<br />
  7. 7. Typical Scripting Flow<br />03-12-2009<br />7<br />catfile.txt | egrep -v “[0-9]”| sort | uniq | xargstouch<br />….<br />
  8. 8. BASH<br />What about…<br />03-12-2009<br />8<br />
  9. 9. BASH: Bourne-Again Shell<br />BASH it’s not just a simple shell, it’s not just a scripting language nor a simple application<br />BASH is all about integrating software and controlling resources through scripting<br />03-12-2009<br />9<br />Once again, know your distribution!<br />
  10. 10. Gluing with BASH<br />Passing data/information all around<br />Data can be exchanged within hosts or between hosts, using sockets or any other messaging passing method<br />In BASH programs communicate using:<br />Standard streams (stdin, stdout, stderr)<br />The programs exit/return code<br />Signals<br />And sometimes using environment variables<br />The communication can be made by any kind of “out-of-band”protocol” (socket, some file, …)<br />03-12-2009<br />10<br />
  11. 11. Successvs Failure<br />Thereversedtruth: thebooleanupsidedown<br />A successfulprogramterminationreturnstheexitcode 0; therefore “true” in BASH is 0<br />Allabnormalprogramterminationreturns a valuediferentof 0 to the shell; therefore “false” in BASH iseverythingexcept 0<br />Note thatinalmost 100% oflanguages, as inlogic, “false” isrepresentedby 0 and “true” byanyothervalue<br />03-12-2009<br />11<br />
  12. 12. (Some) BASH specialvariables<br />$$<br />Current PID of the running program<br />$! <br />PID of last child process started by the current shell<br />$?<br />Exit code of the last executed program<br />!$ <br />Arguments used in the last command<br />$0 <br />Name of current running script<br />$1..$n<br />First to n-th argument passed to the script<br />03-12-2009<br />12<br />
  13. 13. RedirectingStreams<br />Streamscanberedirected<br />Youcanmakestdoutpoint to some file descriptororyoucanmakestderrpoint to thesame FD usedbystdout<br />Cautionwiththeorderofredirection: theredirectionismadenot as a wholebutfromleft to right<br />&gt; some_file<br />Redirectsstdoutto some_file<br />2&gt;&1<br />Redirectsstderr to thesame FS usedby<br />2&gt;&-<br />Redirectsstderr to null (/dev/null)<br />&gt; bla 2&gt;&1isdifferentfrom2&gt;&1 &gt; bla<br />03-12-2009<br />13<br />
  14. 14. Handling signals<br />A BASH script, as any standard program, can catch and handle signals!<br />You can do this to provide things such as configuration reloading or graceful exits<br />03-12-2009<br />14<br />$LOCK=/tmp/somefile.lock<br />[ -f $LOCK ] && exit 0 <br />trap &quot;{ rm -f $LOCK ; exit 255; }&quot; EXIT <br />touch $LOCK || exit 255<br />….<br />
  15. 15. Paralell Execution<br />Multiple programs can be started in the background, by appending “&” in the command line<br />“wait” can be used to wait for child processes and therefore provide basic means of process synchronization; a PID argument can be given<br />03-12-2009<br />15<br />find /home/ -type d | do_some_weird_stuff&<br />another_command&<br />wait<br />echo “yes, finally I can shutdown!! ” ; shutdown –hnow<br />
  16. 16. Subshells<br />Another shell can be instantiated within the current shell (the subshell has its own PID)<br />A subshell is delimited by parentheses<br />You can use a subshell to execute several programs/commands and “grab” the whole output (stdout/stderr) produced in the context of that subshell to further process it!<br />03-12-2009<br />16<br />(find /home/ -type d;ls /home/sergio) | grepgold<br />
  17. 17. Tricky snippeTs<br />Going deep through and hacking with…<br />03-12-2009<br />17<br />
  18. 18. Introduction to “Hacking” Web sitesTools<br />A protocol analyzer is your friend (e.g. Wireshark / tshark) but it won’t deal with HTTPs<br />You can use Firefox plus some useful extensions no analyze HTTP(s) requests:<br />Live HTTP headers<br />Tamperdata<br />Modify Headers<br />In most Linux distributions you have HTTP clients that can be easily integrated with BASH<br />GET/POST<br />wget (preferred)<br />03-12-2009<br />18<br />
  19. 19. Introduction to “Hacking” Web sitesHow-to (1/2)<br />View the source of web pages, they’ll reveal much of the web site logic (and you’ll find many surprising things!)<br />Sites often do validation in the client side using JavaScript =&gt; security problems,hidden or extended features<br />03-12-2009<br />19<br />
  20. 20. Introduction to “Hacking” Web sitesHow-to (2/2)<br />Many parameters inside HTML are pure garbage<br />Some sites demand HTTP requests to be made using POST while for other it’s indifferent<br />Some sites do redirect on POST! So your client must either implement this behavior or else POST to the final URL<br />03-12-2009<br />20<br />
  21. 21. Fetchandsavemusicstreams<br />03-12-2009<br />21<br />pnumber[1]=2904; pname[1]=&quot;Antena3DanceClub”; pnumber[2]=1078; name[2]=&quot;AmbientaSons“; max=2; basedir=&quot;antena3“; totchilds=0; lastnchilds=-1<br />for n in`seq 1 $max`; do<br />echo &quot;processing${pname[$n]}...”; mkdir -p &quot;$basedir/${pname[$n]}“<br />wmafiles=`GET -P &quot;http://ww1.rtp.pt/multimedia/programa.php?hist=1&prog=${pnumber[$n]}&quot; <br />| egrep -o &quot;mms://.*.wma&quot;|sort|uniq`<br /> for wma in $wmafiles; do<br />file=`basename &quot;$wma&quot;`<br />filepath=&quot;$basedir/${pname[$n]}/$file“<br />fprocs=`lsof$filepath2&gt;- | wc -l 2&gt;-`<br />if [ &quot;$fprocs&quot; -gt 0 ]<br />then<br /> echo &quot;someone is accessing $file... bypassing..&quot;<br />else<br />echo &quot;fetching$file...&quot;<br />nohupmimms -r &quot;$wma&quot; &quot;$filepath&quot; &gt;- 2&gt;&1 &<br /> ((totchilds++))<br />fi<br />done<br />done<br />echo &quot;waiting for background processes to finish fetching all musics...“<br />wait<br />
  22. 22. Send SMS using a mobile operator’s site<br />03-12-2009<br />22<br />if [ $# -le 0 ]<br />then<br /> echo &quot;wrong syntax. use vod.sh dest message&quot;<br /> exit<br />fi<br />user=9100000; pass=“yourpassword”; dest=$1 ; msg=$2<br />rm -f cookies.txt<br />wget -O - --keep-session-cookies --save-cookies cookies.txt &quot;http://www.vodafone.pt/main/myVodafone/&quot; &gt;-<br />wget -O - --load-cookies cookies.txt --keep-session-cookies --save-cookies cookies.txt &quot;--post-data=userid=$user&password=$pass&sru=https%3A%2F%2Fmy.vodafone.pt%2Fpm%2FSPMDispatcher.aspx%3FPMcmd%3D17%26userClass%3D10%26Guid%3D%7BB9DED956-B87F-4C24-852C-70A3BBBB0161%7D%26ReturnUrl%3Dhttps%253a%252f%252fmy.vodafone.pt%252fguest%252fhomepagePre.htm&fru=https%3A%2F%2Fmy.vodafone.pt%2Fguest%2FhomepagePre.htm&svc_id=myprodpub&quot; “https://id.vodafone.pt/ucp//auth/login.asp?&ou=&crypt=0&prf=0&key=alias“ &gt;-<br />wget -O - --load-cookies cookies.txt --keep-session-cookies --save-cookies cookies.txt &quot;http://mysms.vodafone.pt/rules/sms/sms_envio.asp&quot; &gt;-<br />wget -O - --load-cookies cookies.txt --keep-session-cookies --save-cookies cookies.txt &quot;--post-data=mydate3=&indicativo=91&telefone=&mensagem1=$msg&programado=nao&h2=1&phones=$dest%2F&prog=&dataf=12%2F29%2F2009+10%3A58%3A23+PM&agt=&quot; <br /> &quot;--referer=http://mysms.vodafone.pt/rules/sms/sms_envio.asp&quot; “http://mysms.vodafone.pt/rules/sms/sms_envio.asp?submit=ok” &gt;- <br />
  23. 23. Accessingan online Bankandgetting data<br />03-12-2009<br />23<br />datainicial=&quot;2009-11-01&quot;<br />maxdias=$((365*5))<br />hoje=`date +%Y-%m-%d`<br />fundo=&quot;BPI Portugal&quot;<br />for i in`seq 0 $maxdias`; do<br />data=`date +%Y-%m-%d -d &quot;$datainicial + $i day&quot;`<br />if [ &quot;$data&quot; == &quot;$hoje&quot; ]; thenbreakfi<br />readvaldt&lt; &lt;(GET &quot;http://www.bpiinvestimentos.pt/Fundos/QuadroCotacoesfundos.asp?opc=1&DataPesquisa=$data&quot; | grep -A 2 &quot;$fundo&quot;| egrep -o &quot;[0-9]{2}-[0-9]{2}-[0-9]{4}|[0-9]{1,2},[0-9]{1,}&quot; |tr &quot; &quot; &quot; &quot;| awk &apos;{print $2,$1}‘ )<br />val=$(echo$val | tr &quot;,&quot; &quot;.&quot;)<br />dt=$(echo$dt | tr &quot;-&quot; &quot; &quot;| awk &apos;{print $3,$2,$1}&apos; | tr &quot; &quot; &quot;-&quot;)<br />psql -q -h localhost -U postgres -c &quot;deletefromcotacoeswherefundo=&apos;$fundo&apos; anddata=&apos;$dt&apos;&quot; fundos<br />psql -q -h localhost -U postgres -c &quot;insertintocotacoes (fundo,data,valor) values (&apos;$fundo&apos;,&apos;$dt&apos;,$val)&quot; fundos<br />done<br />
  24. 24. EXTRA STUFF<br />If we have time, we gonna take a bit of…<br />03-12-2009<br />24<br />
  25. 25. “here” / inlinedocuments<br />A heredocumentallowsthespecificationof a blockofthatcanbefed to a variable, to BASH functionsordirectly to the STDIN of a program<br />03-12-2009<br />25<br />multiline_variable=$(cat&lt;&lt;END_DELIMITER<br />a<br />b<br />END_DELIMITER)<br />echo “$multiline_variable”<br />./some_app&lt;&lt;SEQUENCE_OF_APP_COMMANDS<br />do this<br />do that<br />…<br />SEQUENCE_OF_COMMANDS<br />
  26. 26. Processsubstitution<br />“A kind of a reverse pipe”<br />the output of a command (list) is redirected to a temporary file descriptor which can be used as the STDIN (directly or indirectly through an argument) of a program<br />03-12-2009<br />26<br />echo&lt;( ls / )<br /> /dev/fd/63<br />cat&lt;( ls / )<br /> /home<br /> …<br /># thefollowinglinewouldbe “hard” to do withpipes, unlessusing a subshelllike (ls /; ls /home) | grep “o”<br />grep “o” &lt;( ls / ) &lt;( ls /home)<br /> /dev/fd/63:boot<br /> /dev/fd/63:cdrom<br /> /dev/fd/62:sergio<br /> /dev/fd/62:sergio<br /> …<br />
  27. 27. References/Further reading<br />AdvancedBash-ScriptingGuide<br />http://tldp.org/LDP/abs/html/<br />Linux Shell Scripting Tutorial - A Beginner&apos;s handbook<br />http://bash.cyberciti.biz/guide/Main_Page<br />Snipt<br />http://snipt.net/public/tag/bash<br />Bash by example, IBM developerWorks<br />http://www.ibm.com/developerworks/library/l-bash.html<br />03-12-2009<br />27<br />
  28. 28. Thanks / Obrigado<br />By:<br />Sergio Freire<br />Mensagens e RedesConvergentes / Departamento de Redes e Protocolos<br />Tel: +351 234 403609<br />sergio-s-freire@ptinovacao.pt<br />Thisdocumentisintellectualpropertyof PT Inovação, SA andshouldnotbeused without express written permission.<br />