3. Introduction
Why use the shell?
(copy/paste from “Why automate?” slide from “Getting Started with Puppet - PuppetConf
2013” https://www.youtube.com/watch?v=TdAmAj3eaFI)
•Speed
•Consistency
•To get to fun things
3
4. PLAN
Demonstration and quick displays of code on the following
topics:
1.Start simple: analyse some logs
2.Write your own scripts
3.Reuse your scripts to do automation
5. 5
1. Start simple: to analyse logs
Read the logs of one batch application
•Find out if there are any errors
•List the document ids in error
Check whether those documents exists in the collection
6. 6
Useful commands
Ls
Echo
wc -l
Cat / more / less
touch
Vi / vim / emacs
Grep / egrep / zgrep
Find
Awk
Sed
Cut
Xargs
Host
Wget
Curl
Watch
ps
Mongo
Java
Mysql
git
8. 8
A few hints
• Add colours to your shell
• If you do something frequently…….
…… write a script
• Get the job done,
do not polish it,
you will get to it on next time
9. 9
Happy Consequences
• You do not prevent yourself from writing scripts
because it is going to take ages
• You progress and learn gradually, over time
• You stop looking for big tools, you answer simple needs
with simple solutions
> Example: Monitoring tool for daily updates on categories
10. 10
3. Reuse your scripts to do automation
Sample with the Monitoring tool
Cron (scheduled task) ran every day
Script to monitor: monitorDailyUpdates.sh
Dependencies:
• checkDailyUpdates.sh
• sendMail.sh
Retrieve all document references that could not be processed in taxonomy batch application.
grep "document could not be processed because it was not found in the index" /tmp/processQueueMessages.log | awk '{print $19}' | cut -c -7 | sort | uniq
grep "document could not be processed because it was not found in the index" /tmp/processQueueMessages.log | awk '{print $19}'| sed "s/,//g" | head -3 | xargs -I {} curl --silent "http://localhost:8985/solr/IAVIEWS_SNAPSHOT/select?q=DOCREFERENCE%3A{}" | egrep "(numFound|DOCREFERENCE)"
sendMail.sh “email@department.gov.uk" "Untagged Docs on series WO 185" "$(echo "$(cat untaggedDocsOnSeries-WO_185-20150605.json)" | awk ' {print $0"<br/>"}')“