2010 Smith Scripting101

1,314 views

Published on

The plumbing metaphor goes back 40 years to the beginning of Unix/Linux and still works today.
In this session the fundamentals of shell scripting will be illustrated through a cumulative example built on pipes, filters, valves, and screens. Environmentally friendly waste cleanup will also be covered.

  • Be the first to comment

  • Be the first to like this

2010 Smith Scripting101

  1. 1. Linux Shell Scripting 101 Plumbing for DBAs Ray Smith Portland General Electric Copyright 2010 Raymond S. Smith
  2. 2. Agenda • Why Plumbing • Tools • Fittings and Parts • Procedures • Leaks and Repairs
  3. 3. The Plumbing Metaphor • Part of the Unix tradition – 40 year track record • Part of the lexicon – Pipes | – Redirects > – Input and Output – Buckets and spools
  4. 4. Part 1: Tools • Books and man pages • Spare parts • Stuff in the truck • Other plumbers
  5. 5. Most Important Tool: Nutshells
  6. 6. Handbooks Ken O. Burtch Anatole Olczak
  7. 7. Steal, Adapt, Repeat • If it works – Find out why it works – Test it for your problem – Apply it
  8. 8. Ask for Help • Hit the blog-o-sphere • Ask your SA – Have an example in hand • Ask other DBAs • Hit the man page David Korn (ksh)
  9. 9. Part 2 : Fittings and Parts • Input and Output • Pipes • Valves • Filters • Buckets
  10. 10. Input and Output • Unix Fundamental Rules – Simple, predictable interfaces – Text/string input expected – Text output anticipated – Something happens in between Text In Text Out
  11. 11. grep “string” grep –i grep –v cut –d: -f2 awk –F:„{print $2}‟
  12. 12. Redirect out example echo “Hello, world” > anyfile.txt cat anyfile.txt Hello, world echo “Collaborate OK?” >> anyfile.txt cat anyfile.txt Hello, world Collaborate OK?
  13. 13. Redirect ‘in’ example mailx –s “Collaborate” ray@mail.com < anyfile.txt
  14. 14. Subshell redirection sqlplus / nolog << EOF connect / as sysdba shutdown abort exit EOF • Subshell redirection – Opens a subshell to your shell script – Executes the commands – Exits when the matching ‘EOF’ is seen • EOF must be on left column of text file • Use this syntax for sqlplus, RMAN, etc
  15. 15. Shell scripting is pipe work echo “Hello, world” > anyfile.txt echo “Collaborate OK? >> anyfile.txt echo “Collaborate OK? >> anyfile.txt cat anyfile.txt | sort –u Collaborate OK? Hello, world cat anyfile.txt | grep OK Collaborate OK? cat anyfile.txt | grep OK | cut –f2 OK?
  16. 16. Part 3: Procedures • How to write a shell script – Define the output – Define the input – Fill in the gaps
  17. 17. oratab Only 10g hostname Skip comments Wanted: List of 10g databases Report
  18. 18. Sample oratab script #!/bin/bash WORKFILE=/tmp/workfile.lst REPORT=/tmp/oratab_report.rpt export HOSTNAME=`hostname` touch $REPORT cat /etc/oratab | grep “10.” | grep –v ^# >$WORKFILE for thisENTRY in `cat $WORKFILE`; do thisSID=`cat $thisENTRY | cut –d: –f1` echo “$thisSID is on $HOSTNAME” >>$REPORT done cat $REPORT
  19. 19. Sample oratab script #!/bin/bash WORKFILE=/tmp/workfile.lst •REPORT=/tmp/oratab_report.rpt Interpreter callout export HOSTNAME=`hostname` touch $REPORT cat /etc/oratab | grep “10.” | grep –v ^# >$WORKFILE for thisENTRY in `cat $WORKFILE`; do thisSID=`cat $thisENTRY | cut –d: –f1` echo “$thisSID is on $HOSTNAME” >>$REPORT done cat $REPORT
  20. 20. Sample oratab script #!/bin/bash WORKFILE=/tmp/workfile.lst REPORT=/tmp/oratab_report.rpt export HOSTNAME=`hostname` •touch $REPORT declaration Variables cat /etc/oratab | grepOnly forgrep –v ^# >$WORKFILE • No ‘export’ : “10.” | this script for thisENTRY ‘export’: Session variable • With in `cat $WORKFILE`; do thisSID=`cat $thisENTRY | cut –d: –f1` echo export is on $HOSTNAME” >>$REPORT “$thisSID WORKFILE=xyz.lst done cat $REPORT
  21. 21. Sample oratab script •#!/bin/bash ^# Grep -v WORKFILE=/tmp/workfile.lst • Dash v = ignore REPORT=/tmp/oratab_report.rpt export HOSTNAME=`hostname` • Caret = starts with touch $REPORT cat /etc/oratab | grep “10.” | grep –v ^# >$WORKFILE for thisENTRY in `cat $WORKFILE`; do #/etc/oratab thisSID=`cat $thisENTRY | cut –d: –f1` echo “$thisSID is on $HOSTNAME” >>$REPORT ##Production:Sales:18-6:10.2.0.4:Morgan done salesp:/oracle/product/10.2.0:Y cat##Production:Finance:24-7:11.1.0.7:Ames $REPORT finp:/oracle/product/11.2.0:Y
  22. 22. Do loops for thisENTRY in `cat $WORKFILE`; do thisSID=`cat $thisENTRY | cut –d: –f1` echo “$thisSID is on $HOSTNAME” >>$REPORT done • For something in something else • Do something • Quit when I say you’re done for thisENTRY in file1.txt file2.txt file3.txt; do thisSID=`cat $thisENTRY | cut –d: –f1` echo “$thisSID is on $HOSTNAME” >>$REPORT done
  23. 23. Manipulating strings w/ cut cat $thisENTRY | cut –d: –f1 • Cut the string using • Delimiter (-d) of colon • Return field (-f) -f1 salesp:/oracle/product/10.2.0:Y
  24. 24. Manipulating strings w/ awk cat $thisENTRY | awk –F: „{print $1} • Cut the string using • Delimiter (-F) of colon • Print first string ( print $1 ) ${1} salesp:/oracle/product/10.2.0:Y
  25. 25. Database No filter Email address Wanted: Oracle accounts with DBA role Mail
  26. 26. SQL query script #!/bin/bash SPOOL01=/tmp/spoolfile_01.lst DISTRIBUTION=“myemail@someplace.com” [ $SPOOL01 ] && rm –f ${SPOOL01} sqlplus / as sysdba <<EOF spool ${SPOOL01} SELECT grantee FROM dba_role_privs WHERE granted_role = „DBA‟; spool off exit EOF mailx –s “DBA grantees” ${DISTRIBUTION} <${SPOOL01} [ $SPOOL01 ] && rm –f ${SPOOL01}
  27. 27. RMAN script #!/bin/bash RMN_SCRIPT=/tmp/rman_script.rmn WORKFILE=/tmp/rman_workfile.lst DISTRIBUTION=“myemail@someplace.com” echo "connect target /" > ${RMN_SCRIPT} echo "connect catalog ${USER}/${PASS}@${RMANCAT}“ >>${RMN_SCRIPT} echo "resync catalog;" >>${RMN_SCRIPT} echo "exit;" >>${RMN_SCRIPT} rman @RMN_SCRIPT <<EOF | tee –a $WORKFILE exit EOF mailx –s “RMAN execution” ${DISTRIBUTION} <${SPOOL01}
  28. 28. Pipe to a Tee
  29. 29. OEM emcli script #!/bin/bash BO_NAME=scripted_blackout_${thisSID} echo "Creating blackout named '${BO_NAME}' ..." ${EMCLI} create_blackout -name="${BO_NAME}" -add_targets=${thisTARGET}:oracle_database -schedule="duration::360;tzinfo:specified;tzregion:America/Los_Angeles" -reason="Scripted blackout for maintenance or refresh" sleep 5 $ECHO "Getting blackout information for '${BO_NAME}' ..." ${EMCLI} get_blackout_details -name="${BO_NAME}"
  30. 30. Part 4: Leaks and Repairs Rule #1: SHELL SCRIPTS MUST WORK Rule #2: SCRIPTS MUST KEEP WORKING Design for maintenance – Clarity – Simplicity – Scalability
  31. 31. Predictable Layout • Consistent layout for all your scripts 1. Header block 2. Change log 3. Independent variables 4. Dependent variables 5. Functions 6. Run-time procedure • Take out the guesswork
  32. 32. Layout example #!/bin/bash #set -x ######################################################### # File : sample_script.sh # Parameter : Database name (optional) # Purpose : Amaze others ######################################################## #======================================================= # Independent variables #======================================================= export BASEDIR=/usr/lbin/orascripts #======================================================= # Dependent variables # Nothing to change below this line #======================================================= LOGDIR=$BASEDIR/logs WORKFILE=$LOGDIR/workfile.lst
  33. 33. Visual clarity
  34. 34. Just like a SQL statement … select distinct table_name,column_name from all_tab_columns where column_name like '%AGREE%' or column_name like '%ANN%„ or table_name like '%ANN%„ or table_name like „%STOR%„ order by table_name,column_name; SELECT DISTINCT table_name,column_name FROM all_tab_columns WHERE column_name LIKE '%AGREE%' OR column_name LIKE '%ANN%„ OR table_name LIKE '%ANN%' OR table_name LIKE '%STOR%' ORDER BY table_name,column_name;
  35. 35. Emphasize Visual Flow for thisHOST in `cat ${HOSTLIST}`; do if [ ${#thisHOST} -gt 5 ]; then echo "BIG: ${thisHOST} is ${#thisHOST} characters" else if [ ${#thisHOST} -lt 3 ]; then echo "LITTLE: ${thisHOST} is ${#thisHOST} characters" fi fi done for thisHOST in `cat ${HOSTLIST}`; do if [ ${#thisHOST} -gt 5 ]; then echo "BIG: ${thisHOST} name is long" else if [ ${#thisHOST} -lt 3 ]; then echo "LITTLE: ${thisHOST} name is short" fi fi done
  36. 36. The Penny Wise Quiz a. Shorter variable names = Less typing = Less work b. Obscure variable names = Reduced transparency = Poor quality • Save your cycles for decyphering the logic – Not the variable names
  37. 37. Shell script functions function SetPerms770 { echo “Setting permission to 770 for ${thisFILE}” chmod 770 ${thisFILE}/* } > for thisFILE in `cat ${FILELIST}`; do > SetPerms770 > done
  38. 38. function CopyFiles { cd $SOURCE_DIR ls -1 | grep -i $ORACLE_SID >$WORKFILE for thisFILE in `cat $WORKFILE`; do SOURCE_FILE=$SOURCE_DIR/$thisFILE TARGET_FILE=$TARGET_DIR/$thisFILE cp –f $SOURCE_FILE $TARGET_FILE done rm -f ${WORKFILE} } # ----------------------------------------------------------- # Run-time procedure # ------------------------------------------------------------ SOURCE_DIR=${GOLD_DIR}/rman TARGET_DIR=${LIVE_DIR}/rman CopyFiles
  39. 39. Scalability • Scalability goal #1: Never customize a script – Overuse variables – Never hardcode • Passwords • Host or database names • Paths – Use command-line input, ‘read’, or parameter files • Balance risk vs. maintenance cost
  40. 40. Command Line Values #!/bin/bash ${0} ${1} ${2} thisSCRIPT=$0 > backup_db.sh silver hot ORACLE_SID=$1 BU_TYPE=$2 echo “Executing ${thisSCRIPT} for ${thisSID}” if [ ${BU_TYPE} == hot ]; then echo “tRunning a hot backup” TestForArchiveMode else RunColdBackup fi
  41. 41. Command Line Illustration #!/bin/bash # file: input_variables.sh if [ $1 ] ; then echo "Amaze your $1" else echo “provide input, pathetic human!" fi > > ./input_variables.sh friends > Amaze your friends
  42. 42. No tools required • Nothing inside the tested script changes – No surprises – No customizations – One script across the enterprise • Securable – Does not require write permissions for others
  43. 43. Make the Machine do the Work • Create everything you need, every time – Fix permissions too – Before and after pictures = Acts of Kindness if [ ! -d $thisDIR ]; then mkdir $thisDIR fi echo “Directory before:” ls –l chmod 775 $thisDIR echo “Directory after:” ls -l
  44. 44. Thanks Complete evaluation • Session 999 ray.smith@pgn.com

×