Linux Shell Scripting Craftsmanship

1,977 views
1,885 views

Published on

This prentation describes quality in the context of commonly understood (but under-appreciated) Unix programming best practices in three general categories.
 Transparency: The ease with which a script can be understood by reading the code
 Clear communication: How well the script informs the user of its activities
 Scalability: Whether the script can be used across the enterprise without intervention

0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,977
On SlideShare
0
From Embeds
0
Number of Embeds
10
Actions
Shares
0
Downloads
63
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide

Linux Shell Scripting Craftsmanship

  1. 1. Shell Scripting Craftsmanship Ray Smith Portland General Electric if [ You can't read this easily ]; then move up closer else Don't say I didn't warn you fi
  2. 2. Objectives • Keep you awake until the Opening Keynote • Philosophical alignment • Context of quality • Define craftsmanship • What makes a good quality desk? • What are the common elements of good scripts?
  3. 3. • Rule #1: SHELL SCRIPTS MUST WORK • High-visibility tasks • Unforgiving tasks • Repetitive tasks • Rule #2: SCRIPTS MUST KEEP WORKING • Nothing stays the same • Plan on it
  4. 4. Quality, craftsmanship, harmony • Transparency • Maintenance without frustration • Clear communication • Effort without agitation • Scalability • Power without intervention
  5. 5. Spiritual Guidance • The Art of Unix Programming • Eric S. Raymond • Addison-Wesley • Resident anthropologist and roving ambassador of the open source movement • Three decades of unwritten, hard-won software engineering wisdom from 13 Unix pioneers
  6. 6. Transparency • Rule of Robustness • Robustness is the child of transparency and simplicity • Rule of Transparency • Design for visibility, inspection, and debugging • Rule of Simplicity • Design for simplicity • Add complexity only where you must • Rule of Clarity • Clarity is better than cleverness
  7. 7. Clear Explanations • Be generous with internal documentation • Particularly when being clever or ultra-efficient # Find all instances of the string find . –type f –exec fgrep –i “$mySTRING” /tmp/dummy {} ; 2>/dev/null • Document dead-ends and surprises • Programming methods that were tried and did not work • „Obvious‟ data sources that are unreliable
  8. 8. Predictable Layout • Consistent layout for all your scripts 1. Header block 2. Change log 3. Independent variables 4. Dependent variables 5. Functions 6. Run-time procedure • Take out the guesswork
  9. 9. Predictable Layout #!/bin/bash ######################################################### # File : sample_script.sh # Input values : Database name (optional) # Purpose : Amaze others ######################################################## #======================================================= # Independent variables #======================================================= export BASEDIR=/usr/lbin/orascripts #======================================================= # Dependent variables # Nothing to change below this line #======================================================= LOGDIR=$BASEDIR/logs WORKFILE=$LOGDIR/workfile.lst
  10. 10. Visual Simplicity • Be kind to yourself and others • Layouts and formatting USE WHITESPACE Break up long lines with back-slash
  11. 11. Just like a SQL statement … select distinct table_name,column_name from all_tab_columns where column_name like '%AGREE%' or column_name like '%ANN%„ or table_name like '%ANN%„ or table_name like „%STOR%„ order by table_name,column_name; SELECT DISTINCT table_name,column_name FROM all_tab_columns WHERE column_name LIKE '%AGREE%' OR column_name LIKE '%ANN%„ OR table_name LIKE '%ANN%' OR table_name LIKE '%STOR%' ORDER BY table_name,column_name;
  12. 12. Emphasize Visual Flow for thisHOST in `cat ${HOSTLIST}`; do if [ ${#thisHOST} -gt 5 ]; then echo "BIG: ${thisHOST} is ${#thisHOST} characters" else if [ ${#thisHOST} -lt 3 ]; then echo "LITTLE: ${thisHOST} is ${#thisHOST} characters" fi fi done for thisHOST in `cat ${HOSTLIST}`; do if [ ${#thisHOST} -gt 5 ]; then echo "BIG: ${thisHOST} name is long" else if [ ${#thisHOST} -lt 3 ]; then echo "LITTLE: ${thisHOST} name is short" fi fi done
  13. 13. Discoverability • Make ${VARIABLES} stand out in your code • Variable naming conventions • ALL_CAPS for session variables • CamelCase for function names • thisVARIABLE for looped variables • Be internally consistent • Adopt a standard and follow it
  14. 14. Outstanding Variables for thishost in `cat $hostlist`; do if [ $#thishost -gt 5 ]; then longmessage else if [ $#thishost -lt 3 ]; then shortmessage fi fi done for thisHOST in `cat ${HOSTLIST}`; do if [ ${#thisHOST} -gt 5 ]; then LongMessage else if [ ${#thisHOST} -lt 3 ]; then ShortMessage fi fi done
  15. 15. The Penny Wise Quiz a. Shorter variable names = Less typing = Less work b. Obscure variable names = Reduced transparency = Poor quality • Save your cycles for decyphering the logic • Not the variable names
  16. 16. Tuning for Transparency • Rule of Optimization • Prototype before polishing • Get it working before you optimize it • Rule of Modularity • Write simple parts connected by clean interfaces • Functions • Temporary files
  17. 17. Efficiency • Use shell script functions function SetPerms770 { echo “nSetting permission to 770 for ${thisFILE}” chmod 770 ${thisFILE}/* } > for thisFILE in `cat ${FILELIST}`; do > SetPerms770 > done • Modularize all repeated code statements
  18. 18. Handiest function ever • At the beginning of the script • At the end of the script • Any time execution might end function CleanUpFiles { [ $LOGFILE ] && rm –f ${LOGFILE} [ $SPOOL01 ] && rm –f ${SPOOL01} }
  19. 19. # Functions # ------------------------------------------------------------------- function CopyFiles { cd $SOURCE_DIR ls -1 | grep -i $ORACLE_SID >$WORKFILE for thisFILE in `cat $WORKFILE`; do SOURCE_FILE=$SOURCE_DIR/$thisFILE TARGET_FILE=$TARGET_DIR/$thisFILE cp –f $SOURCE_FILE $TARGET_FILE done rm -f ${WORKFILE} } # ------------------------------------------------------------------- # Run-time procedure # • Modularize all repeated code statements ------------------------------------------------------------------- SOURCE_DIR=${GOLD_DIR}/rman TARGET_DIR=${LIVE_DIR}/rman CopyFiles SOURCE_DIR=${GOLD_DIR}/security TARGET_DIR=${LIVE_DIR}/security CopyFiles
  20. 20. Transparency Zen • Rule of Diversity • Distrust all claims of “one true way” • Including your own • Revisit older scripts • Technologies change • Techniques change
  21. 21. The Secret to the (Scripting) Universe Steal Adapt Improve Repeat
  22. 22. Transparency Wrap-up Maintenance without frustration: • Written with maintenance and on-call in mind • Provide useful guidance and instructions • Reflect visual simplicity and clear layout Comments or questions
  23. 23. I’m awake I’m awake
  24. 24. User Communication • Rule of Silence • Nothing surprising to say: say nothing • Rule of Repair • Repair what you can • When you must fail, fail noisily and as soon as possible • Rule of Least Surprise • In interface design, do the least surprising thing • Do the most expected thing
  25. 25. Work with the user • Verify scripted actions echo “These files will be backed up:” cat ${FILELIST} • Keep the user informed • Share status and decisions echo “You asked to delete everything in /etc” read -p "Is this correct? [Y|N]" myVal case $myVal in . . .
  26. 26. Opatch is a gem Running prerequisite checks... OPatch detected non-cluster Oracle Home from the inventory and will patch the local system only. Please shutdown Oracle instances running out of this ORACLE_HOME on the local system. (Oracle Home = '/cisp_orabase/product/10.2.0') Is the local system ready for patching? [y|n] User Responded with: Y Applying patch 7155252... ApplySession applying interim patch '7155252' to OH '/cisp_orabase/product/10.2.0' Backing up files affected by the patch '7155252' for rollback. This might take a while...
  27. 27. Graciousness • Handle errors with grace • Explain the situation • Lead the user to a solution if [ ${#1} -eq 0 ];then echo "The required value for database name" echo "was not passed in the command line" read -p "Enter the database name: " myVal export thisSID=$myVal fi
  28. 28. Communicating Failure • Cryptic feedback is not welcome nor helpful > > FATAL ERROR: bckpinit_strt error 12 > • Terminal output is free, use it if you need it > > File not found: bckpinit_strt.conf > > Corrective action required: > Verify bckpinit_strt.conf exists at ABC > and readable by user xyz > Email notification has been sent
  29. 29. Signs of Life • Same script should work in cron or interactive • Test for tty (terminal id) if tty –s then echo “Oh good, this is interactive” echo “Hello $USER” else date +“Script $0 started %T” >> $LOGFILE fi
  30. 30. Artist and Psychologist • Break the output into usable blocks • Use n and t liberally • If it makes it easier for user to understand • Particularly important for interactive scripts • Push the „read‟ statement into their attention • “Is the local system ready for patching? [y|n]” • Direct their eyes with breaks and groups • Apply the same practices to log files and reports
  31. 31. Electronic Communication • Be complete, be clear • Which host • Which process / job • What happened (in prose) • How to troubleshoot / resolve the problem • Start communicating in the subject line Subject: dbtest1:pg0t log_blaster.ksh FAILURE
  32. 32. Dialog works in any terminal emulator dialog --menu "Select the database instance“ 0 50 5 NICKEL "Five Cent Database" URANIUM "Not-For-Export Database" CUSTOM "User defined instance“
  33. 33. Zenity is cooler, but requires X-server zenity --list --text "Select the database instance" --column "SID" --column "Description" "NICKEL" "Five Cent Database" "URANIUM" "Not-For-Export Database" "CUSTOM" "User defined instance“
  34. 34. Communication Wrap-up Effort without agitation: • Work with the user • Generous with visual guidance and produce attractive, useful log files and reports • Clear and complete feedback Comments or questions
  35. 35. So this is Haight Ashbury
  36. 36. Scalability • Rule of Extensibility • Design for the future • Plan to grow • Scalability goal #1: Never customize a script • Overuse variables • Never hardcode • Passwords • Host or database names • Paths • Use command-line input, „read‟, or parameter files • Balance risk vs. maintenance cost
  37. 37. Your Goal: Stability and Predictability • Consistency • Use the same code across your entire enterprise • Security • Limit editing permissions • 'Them' and you, too • Revision control • Keep a gold copy out there • cfengine
  38. 38. Use the Command Line #!/bin/bash ${0} ${1} ${2} thisSCRIPT=$0 > backup_db.sh silver hot ORACLE_SID=$1 BU_TYPE=$2 echo “Executing ${thisSCRIPT} for ${thisSID}” if [ ${BU_TYPE} == hot ]; then echo “tRunning a hot backup” TestForArchiveMode else RunColdBackup fi
  39. 39. Command Line Illustration #!/bin/bash # file: input_variables.sh if [ $1 ] ; then echo "Amaze your $1" else echo “provide input, pathetic human!" fi > > ./input_variables.sh friends > Amaze your friends
  40. 40. Command Line Upside • Nothing inside the tested script changes • No surprises • No customizations • One script across the enterprise • Securable • Does not require write permissions for others
  41. 41. Command Line Downside • Inflexible • If three variables are needed, three must appear • Very easy to make a run-time error • Not too bad for cron jobs • Can be irritating for ad hoc executions • Gets out-of-hand with too many variables > ./many_input_variables.sh one two three four five six seven eight nine ten eleven
  42. 42. Strength and Flexibility • Use „getopts‟ for command line options • Define your own parameters • No need for parameter files or script editing • Accepts defined flags or input values • Flexible and scalable • Loops through multiple flags • Parses the command line for hyphens
  43. 43. Syntax while getopts ds: thisValue; do case $thisValue in d) echo "Using default values" FILE_NAME=abc123.lst;; s) echo "Making the file $OPTARG Gb" FILE_SIZE=${OPTARG};; esac done
  44. 44. Getopts example while getopts ds: thisValue; do case $thisValue in d) echo "Using default values" FILE_NAME=abc123.lst;; s) echo "Making the file $OPTARG Gb" FILE_SIZE=${OPTARG};; esac Done  script_name.sh –d –s 20  Using the default values  Making the file 20 Gb
  45. 45. Getopts Considerations • Very flexible • Flags determine which variables are used • Not position-sensitive • Except if used with non-getopts variables • Requires more scripting and testing • Same amount of validation • Opportunities to use functions • Man page is good when you have the concept • Steal examples on-line and adapt as needed
  46. 46. Channel the Power • Rule of Economy • Use machine resources instead of typing • Rule of Generation • Avoid hand-hacking • Write programs to write programs when you can • Just like dynamic SQL
  47. 47. Make the Machine do the Work • Create everything you need, every time • Fix permissions too • Before and after pictures = Acts of Kindness if [ ! -d $thisDIR ]; then mkdir $thisDIR fi echo “Directory before:” ls –l chmod 775 $thisDIR echo “Directory after:” ls -l
  48. 48. Resourcefulness • Single-point maintenance • Central script repository • Common „data‟ files • Host list • SID list • Use existing files • /etc/passwd, var/opt/oracle/oratab • Create your own common files • Environment profiles
  49. 49. Scalability Wrap-up Power without intervention: • Editing is never required for a new host / installation • Expected problems are handled from the start • Existing resources are used Comments or questions
  50. 50. Quality, craftsmanship, harmony • Transparency for maintenance and clarity • Maintenance without frustration • Clear communication with the user • Effort without agitation • Scalability without edits • Power without intervention
  51. 51. You are a Unix Programmer “To do the Unix philosophy right, you have to be loyal to excellence. You have to believe that software design is a craft worth all the intelligence, creativity, and passion you can muster.” -- Eric S. Raymond
  52. 52. Further Reading The Art Of Unix Programming Eric S. Raymond Addison-Wesley Available at: www.powells.com

×