Geecon 2019 - Taming Code Quality in the Worst Language I Know: Bash
•Download as PPTX, PDF•
0 likes•66 views
I don't know any other languages with more pitfalls, perils and gotchas than Bash. Still, we use it in almost every larger project for deployment or maintenance scripts, because there is no better, more powerful and more universal choice on Unix platform. However, there is ridiculous amount of things that could go wrong if you don't have deep understanding of shell scripting. Your experience about typical issues with Java or other JVM languages is definitely not enough here. You need to deeply understand Linux ecosystem and its history in order to write correct script... or you don't? I will prove to you that Bash could be tamed and made easy if proper code quality standards and static analysis tools are applied and enforced in your delivery pipelines. I'll share my opinions and experiences from a large banking project and I'll tell you which tools and style guides we use.
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Geecon 2019 - Taming Code Quality in the Worst Language I Know: Bash
Similar to Geecon 2019 - Taming Code Quality in the Worst Language I Know: Bash (20)
Recently uploaded
Recently uploaded (20)
Geecon 2019 - Taming Code Quality in the Worst Language I Know: Bash
- 1. www.luxoft.com Taming Code Quality in the Worst Language I Know: Bash Michal Kordas
- 2. www.luxoft.com whoami • Passion for Java, Groovy, JVM • “Quality Guardian” at Luxoft for the last 6 years • Bringing feedback loops to the micro-level • Focus on well-written code and efficient processes • Contributor to the open source and StackOverflow • Trainer at the Luxoft Training Center @michal_kordas
- 4. www.luxoft.com ./download-feeds --target-directory … To make script idempotent there was rm -rf inside...
- 10. www.luxoft.com
- 11. www.luxoft.com
- 12. www.luxoft.com
- 13. www.luxoft.com
- 14. www.luxoft.com
- 15. www.luxoft.com
- 16. www.luxoft.com Kraków Built by Bash Programmers
- 20. www.luxoft.com
- 21. www.luxoft.com
- 22. www.luxoft.com Globbing msg="Please enter a file name matching *.zip" echo $msg Please enter a file name matching cute_cats.zip naked_girls.zip
- 23. www.luxoft.com
- 24. www.luxoft.com cmd $var in another language would be cmd(glob(split($var))) cmd "$var" in another language would be cmd($var)
- 25. www.luxoft.com
- 26. www.luxoft.com 26 ● correct quoting ● word splitting ● globbing ● stream redirection ● pipe failure ● command substitution ● brace expansion ● parameter expansion ● file clobbering Tricky Bits of Bash
- 27. www.luxoft.com 27 ●Slowness ●No data structures (forget about queue, stack or tree) ●Being stringly typed (ok, we have arrays) ●No try/catch ●No exceptions ●Functions cannot return values (just streams) ●Cannot pass reference no variable, array or function ●Global function names - reusability horror Bash Weaknesses
- 29. www.luxoft.com When to Use Bash • In wrapper scripts • To interact with processes • There is no complex data manipulation • Performance doesn’t matter • Interaction with OS Use others when • Logic requires arrays • Script has more than 100 lines (and scripts tend to grow)
- 30. Bash is useful only as a glue… ...still, it’s the best glue available
- 32. man bash man man help help help info info info man -a intro Other reference commands:
- 33. www.luxoft.com Bash vs Sh GNU Bash • Most known • Deployed on almost all *nixes • Extended version of POSIX • One implementation → Bourne Again Shell POSIX Shell • More portable • Standardized • Limited • Multiple implementations → ksh88 → dash → Bourne Shell #!/bin/bash #!/usr/bin/env bash #!/bin/sh #!/usr/bin/env sh
- 34. www.luxoft.com Why Use Bash Local variables (in POSIX all are global) Array support [[ instead of [ Arithmetic loops "for ((i=0; i<10; i++)); do … ; done“ function f { → no reason to use
- 35. ... reading the UNIX source code to the Bourne shell I was shocked at how much simple algorithms could be made cryptic, and therefore useless, by a poor choice of code style. I asked myself, "Could someone be proud of this code?" --Landon Noll
- 37. What Can We Do?
- 38. www.luxoft.com Apply Rules From Other Languages • Use descriptive names ○ ec=1 #cryptic error_code=1 #understandable • Break into simple functions with proper names • Use simple syntax and avoid redundancy ○ COMMAND if [ $? -eq 0 ] # Complex and redundant if COMMAND # Simple
- 39. www.luxoft.com Style Guides • Unofficial Shell Scripting Stylesheet • http://tldp.org/LDP/abs/html/unofficialst.html (Advanced Bash-Scripting Guide) • Bash Style Guide • https://github.com/bahamas10/bash-style-guide • Bash Coding Style • https://github.com/icy/bash-coding-style • Scripting With Style • https://wiki-dev.bash-hackers.org/scripting/style • Shell Style Guidelines • http://www.chromium.org/chromium-os/shell-style-guidelines • Community Bash Style Guide • https://github.com/azet/community_bash_style_guide
- 40. www.luxoft.com Google Shell Style Guide
- 41. www.luxoft.com Which Shell to Use? Bash is the only shell scripting language permitted for executables
- 42. www.luxoft.com Extensions Executables (no extension) executable Libraries (.sh extension) library.sh
- 43. www.luxoft.com Comments File Header • Start each file with a description of its contents Function Comments • Any function that is not both obvious and short must be commented. Any function in a library must be commented regardless of length or complexity Implementation Comments • Comment tricky, non-obvious, interesting or important parts of your code #!/usr/bin/env bash # # Perform backup of Jenkins configuration ####################################### # Cleanup files from the backup dir # Arguments: # None # Returns: # None ####################################### cleanup() { … }
- 44. www.luxoft.com Naming Conventions Global variables: UPPER_SNAKE_CASE • Prefer constants and use readonly • Top of the file Local variables, function names, file names: lower_snake_case • Use local Order in a file • constants • functions • main() function • executable code #!/usr/bin/env bash readonly CONSTANT=42 my_function() { local my_variable=7 } main() { ... } main “$@”
- 45. www.luxoft.com Command Substitution `...` (backtics) • Deprecated • Nested quoting is hard • Nesting is hard (put more and more backslashes `, `, `) • Backslashes are handled in non-obvious way $() • Easy to nest • Use it! Conditions • [[ ... ]] is preferred over [, test and /usr/bin/[
- 46. www.luxoft.com Tradition if ${event}; then ... fi while ${event}; do ... done for v in ${list[@]}; do ... done
- 47. www.luxoft.com 49 TABS vs SPACES Use 2 or 4 spaces Never use tabs Google recommends to use 2 spaces...
- 48. www.luxoft.com 51 BE CONSISTENT Common Sense
- 49. You Don't Need Cat UUOC - Useless Use of Cat aka Cat Abuse cat "$file" | grep "$pattern" cat "$file" | less can instead be written as grep "$pattern" "$file" less "$file"
- 50. Useless Use of kill -9 Award No no no. Don't use kill -9. It doesn't give the process a chance to cleanly: 1. shut down socket connections 2. clean up temp files 3. inform its children that it is going away and so on and so on and so on. Generally, send 15, and wait a second or two, and if that doesn't work, send 2, and if that doesn't work, send 1. If that doesn't, REMOVE THE BINARY because the program is badly behaved! Don't use kill -9. Don't bring out the combine harvester just to tidy up the flower pot.
- 51. www.luxoft.com
- 52. www.luxoft.com Bash Unofficial Strict Mode set -euo pipefail • e — fail fast on every error (except gotchas) • u — fail when referencing an undefined variable • pipefail — fail fast if anything goes wrong when piping cd "$directory" rm -rf *
- 53. www.luxoft.com Gotchas set -e f() { local var=$(somecommand that fails); } f # will not exit g() { local var; var=$(somecommand that fails); } g # will exit http://mywiki.wooledge.org/BashFAQ/105
- 54. www.luxoft.com 59 apt-get install shellcheck
- 55. www.luxoft.com 60 ShellCheck is… • GPLv3: free as in freedom • Available on GitHub • Already packaged for your distro or package manager • Supported as an integrated linter in major editors • Available in CodeClimate, Codacy and CodeFactor to auto-check your GitHub repo • Written in Haskell, if you're into that sort of thing
- 57. www.luxoft.com
- 58. www.luxoft.com
- 59. www.luxoft.com Healthy whitespace gives you breathing space 67 GreyCat's BashGuide
- 65. Debugging
- 66. www.luxoft.com Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it. Brian Kernighan
- 67. www.luxoft.com 77 How to Debug • printf / echo or tee • set -n or set -o noexec • don’t execute, just check syntax • set -v or set -o verbose • print each command before execution • set -nv to perform verbose syntax check • set -x or set -o xtrace #!/bin/bash -xv bash -xv script PS4='$LINENO: ' http://bashdb.sourceforge.net/bashdb.html
- 68. www.luxoft.com 78 Reality of Testing No one thinks about testing before creating scripts Manual testing is a way even for most popular scripts We say script is done after passing 1-2 happy path scenarios
- 70. www.luxoft.com Unit testing is for isolated code Not possible if logic is mixed with side effects Typical task for Bash is interacting with other apps/tools and operating system: • Right executables • Right order • Proper arguments
- 71. www.luxoft.com 81 Prepare env and run Docker Jenkins workspace You (Most Probably) Need Just Integration Testing
- 72. www.luxoft.com 82 Unit Testing in Bash is an Exception, not a Rule Use other languages • Kotlin • Groovy • Python
- 73. www.luxoft.com our Bash scripts will run on production for years 83 they’d better work there are lots of tools for Bash leverage them
- 74. www.luxoft.com Summary 8 Learn Bash and follow style guides Verify code with ShellCheck Use BashSupport in IDEA Perform integration testing
- 76. www.luxoft.com