Successfully reported this slideshow.
Your SlideShare is downloading. ×

Infrastructure automation with Gradle and Puppet for GR8Conf US 2015

Infrastructure automation with Gradle and Puppet for GR8Conf US 2015

Download to read offline

Infrastructure-as-code is becoming a mandatory part of any successful project. During this presentation we will show how to use Gradle plugins and Puppet manifests to provision and maintain consistent infrastructure configuration. We will also touch tools like Jenkins, Vagrant, Packer, VirtualBox and AWS.

Infrastructure-as-code is becoming a mandatory part of any successful project. During this presentation we will show how to use Gradle plugins and Puppet manifests to provision and maintain consistent infrastructure configuration. We will also touch tools like Jenkins, Vagrant, Packer, VirtualBox and AWS.

More Related Content

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all

Infrastructure automation with Gradle and Puppet for GR8Conf US 2015

  1. 1. 01
  2. 2. About me 02
  3. 3. Andrey Adamovich Bio: Developer, coach, speaker, author Company: Aestas/IT (http://aestasit.com) E­mail: andrey@aestasit.com Linkedin: http://www.linkedin.com/in/andreyadamovich Lanyrd: http://lanyrd.com/profile/andrey­adamovich GitHub: https://github.com/aadamovich SO: http://stackoverflow.com/users/162792/andrey­adamovich Twitter: @codingandrey, @aestasit • • • • • • • • 03
  4. 4. Workshop structure Preparations and installation Introduction, background, path, reasoning Demo and try­out of (biased) integration of Gradle + Puppet • • • 04
  5. 5. WARNING! 05
  6. 6. Preparation 06
  7. 7. Install tools Gradle 2.+ VirtualBox 4.3.+ Vagrant 1.7.+ • • • 07
  8. 8. Vagrant box Download Vagrant box file: http://bit.ly/1M5shfd Register box in Vagrant: vagrant box add puphpet/ubuntu1404‐x64 ub...x.box ‐‐force • • 01. 08
  9. 9. Project source Source (GitHub): http://bit.ly/1JLTLUc Slides (GitHub): http://bit.ly/1MvTz1B Slides (online): http://bit.ly/1U5aHwn • • • 09
  10. 10. Project setup (step 1) Download apache­tomcat­8.0.18.tar.gz and put it into the repository  directory. Build the  petclinic  application ( gradle clean build ) in the application/spring‐petclinic  directory and place the application/spring‐petclinic/build/libs/petclinic‐0.1‐ SNAPSHOT.war  archive into the  repository  directory. • • 10
  11. 11. Project setup (step 2) Run  vagrant up  to start all the virtual machines. Run  gradle initNode  to initialize software needed for provisioning (Puppet) to work. Run  gradle puppetapply  to apply Puppet configuration to virtual servers. • • • 11
  12. 12. Timing Download tomcat ~ 4 minutes spring­petclinic/ gradle clean build  ~ 3 minutes (+ library download) vagrant up  ~ 15 minutes (­ vagrant box) gradle initNode  ~ 4 minutes gradle puppetApply  ~ 8 minutes • • • • • 12
  13. 13. Target 1 13
  14. 14. Target 2 14
  15. 15. Background 15
  16. 16. Background Big projects built by Ant, Maven, and eventually Gradle Teams composed mostly of Java developers Complex (sometimes, over­engineered) architectures Many environments (DEV, TEST, QA, SIT, UAT, PRE­PROD, PROD) to support • • • • 16
  17. 17. Problems I Infrastructure is influenced by (relatively) frequent architecture changes (components, versions, layers) We want our environments to be the same (or at least quite similar) to avoid any side effects during development, testing and production We don't want to spend hours/days/weeks on configuring each and every new server and keeping them in­sync • • • 17
  18. 18. Problems II Operations guys are not always available (e.g. busy supporting production systems or just not skilled enough) Development infrastructure (Jenkins, Sonar, Version Control, Load Testing etc.) also needs maintenance We want to reuse experience available in our team and avoid throwing in too many various trendy technologies that will fail our expectations • • • 18
  19. 19. Automation is the key! 19
  20. 20. Questions How do I connect (ssh) to my server to perform actions required for deployment? How do I store my configuration for various servers? How do I store secrets? How do I ensure that remote server has everything on it to run my application? How do I verify that my automation works (prefferably without breaking anything important)? • • • • 20
  21. 21. Changing architecture 21
  22. 22. Local development 22
  23. 23. Splitting layers 23
  24. 24. Scaling out 24
  25. 25. Fail­over 25
  26. 26. Performance tuning 26
  27. 27. Maintaining system 27
  28. 28. What if? 28
  29. 29. Or even? 29
  30. 30. Splitting layers 30
  31. 31. Scaling out 31
  32. 32. Maintaining system 32
  33. 33. Late alignment issues 33
  34. 34. Similarity levels DEV << QA <<< PROD DEV < QA < PROD DEV ~ QA ~ PROD DEV = QA = PROD 1. 2. 3. 4. 34
  35. 35. Infrastructure as code 35
  36. 36. Keep it in version control 36
  37. 37. Connectivity 37
  38. 38. How do I connect to my server?38
  39. 39. First Blood 39
  40. 40. Ant + Gradle ant.taskdef(   name: 'scp',    classname: 'o.a.t.a.t.o.ssh.Scp',    classpath: configurations.secureShell.asPath)  ant.taskdef(   name: 'sshexec',    classname: 'o.a.t.a.t.o.ssh.SSHExec',    classpath: configurations.secureShell.asPath)  01. 02. 03. 04. 05.06. 07. 08. 09. 40
  41. 41. Simple call ant.sshexec(   host: host,    username: user,    password: password,    command: command,    trust: 'true',    failonerror: failOnError) 01. 02. 03. 04. 05. 06. 07. 41
  42. 42. Next step: wrapper function def ssh(String command,          Properties props,          boolean failOnError = false,          String suCommandQuoteChar = "'",          String outputProperty = null) {   ...  } 01. 02. 03. 04. 05. 06. 07. 42
  43. 43. Next step: wrapper function def scp(String file,          String remoteDir,          Properties props) {   ... } 01. 02. 03. 04. 05. 43
  44. 44. Task example I task installFonts << {   forAllServers { props ‐>     ssh('yes | yum install *font*', props)   } } 01. 02. 03. 04. 05. 44
  45. 45. Task example II task uninstallNginx  << {   forAllServers { props ‐>     ssh('/etc/init.d/nginx stop', props)     ssh('yes | yum remove nginx', props, true)     ssh('rm ‐rf /etc/yum.repos.d/nginx.repo', props)     ssh('rm ‐rf /var/log/nginx', props)     ssh('rm ‐rf /etc/nginx /var/nginx', props)   } } 01. 02. 03. 04. 05. 06. 07. 08. 09. 45
  46. 46. Drawbacks New connection each time Excplicit repeating parameters Complex scripts are hard to maintain Tasks are not idempotent • • • • 46
  47. 47. Libraries jsch sshj overthere sshoogr • • • • 47
  48. 48. Sshoogr 48
  49. 49. Sshoogr features Groovy­based SSH DSL for: Remote command execution File uploading/downloading Tunneling • • • 49
  50. 50. Sshoogr usage (import) @Grab(   group='com.aestasit.infrastructure.sshoogr',   module='sshoogr',   version='0.9.16') import static com.aestasit.ssh.DefaultSsh.* 01. 02. 03. 04. 05. 50
  51. 51. Sshoogr usage (defaults) defaultUser    = 'root' defaultKeyFile = new File('secret.pem') execOptions {   verbose      = true   showCommand  = true } 01. 02. 03. 04. 05. 06. 51
  52. 52. Sshoogr usage (connection) remoteSession {   url = 'user2:654321@localhost:2222'   exec 'rm ‐rf /tmp/*'   exec 'touch /var/lock/my.pid'   remoteFile('/var/my.conf').text = "enabled=true" } 01. 02. 03. 04. 05. 06. 52
  53. 53. Sshoogr usage (multi­line content) remoteFile('/etc/yum.repos.d/puppet.repo').text = '''   [puppet]   name=Puppet Labs Packages   baseurl=http://yum.puppetlabs.com/el/   enabled=0   gpgcheck=0 ''' 01. 02. 03. 04. 05. 06. 07. 53
  54. 54. Sshoogr usage (file copying) remoteSession {   scp {     from { localDir "$buildDir/application" }     into { remoteDir '/var/bea/domain/application' }   } } 01. 02. 03. 04. 05. 06. 54
  55. 55. Sshoogr usage (command result) def result = exec(command: '/usr/bin/mycmd',   failOnError: false, showOutput: false) if (result.exitStatus == 1) {   result.output.eachLine { line ‐>     if (line.contains('WARNING')) {       throw new RuntimeException("Execution failed with: ${line}")     }   } } 01. 02. 03. 04. 05. 06. 07. 08. 09. 55
  56. 56. Sshoogr usage (shortcuts) if (ok('/usr/bin/mycmd')) {   ... } if (fail('/usr/bin/othercmd')) {   ... } 01. 02. 03. 04. 05. 06. 56
  57. 57. Sshoogr usage (tunnels) tunnel('1.2.3.4', 8080) { int localPort ‐>   def url = "http://localhost:${localPort}/flushCache"   def result = new URL(url).text   if (result == 'OK') {     println "Cache is flushed!"   } else {     throw new RuntimeException(result)   } } 01. 02. 03. 04. 05. 06. 07. 08. 09. 57
  58. 58. Sshoogr usage (prefix/suffix) prefix('sudo ') {   exec 'rm ‐rf /var/log/abc.log'   exec 'service abc restart' } suffix(' >> output.log') {   exec 'yum ‐y install nginx'   exec 'yum ‐y install mc'   exec 'yum ‐y install links' } 01. 02. 03. 04. 05. 06. 07. 08. 09. 58
  59. 59. From scripts to declarations59
  60. 60. Puppet 60
  61. 61. Puppet example 61
  62. 62. Puppet provisioning 62
  63. 63. Puppet provisioning 63
  64. 64. Puppet provisioning 64
  65. 65. Puppet provisioning 65
  66. 66. Puppet state management 66
  67. 67. Puppet state management 67
  68. 68. Puppet state management 68
  69. 69. Puppet modules 69
  70. 70. Puppet modules 70
  71. 71. Puppet modules 71
  72. 72. Deployment vs. Infrastructure management72
  73. 73. Push vs. Pull 73
  74. 74. Pull 74
  75. 75. Push 75
  76. 76. Push imple­ mentation 76
  77. 77. Sshoogr + Gradle + Puppet77
  78. 78. Upload modules task uploadModules << {   remoteSession {     exec 'rm ‐rf /tmp/repo.zip'     scp {       from { localFile "${buildDir}/repo.zip" }       into { remoteDir "/root" }     }     ... 01. 02. 03. 04. 05. 06. 07. 08. 78
  79. 79. Upload modules     ...     exec 'rm ‐rf /etc/puppet/modules'     exec 'unzip /tmp/repo.zip ‐d /etc/puppet/modules'   } } 01. 02. 03. 04. 05. 79
  80. 80. Apply manifests task puppetApply(dependsOn: uploadModules) << {   remoteSession {     scp {       from { localFile "${buildDir}/setup.pp" }       into { remoteDir "/tmp" }     }     exec 'puppet apply /tmp/setup.pp'   } } 01. 02. 03. 04. 05. 06. 07. 08. 09. 80
  81. 81. What we solved? Separated infrastructure state description and operations tasks Scripts became more maintainable and idempotent • • 81
  82. 82. Declaration reuse 82
  83. 83. How do I store my configuration?83
  84. 84. *.properties files 84
  85. 85. ConfigSlurper 85
  86. 86. Topology What to do if environment has more than 1 server? 86
  87. 87. Server roles 87
  88. 88. Hiera 88
  89. 89. External dependencies 89
  90. 90. Puppet module tool... 90
  91. 91. is not that good! 91
  92. 92. Puppet module repositories Puppet forge Pulp GitHub Maven as a repository • • • • 92
  93. 93. Dependencies managers Puppet module tool Puppet librarian r10k Puppet Module Installer • • • • 93
  94. 94. Demo / Playground 94
  95. 95. Summary 95
  96. 96. Configuration code coverage 96
  97. 97. Images, manifests, tasks 97
  98. 98. Questions? 98
  99. 99. Thank you! 99
  100. 100. Happy infrastructure management!100

×