Successfully reported this slideshow.
Your SlideShare is downloading. ×

Groovy DevOps in the Cloud for GR8Conf US 2015

Groovy DevOps in the Cloud for GR8Conf US 2015

Download to read offline

The talk is focused on open-source Groovy/Java libraries that we have developed in order to solve our daily DevOps problems including SSH connectivity, cloud instance management, OS image management and integration testing of Puppet modules with the help of Groovy and Junit.

The talk is focused on open-source Groovy/Java libraries that we have developed in order to solve our daily DevOps problems including SSH connectivity, cloud instance management, OS image management and integration testing of Puppet modules with the help of Groovy and Junit.

More Related Content

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all

Groovy DevOps in the Cloud for GR8Conf US 2015

  1. 1. 01
  2. 2. About me 02
  3. 3. Andrey Adamovich Bio: Developer, coach, speaker, author Company: Aestas/IT (http://aestasit.com) E­mail: andrey@aestasit.com Linkedin: http://www.linkedin.com/in/andreyadamovich Lanyrd: http://lanyrd.com/profile/andrey­adamovich GitHub: https://github.com/aadamovich SO: http://stackoverflow.com/users/162792/andrey­adamovich Twitter: @codingandrey, @aestasit • • • • • • • • 03
  4. 4. What's this presentation about? DevOps Intrastructure Provisioning Continuous Integration Continuous Delivery • • • • 04
  5. 5. Technologies Groovy ­ http://groovy.codehaus.org Gradle ­ http://gradle.org Jenkins ­ http://jenkins­ci.org Puppet ­ http://puppetlabs.com AWS ­ http://aws.amazon.com • • • • • 05
  6. 6. Developers + Operations = ?06
  7. 7. First Blood 07
  8. 8. Ant + Gradle ant.taskdef(   name: 'scp',    classname: 'o.a.t.a.t.o.ssh.Scp',    classpath: configurations.secureShell.asPath)  ant.taskdef(   name: 'sshexec',    classname: 'o.a.t.a.t.o.ssh.SSHExec',    classpath: configurations.secureShell.asPath)  01. 02. 03. 04. 05.06. 07. 08. 09. 08
  9. 9. Simple call ant.sshexec(   host: host,    username: user,    password: password,    command: command,    trust: 'true',    failonerror: failOnError) 01. 02. 03. 04. 05. 06. 07. 09
  10. 10. Next step: wrapper function def ssh(String command,          Properties props,          boolean failOnError = false,          String suCommandQuoteChar = "'",          String outputProperty = null) {   ...  } 01. 02. 03. 04. 05. 06. 07. 10
  11. 11. Next step: wrapper function def scp(String file,          String remoteDir,          Properties props) {   ... } 01. 02. 03. 04. 05. 11
  12. 12. Task example I task installFonts << {   forAllServers { props ‐>     ssh('yes | yum install *font*', props)   } } 01. 02. 03. 04. 05. 12
  13. 13. Task example II task uninstallNginx  << {   forAllServers { props ‐>     ssh('/etc/init.d/nginx stop', props)     ssh('yes | yum remove nginx', props, true)     ssh('rm ‐rf /etc/yum.repos.d/nginx.repo', props)     ssh('rm ‐rf /var/log/nginx', props)     ssh('rm ‐rf /etc/nginx /var/nginx', props)   } } 01. 02. 03. 04. 05. 06. 07. 08. 09. 13
  14. 14. Drawbacks New connection each time Excplicit repeating parameters Complex scripts are hard to maintain Tasks are not idempotent • • • • 14
  15. 15. Sshoogr 15
  16. 16. Sshoogr features Groovy­based SSH DSL for: Remote command execution File uploading/downloading Tunneling • • • 16
  17. 17. Sshoogr usage (import) @Grab(   group='com.aestasit.infrastructure.sshoogr',   module='sshoogr',   version='0.9.16') import static com.aestasit.ssh.DefaultSsh.* 01. 02. 03. 04. 05. 17
  18. 18. Sshoogr usage (defaults) defaultUser    = 'root' defaultKeyFile = new File('secret.pem') execOptions {   verbose      = true   showCommand  = true } 01. 02. 03. 04. 05. 06. 18
  19. 19. Sshoogr usage (connection) remoteSession {   url = 'user2:654321@localhost:2222'   exec 'rm ‐rf /tmp/*'   exec 'touch /var/lock/my.pid'   remoteFile('/var/my.conf').text = "enabled=true" } 01. 02. 03. 04. 05. 06. 19
  20. 20. Sshoogr usage (multi­line content) remoteFile('/etc/yum.repos.d/puppet.repo').text = '''   [puppet]   name=Puppet Labs Packages   baseurl=http://yum.puppetlabs.com/el/   enabled=0   gpgcheck=0 ''' 01. 02. 03. 04. 05. 06. 07. 20
  21. 21. Sshoogr usage (file copying) remoteSession {   scp {     from { localDir "$buildDir/application" }     into { remoteDir '/var/bea/domain/application' }   } } 01. 02. 03. 04. 05. 06. 21
  22. 22. Sshoogr usage (command result) def result = exec(command: '/usr/bin/mycmd',   failOnError: false, showOutput: false) if (result.exitStatus == 1) {   result.output.eachLine { line ‐>     if (line.contains('WARNING')) {       throw new RuntimeException("Warning!!!")     }   } } 01. 02. 03. 04. 05. 06. 07. 08. 09. 22
  23. 23. Sshoogr usage (shortcuts) if (ok('/usr/bin/mycmd')) {   ... } if (fail('/usr/bin/othercmd')) {   ... } 01. 02. 03. 04. 05. 06. 23
  24. 24. Sshoogr usage (tunnels) tunnel('1.2.3.4', 8080) { int localPort ‐>   def url = "http://localhost:${localPort}/flushCache"   def result = new URL(url).text   if (result == 'OK') {     println "Cache is flushed!"   } else {     throw new RuntimeException(result)   } } 01. 02. 03. 04. 05. 06. 07. 08. 09. 24
  25. 25. Sshoogr usage (prefix/suffix) prefix('sudo ') {   exec 'rm ‐rf /var/log/abc.log'   exec 'service abc restart' } suffix(' >> output.log') {   exec 'yum ‐y install nginx'   exec 'yum ‐y install mc'   exec 'yum ‐y install links' } 01. 02. 03. 04. 05. 06. 07. 08. 09. 25
  26. 26. Still problems Complex scripts are still not easy to maintain Scripts are usually not idempotent • • 26
  27. 27. Puppet 27
  28. 28. Why Puppet? More mature than competition Large community Readable DSL Good acceptance from DEVs and OPs No need to learn Ruby ;) • • • • • 28
  29. 29. Puppet example 29
  30. 30. Puppet provisioning 30
  31. 31. Puppet provisioning 31
  32. 32. Puppet provisioning 32
  33. 33. Puppet provisioning 33
  34. 34. Puppet state management 34
  35. 35. Puppet state management 35
  36. 36. Puppet state management 36
  37. 37. Puppet modules 37
  38. 38. Puppet modules 38
  39. 39. Puppet modules 39
  40. 40. Deployment vs. Infrastructure management40
  41. 41. Push vs. Pull 41
  42. 42. Pull 42
  43. 43. Push 43
  44. 44. Sshoogr + Gradle + Puppet44
  45. 45. Upload modules task uploadModules << {   remoteSession {     exec 'rm ‐rf /tmp/repo.zip'     scp {       from { localFile "${buildDir}/repo.zip" }       into { remoteDir "/root" }     }     ... 01. 02. 03. 04. 05. 06. 07. 08. 45
  46. 46. Upload modules     ...     exec 'rm ‐rf /etc/puppet/modules'     exec 'unzip /tmp/repo.zip ‐d /etc/puppet/modules'   } } 01. 02. 03. 04. 05. 46
  47. 47. Apply manifests task puppetApply(dependsOn: uploadModules) << {   remoteSession {     scp {       from { localFile "${buildDir}/setup.pp" }       into { remoteDir "/tmp" }     }     exec 'puppet apply /tmp/setup.pp'   } } 01. 02. 03. 04. 05. 06. 07. 08. 09. 47
  48. 48. What we solved? Separated infrastructure state description and operations tasks Scripts became more maintainable and idempotent • • 48
  49. 49. In the meanwhile... We started developing complex/generic Puppet modules Modules need proper testing ...on different platforms • • • 49
  50. 50. Do you test, right? How to test this stuff? How to reuse a JUnit approach to testing? We wanted things to be SIMPLE! • • • 50
  51. 51. PUnit 51
  52. 52. PUnit Simple testing tool for verifying remote server state Uses Sshoogr and JUnit Reuse reporting features of JUnit As simple as ... • • • • 52
  53. 53. PUnit example (derby) class DerbyInstallTest      extends BasePuppetIntegrationTest {   @Before   void installDerby() {     apply("include derby")   }   ... } 01. 02. 03. 04. 05. 06. 07. 08. 53
  54. 54. PUnit example (derby) @Test void ensureDerbyRunning() {   command('service derby status > derbystatus.log')   assertTrue fileText("/root/derbystatus.log")                .contains('Derby')   assertTrue fileText("/root/derbystatus.log")                .contains('is running.') } 01. 02. 03. 04. 05. 06. 07. 08. 54
  55. 55. PUnit example (derby) @Test void ensureCanConnect() {   Thread.sleep(10000)   uploadScript()   command('/opt/derby/db‐derby‐10.9.1.0‐bin/bin/ij ' +            'testDataScript.sql > derbytest.log')   ... 01. 02. 03. 04. 05. 06. 07. 55
  56. 56. Continuous integration 56
  57. 57. Jenkins build 57
  58. 58. Next problem? 58
  59. 59. Scalability How do we test on different OS? How do we run parallel tests on multiple architectures? How do we avoid selling our houses? • • • 59
  60. 60. Amazon Web Services 60
  61. 61. Elastic Compute Cloud Mature Great API Virtual hardware variety OS variety • • • • 61
  62. 62. Gramazon 62
  63. 63. Gramazon Groovy­based API for interacting with EC2 Integration with Gradle • • 63
  64. 64. Gramazon example I task startInstance(type: StartInstance) {   keyName       'cloud‐do'   securityGroup 'cloud‐do'   instanceName  'gramazon/cloud‐do'   stateFileName 'cloud‐do.json'   ami           'ami‐6f07e418'          instanceType  't1.micro'   waitForStart  true } 01. 02. 03. 04. 05. 06. 07. 08. 09. 64
  65. 65. Gramazon example II task terminateInstance(type: TerminateInstance) {   stateFileName 'cloud‐do.json' } 01. 02. 03. 65
  66. 66. The flow Start instance(s) Upload manifests Run tests Generate report Terminate instance(s) 1. 2. 3. 4. 5. 66
  67. 67. Next issue? 67
  68. 68. Imgr 68
  69. 69. Imgr A tool for building images Inspired by Packer • • 69
  70. 70. Supports Shell Puppet • • 70
  71. 71. Configuration example 71
  72. 72. Summary 72
  73. 73. Images, manifests, tasks 73
  74. 74. The big picture 74
  75. 75. Conclusions Groovy/Gradle is an ultimate automation glue! Reuse your existing Java knowledge ...to build a bridge between DEVs and OPs Reuse development best practices for OPs Infrastructure configuration is outsourcable now! Automate! • • • • • • 75
  76. 76. Source code Sshoogr: https://github.com/aestasit/sshoogr Sshoogr Gradle: https://github.com/aestasit/sshoogr­gradle Groowin: https://github.com/aestasit/groowin Groowin Gradle: https://github.com/aestasit/groowin­gradle PUnit: https://github.com/aestasit/puppet­unit Gramazon: https://github.com/aestasit/gramazon Imgr: https://github.com/aestasit/imgr • • • • • • • 76
  77. 77. Questions? 77
  78. 78. Thank you! 78

×