Groovy DevOps in the Cloud for Devoxx UK 2014

923 views

Published on

This talk focuses on a set of tools to automate the provisioning of virtual machines on Amazon EC2 using Groovy programming language and libraries.
We will explore how to leverage those to create an infrastructure for building, configuring and testing the provisioning of boxes in the cloud - elegant and groovy.

Published in: Software, Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
923
On SlideShare
0
From Embeds
0
Number of Embeds
54
Actions
Shares
0
Downloads
8
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Groovy DevOps in the Cloud for Devoxx UK 2014

  1. 1. 01
  2. 2. About me 02
  3. 3. Andrey Adamovich Bio: Developer, coach, speaker, author Company: Aestas/IT (http://aestasit.com) E-mail: andrey@aestasit.com Linkedin: http://www.linkedin.com/in/andreyadamovich Twitter: @aestasit • • • • • 03
  4. 4. What's this presentation about? Our take on: DevOps Intrastructure Provisioning Continuous Integration Continuous Delivery • • • • 04
  5. 5. Technologies Groovy - http://groovy.codehaus.org Gradle - http://gradle.org Jenkins - http://jenkins-ci.org Puppet - http://puppetlabs.com AWS - http://aws.amazon.com • • • • • 05
  6. 6. Developers + Operations = ?06
  7. 7. Silos 07
  8. 8. Conflicts 08
  9. 9. Risk 09
  10. 10. Agile 10
  11. 11. What is DevOps? 11
  12. 12. C.A.M.S. Culture : People over processes and tools. Software is made by and for people. Automation : Automation is essential for DevOps to gain quick feedback. Measurement : DevOps finds a specific path to measurement. Quality and shared (or at least aligned) incentives are critical. Sharing : Creates a culture where people share ideas, processes, and tools. • • • • 12
  13. 13. It's not about tools! 13
  14. 14. It's about culture and process!14
  15. 15. But without tools... 15
  16. 16. ...it's definitely harder!16
  17. 17. DevOps imply automation! 17
  18. 18. DevOps imply structure! 18
  19. 19. Infrastructure as code 19
  20. 20. Infrastructure as code Automate the provisioning and maintenance of servers: Build from source control Utilize existing tools Ensure testability • • • 20
  21. 21. Configuration propagation 21
  22. 22. Configuration propagation 22
  23. 23. Changes Imagine uploading *.class files and repackaging JAR directly on production servers when you have an urgent code change. 23
  24. 24. Deployment is automatic! 24
  25. 25. And, so, should be... 25
  26. 26. infrastructure configuration changes!26
  27. 27. No manual changes! 27
  28. 28. Building an automation toolkit Automation is key We are JVM hackers Fragmented ecosystem • • • 28
  29. 29. Initial toolset Gradle Groovy Ant Python/WLST Shell scripts • • • • • 29
  30. 30. Required tooling Infrastructure connectivity Infrastructure provisioning Infrastructure virtualization Infrastructure testing • • • • 30
  31. 31. First Blood 31
  32. 32. Ant + Gradle ant.taskdef( name: 'scp', classname: 'o.a.t.a.t.o.ssh.Scp', classpath: configurations.secureShell.asPath) ant.taskdef( name: 'sshexec', classname: 'o.a.t.a.t.o.ssh.SSHExec', classpath: configurations.secureShell.asPath) 01. 02. 03. 04. 05.06. 07. 08. 09. 32
  33. 33. Simple call ant.sshexec( host: host, username: user, password: password, command: command, trust: 'true', failonerror: failOnError) 01. 02. 03. 04. 05. 06. 07. 33
  34. 34. Next step: wrapper function def ssh(String command, Properties props, boolean failOnError = false, String suCommandQuoteChar = "'", String outputProperty = null) { ... } 01. 02. 03. 04. 05. 06. 07. 34
  35. 35. Next step: wrapper function def scp(String file, String remoteDir, Properties props) { ... } 01. 02. 03. 04. 05. 35
  36. 36. Task example I task installFonts << { forAllServers { props -> ssh('yes | yum install *font*', props) } } 01. 02. 03. 04. 05. 36
  37. 37. Task example II task uninstallNginx << { forAllServers { props -> ssh('/etc/init.d/nginx stop', props) ssh('yes | yum remove nginx', props, true) ssh('rm -rf /etc/yum.repos.d/nginx.repo', props) ssh('rm -rf /var/log/nginx', props) ssh('rm -rf /etc/nginx /var/nginx', props) } } 01. 02. 03. 04. 05. 06. 07. 08. 09. 37
  38. 38. Drawbacks New connection each time Excplicit repeating parameters Complex scripts are hard to maintain Tasks are not idempotent • • • • 38
  39. 39. Sshoogr 39
  40. 40. Sshoogr features Groovy-based SSH DSL for: Remote command execution File uploading/downloading Tunneling • • • 40
  41. 41. Why Groovy? Groovy is perfect choice for scripting Gradle build scripts are Groovy Very mature, concise syntax Extremely easy to produce DSL We wrote a book about it! • • • • • 41
  42. 42. Shameless plug 42
  43. 43. Sshoogr usage (import) @Grab( group='com.aestasit.infrastructure.sshoogr', module='sshoogr', version='0.9.16') import static com.aestasit.ssh.DefaultSsh.* 01. 02. 03. 04. 05. 43
  44. 44. Sshoogr usage (defaults) defaultUser = 'root' defaultKeyFile = new File('secret.pem') execOptions { verbose = true showCommand = true } 01. 02. 03. 04. 05. 06. 44
  45. 45. Sshoogr usage (connection) remoteSession { url = 'user2:654321@localhost:2222' exec 'rm -rf /tmp/*' exec 'touch /var/lock/my.pid' remoteFile('/var/my.conf').text = "enabled=true" } 01. 02. 03. 04. 05. 06. 45
  46. 46. Sshoogr usage (multi-line content) remoteFile('/etc/yum.repos.d/puppet.repo').text = ''' [puppet] name=Puppet Labs Packages baseurl=http://yum.puppetlabs.com/el/ enabled=0 gpgcheck=0 ''' 01. 02. 03. 04. 05. 06. 07. 46
  47. 47. Sshoogr usage (file copying) remoteSession { scp { from { localDir "$buildDir/application" } into { remoteDir '/var/bea/domain/application' } } } 01. 02. 03. 04. 05. 06. 47
  48. 48. Sshoogr usage (command result) def result = exec(command: '/usr/bin/mycmd', failOnError: false, showOutput: false) if (result.exitStatus == 1) { result.output.eachLine { line -> if (line.contains('WARNING')) { throw new RuntimeException("Warning!!!") } } } 01. 02. 03. 04. 05. 06. 07. 08. 09. 48
  49. 49. Sshoogr usage (shortcuts) if (ok('/usr/bin/mycmd')) { ... } if (fail('/usr/bin/othercmd')) { ... } 01. 02. 03. 04. 05. 06. 49
  50. 50. Sshoogr usage (tunnels) tunnel('1.2.3.4', 8080) { int localPort -> def url = "http://localhost:${localPort}/flushCache" def result = new URL(url).text if (result == 'OK') { println "Cache is flushed!" } else { throw new RuntimeException(result) } } 01. 02. 03. 04. 05. 06. 07. 08. 09. 50
  51. 51. Sshoogr usage (prefix/suffix) prefix('sudo ') { exec 'rm -rf /var/log/abc.log' exec 'service abc restart' } suffix(' >> output.log') { exec 'yum -y install nginx' exec 'yum -y install mc' exec 'yum -y install links' } 01. 02. 03. 04. 05. 06. 07. 08. 09. 51
  52. 52. Still problems Complex scripts are still not easy to maintain Scripts are usually not idempotent • • 52
  53. 53. Puppet 53
  54. 54. Why Puppet? More mature than competition Large community Readable DSL Good acceptance from DEVs and OPs No need to learn Ruby ;) • • • • • 54
  55. 55. Puppet example 55
  56. 56. Puppet provisioning 56
  57. 57. Puppet provisioning 57
  58. 58. Puppet provisioning 58
  59. 59. Puppet provisioning 59
  60. 60. Puppet state management 60
  61. 61. Puppet state management 61
  62. 62. Puppet state management 62
  63. 63. Puppet modules 63
  64. 64. Puppet modules 64
  65. 65. Puppet modules 65
  66. 66. Sshoogr + Gradle + Puppet66
  67. 67. Upload modules task uploadModules << { remoteSession { exec 'rm -rf /tmp/repo.zip' scp { from { localFile "${buildDir}/repo.zip" } into { remoteDir "/root" } } ... 01. 02. 03. 04. 05. 06. 07. 08. 67
  68. 68. Upload modules ... exec 'rm -rf /etc/puppet/modules' exec 'unzip /tmp/repo.zip -d /etc/puppet/modules' } } 01. 02. 03. 04. 05. 68
  69. 69. Apply manifests task puppetApply(dependsOn: uploadModules) << { remoteSession { scp { from { localFile "${buildDir}/setup.pp" } into { remoteDir "/tmp" } } exec 'puppet apply /tmp/setup.pp' } } 01. 02. 03. 04. 05. 06. 07. 08. 09. 69
  70. 70. What we solved? Separated infrastructure state description and operations tasks Scripts became more maintainable and idempotent • • 70
  71. 71. In the meanwhile... We started developing complex/generic Puppet modules Modules need proper testing ...on different platforms • • • 71
  72. 72. Do you test, right? How to test this stuff? How to reuse a JUnit approach to testing? We wanted things to be SIMPLE! • • • 72
  73. 73. PUnit 73
  74. 74. PUnit Simple testing tool for verifying remote server state Uses Sshoogr and JUnit Reuse reporting features of JUnit As simple as ... • • • • 74
  75. 75. PUnit example (derby) class DerbyInstallTest extends BasePuppetIntegrationTest { @Before void installDerby() { apply("include derby") } ... } 01. 02. 03. 04. 05. 06. 07. 08. 75
  76. 76. PUnit example (derby) @Test void ensureDerbyRunning() { command('service derby status > derbystatus.log') assertTrue fileText("/root/derbystatus.log") .contains('Derby') assertTrue fileText("/root/derbystatus.log") .contains('is running.') } 01. 02. 03. 04. 05. 06. 07. 08. 76
  77. 77. PUnit example (derby) @Test void ensureCanConnect() { Thread.sleep(10000) uploadScript() command('/opt/derby/db-derby-10.9.1.0-bin/bin/ij ' + 'testDataScript.sql > derbytest.log') ... 01. 02. 03. 04. 05. 06. 07. 77
  78. 78. PUnit example (derby) ... // Check if the log of the insert // operation contains the word ERROR. assertFalse( "The script should return at least one error", fileText("/root/derbytest.log") .contains('ERROR') ) ... 01. 02. 03. 04. 05. 06. 07. 08. 09. 78
  79. 79. PUnit example (derby) ... // Check on data that was inserted into a table. assertTrue( "The log should contain a SELECT result", fileText("/root/derbytest.log") .contains('Grand Ave.') ) } 01. 02. 03. 04. 05. 06. 07. 08. 79
  80. 80. PUnit example (jenkins) session { tunnel ('127.0.0.1', 8080) { int localPort -> def driver = new HtmlUnitDriver(false) driver.manage() .timeouts() .pageLoadTimeout(300, TimeUnit.SECONDS) .implicitlyWait(30, TimeUnit.SECONDS) driver.get("http://127.0.0.1:${localPort}/login") ... 01. 02. 03. 04. 05. 06. 07. 08. 09. 10. 80
  81. 81. PUnit example (jenkins) ... def input = driver.findElement(By.name('j_username')) input.sendKeys('john') input = driver.findElement(By.name('j_password')) input.sendKeys('123456') input.submit() ... 01. 02. 03. 04. 05. 06. 07. 81
  82. 82. PUnit example (jenkins) ... def wait = new WebDriverWait(driver, 30) wait.until ExpectedConditions. presenceOfElementLocated (By.linkText('John Doe')) ... } } 01. 02. 03. 04. 05. 06. 07. 82
  83. 83. PUnit example (svn) session { tunnel ('127.0.0.1', 80) { int localPort -> // Initilize repository connection data. DAVRepositoryFactory.setup() def url = SVNURL.create('http', null, '127.0.0.1', localPort, 'repos/cafebabe', true) def repository = SVNRepositoryFactory.create(url) println "Verifying SVN repository at ${url}" ... 01. 02. 03. 04. 05. 06. 07. 08. 09. 83
  84. 84. PUnit example (svn) ... // Setup credentials. def authManager = SVNWCUtil. createDefaultAuthenticationManager('joe', '123456') repository.setAuthenticationManager(authManager) // Verify repository is at revision 0. assertEquals 0, repository.getLatestRevision() ... 01. 02. 03. 04. 05. 06. 07. 08. 09. 84
  85. 85. PUnit example (svn) ... // Commit first revision. ISVNEditor editor = repository. getCommitEditor("Initial commit.", null) editor.with { openRoot(-1) addFile('dummy.txt', null, -1) applyTextDelta('dummy.txt', null) def deltaGenerator = new SVNDeltaGenerator() 01. 02. 03. 04. 05. 06. 07. 08. 09. 85
  86. 86. PUnit example (svn) ... def checksum = deltaGenerator.sendDelta('dummy.txt', new ByteArrayInputStream("data".getBytes()), editor, true) closeFile('dummy.txt', checksum) def commitInfo = closeEdit() println commitInfo } ... 01. 02. 03. 04. 05. 06. 07. 08. 09. 86
  87. 87. PUnit example (svn) ... // Verify repository is at revision 1 now. assertEquals 1, repository.getLatestRevision() } } 01. 02. 03. 04. 05. 87
  88. 88. Continuous integration 88
  89. 89. Why Jenkins? De-facto standard Stable There is a plugin for that! • • • 89
  90. 90. Jenkins build 90
  91. 91. Next problem? 91
  92. 92. Scalability How do we test on different OS? How do we run parallel tests on multiple architectures? How do we avoid selling our houses? • • • 92
  93. 93. Amazon Web Services 93
  94. 94. Elastic Compute Cloud Mature Great API Virtual hardware variety OS variety • • • • 94
  95. 95. Gramazon 95
  96. 96. Gramazon Groovy-based API for interacting with EC2 Integration with Gradle • • 96
  97. 97. Gramazon example I task startInstance(type: StartInstance) { keyName 'cloud-do' securityGroup 'cloud-do' instanceName 'gramazon/cloud-do' stateFileName 'cloud-do.json' ami 'ami-6f07e418' instanceType 't1.micro' waitForStart true } 01. 02. 03. 04. 05. 06. 07. 08. 09. 97
  98. 98. Gramazon example II task terminateInstance(type: TerminateInstance) { stateFileName 'cloud-do.json' } 01. 02. 03. 98
  99. 99. The flow Start instance(s) Upload manifests Run tests Generate report Terminate instance(s) 1. 2. 3. 4. 5. 99
  100. 100. Next issue? 100
  101. 101. Imgr 101
  102. 102. Imgr A tool for building images Inspired by Packer • • 102
  103. 103. Supports Shell Puppet • • 103
  104. 104. Configuration example 104
  105. 105. Summary 105
  106. 106. Images, manifests, tasks 106
  107. 107. The big picture 107
  108. 108. Aetomation 108
  109. 109. Conclusions Reuse your existing Java knowledge ...to build a bridge between DEVs and OPs Reuse development best practices for OPs Don't be afraid to try new technologies Automate! • • • • • 109
  110. 110. Next steps? Create more documentation and examples Add more DSL convience methods Extend integration with Gradle Add Windows connectivity/scripting support Define richer model for EC2 and potentially other clouds Extend support for other provisioning tools • • • • • • 110
  111. 111. Reading material 111
  112. 112. The Phoenix Project 112
  113. 113. Continuous Delivery 113
  114. 114. Release It 114
  115. 115. Programming Amazon EC2 115
  116. 116. Gradle in Action 116
  117. 117. Groovy 2 Cookbook 117
  118. 118. Technologies to follow Vagrant - http://www.vagrantup.com/ Docker - https://www.docker.io/ Packer - http://www.packer.io/ Qemu - http://wiki.qemu.org/ jclouds - http://jclouds.apache.org/ Cloudbees - http://www.cloudbees.com/ • • • • • • 118
  119. 119. One more thing... 119
  120. 120. It's all Open Source! 120
  121. 121. Source code Sshoogr: https://github.com/aestasit/sshoogr Sshoogr Gradle: https://github.com/aestasit/sshoogr-gradle PUnit: https://github.com/aestasit/puppet-unit Gramazon: https://github.com/aestasit/gramazon Imgr: https://github.com/aestasit/imgr • • • • • 121
  122. 122. Seeking contributors! 122
  123. 123. Questions? 123
  124. 124. Thank you! 124

×