Successfully reported this slideshow.
Your SlideShare is downloading. ×

Groovy DevOps in the Cloud for DevOpsDays in Ljubljana 2014

Groovy DevOps in the Cloud for DevOpsDays in Ljubljana 2014

Download to read offline

This presentation focuses on a set of tools used to automate the provisioning of (cloud) servers, using Groovy libraries and Gradle plugins. We will explore how to leverage these to create an infrastructure for building, configuring and testing the provisioning of boxes in the cloud – elegant and groovy.

Java/Groovy developers interested in reusing their existing skills for infrastructure provisioning and learning more about problems encountered during system operations should come to this presentation.

This presentation focuses on a set of tools used to automate the provisioning of (cloud) servers, using Groovy libraries and Gradle plugins. We will explore how to leverage these to create an infrastructure for building, configuring and testing the provisioning of boxes in the cloud – elegant and groovy.

Java/Groovy developers interested in reusing their existing skills for infrastructure provisioning and learning more about problems encountered during system operations should come to this presentation.

More Related Content

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all

Groovy DevOps in the Cloud for DevOpsDays in Ljubljana 2014

  1. 1. Groovy DevOps in the Cloud 01
  2. 2. About me 02
  3. 3. Andrey Adamovich Bio: Developer, coach, speaker, author Company: Aestas/IT (http://aestasit.com) E-mail: andrey@aestasit.com Linkedin: http://www.linkedin.com/in/andreyadamovich Twitter: @aestasit • • • • • 03
  4. 4. What's this presentation about? Our take on: DevOps Server Provisioning Continuous Integration Continuous Delivery • • • • 04
  5. 5. Technologies AWS - http://aws.amazon.com Groovy - http://groovy.codehaus.org Gradle - http://gradle.org Jenkins - http://jenkins-ci.org Puppet - http://puppetlabs.com • • • • • 05
  6. 6. Developers + Operations = ?06
  7. 7. Silos 07
  8. 8. Conflicts 08
  9. 9. Risk 09
  10. 10. Agile 10
  11. 11. What is DevOps? 11
  12. 12. It's not about tools! 12
  13. 13. It's about culture and process!13
  14. 14. But without tools... 14
  15. 15. ...it's definitely harder!15
  16. 16. DevOps imply automation! 16
  17. 17. DevOps imply structure! 17
  18. 18. Infrastructure as Code Automate the provisioning and maintenance of servers: Build from source control Utilize Open Source tools Ensure testability • • • 18
  19. 19. Configuration propagation 19
  20. 20. Changes No Manual Changes! 20
  21. 21. Building an automation toolkit Automation is key We are JVM hackers Fragmented ecosystem Full-stack approach • • • • 21
  22. 22. Sshoogr 22
  23. 23. Sshoogr Features Remote command execution File uploading/downloading Tunneling • • • 23
  24. 24. Sshoogr Example I remoteSession { url = 'user2:654321@localhost:2222' exec 'rm -rf /tmp/*' exec 'touch /var/lock/my.pid' remoteFile('/var/my.conf').text = "enabled=true" } 01. 02. 03. 04. 05. 06. 24
  25. 25. Sshoogr Example II remoteSession { scp { from { localDir "$buildDir/application" } into { remoteDir '/var/bea/domain/application' } } } 01. 02. 03. 04. 05. 06. 25
  26. 26. Sshoogr Example III def result = exec(command: '/usr/bin/mycmd', failOnError: false, showOutput: false) if (result.exitStatus == 1) { result.output.eachLine { line -> if (line.contains('WARNING')) { throw new RuntimeException("Warning!!!") } } } 01. 02. 03. 04. 05. 06. 07. 08. 09. 26
  27. 27. Why Groovy? Groovy is perfect choice for scripting Very mature, concise syntax Extremely easy to produce DSL We wrote a book about it! • • • • 27
  28. 28. Shameless plug 28
  29. 29. Puppet 29
  30. 30. Why Puppet? More mature than competition Large community Readable DSL No need to learn Ruby ;) • • • • 30
  31. 31. Continuous Integration 31
  32. 32. Why Jenkins? De-facto standard Stable There is a plugin for that! • • • 32
  33. 33. Gradle 33
  34. 34. Gradle Example I task uploadModules << { remoteSession { exec 'rm -rf /tmp/repo.zip' scp { from { localFile "${buildDir}/repo.zip" } into { remoteDir "/root" } } ... 01. 02. 03. 04. 05. 06. 07. 08. 34
  35. 35. Gradle Example I ... exec 'rm -rf /etc/puppet/modules' exec 'unzip /tmp/repo.zip -d /etc/puppet/modules' } } 01. 02. 03. 04. 05. 35
  36. 36. Gradle Example II task puppetApply(dependsOn: uploadModules) << { remoteSession { scp { from { localFile "${buildDir}/setup.pp" } into { remoteDir "/tmp" } } exec 'puppet apply /tmp/setup.pp' } } 01. 02. 03. 04. 05. 06. 07. 08. 09. 36
  37. 37. In the meanwhile... We started developing complex Puppet modules Modules needs proper testing ...on different platforms • • • 37
  38. 38. Do you test, right? How to test this stuff? How to reuse a JUnit approach to testing? We wanted things to be SIMPLE! • • • 38
  39. 39. PuppetUnit 39
  40. 40. PUnit Simple testing tool for verifying remote server state Uses sshoogr and JUnit Reuse reporting features of JUnit and Jenkins As simple as ... • • • • 40
  41. 41. PUnit Example I class DerbyInstallTest extends BasePuppetIntegrationTest { @Before void installDerby() { apply("include derby") } ... } 01. 02. 03. 04. 05. 06. 07. 08. 41
  42. 42. PUnit Example II @Test def void ensureDerbyRunning() { command('service derby status > derbystatus.log') assertTrue fileText("/root/derbystatus.log") .contains('Derby') assertTrue fileText("/root/derbystatus.log") .contains('is running.') } 01. 02. 03. 04. 05. 06. 07. 08. 42
  43. 43. PUnit Example III @Test def void ensureCanConnect() { Thread.sleep(10000) uploadScript() command('/opt/derby/db-derby-10.9.1.0-bin/bin/ij ' + 'testDataScript.sql > derbytest.log') ... 01. 02. 03. 04. 05. 06. 07. 43
  44. 44. PUnit Example III ... // Check if the log of the insert // operation contains the word ERROR. assertFalse( "The script should return at least one error", fileText("/root/derbytest.log") .contains('ERROR') ) ... 01. 02. 03. 04. 05. 06. 07. 08. 09. 44
  45. 45. PUnit Example III ... // Check on data that was inserted into a table. assertTrue( "The log should contain a SELECT result", fileText("/root/derbytest.log") .contains('Grand Ave.') ) } 01. 02. 03. 04. 05. 06. 07. 08. 45
  46. 46. PUnit 46
  47. 47. More examples session { tunnel ('127.0.0.1', 8080) { int localPort -> // Login to Jenkins. def driver = new HtmlUnitDriver(false) driver.manage().timeouts().pageLoadTimeout(300, TimeUnit. driver.get("http://127.0.0.1:${localPort}/login") ... 01. 02. 03.04. 05. 06. 07. 08. 09. 47
  48. 48. More examples ... def input = driver.findElement(By.name('j_username')) input.sendKeys('john') input = driver.findElement(By.name('j_password')) input.sendKeys('123456') input.submit() ... 01. 02. 03. 04. 05. 06. 07. 48
  49. 49. More examples ... // Verify login. def wait = new WebDriverWait(driver, 30) wait.until ExpectedConditions.presenceOfElementLocated (B ... } } 01. 02. 03. 04. 05. 06. 07. 49
  50. 50. More examples session { tunnel ('127.0.0.1', 80) { int localPort -> // Initilize repository connection data. DAVRepositoryFactory.setup() def url = SVNURL.create('http', null, '127.0.0.1', localP def repository = SVNRepositoryFactory.create(url) println "Verifying SVN repository at ${url}" ... 01. 02. 03.04. 05. 06. 07. 08. 09. 50
  51. 51. More examples ... // Setup credentials. def authManager = SVNWCUtil.createDefaultAuthenticationMa repository.setAuthenticationManager(authManager) // Verify repository is at revision 0. assertEquals 0, repository.getLatestRevision() ... 01. 02. 03. 04. 05.06. 07. 08. 51
  52. 52. More examples ... // Commit first revision. ISVNEditor editor = repository.getCommitEditor("Initial c editor.with { openRoot(-1) addFile('dummy.txt', null, -1) applyTextDelta('dummy.txt', null) def deltaGenerator = new SVNDeltaGenerator() ... 01. 02. 03. 04. 05. 06. 07. 08. 09. 52
  53. 53. More examples ... def checksum = deltaGenerator.sendDelta('dummy.txt', ne closeFile('dummy.txt', checksum) def commitInfo = closeEdit() println commitInfo } ... 01. 02. 03. 04. 05. 06. 07. 53
  54. 54. More examples ... // Verify repository is at revision 1 now. assertEquals 1, repository.getLatestRevision() } } 01. 02. 03. 04.05. 06. 54
  55. 55. Next problem? 55
  56. 56. Scalability How do we test on different OS? How do we run parallel tests on multiple architectures? How do we avoid selling our houses? • • • 56
  57. 57. Amazon Web Services 57
  58. 58. Elastic Compute Cloud Mature Great API Virtual hardware variety OS variety • • • • 58
  59. 59. Gramazon 59
  60. 60. Gramazon Groovy based API for interacting with EC2 Integrates with the rest of the stack • • 60
  61. 61. Gramazon Example I task startInstance(type: StartInstance) { keyName 'cloud-do' securityGroup 'cloud-do' instanceName 'gramazon/cloud-do' stateFileName 'cloud-do.json' ami 'ami-6f07e418' instanceType 't1.micro' waitForStart true } 01. 02. 03. 04. 05. 06. 07. 08. 09. 61
  62. 62. Gramazon Example II task terminateInstance(type: TerminateInstance) { stateFileName 'cloud-do.json' } 01. 02. 03. 62
  63. 63. Imgr 63
  64. 64. Imgr A tool for building images Inspired by Packer • • 64
  65. 65. Images? 65
  66. 66. Supports Shell Puppet • • 66
  67. 67. Configuration Example 67
  68. 68. The flow Start instance(s) Upload manifests Run tests Generate report Terminate instance(s) 1. 2. 3. 4. 5. 68
  69. 69. Images, manifests, tasks 69
  70. 70. The big picture 70
  71. 71. Aetomation 71
  72. 72. Conclusions Reuse your existing Java knowledge ...to build a bridge between DEVs and OPs Reuse development best practices for OPs Don't be afraid to try new technologies Automate! • • • • • 72
  73. 73. One more thing... 73
  74. 74. It's all Open Source! 74
  75. 75. Source code Sshoogr: https://github.com/aestasit/sshoogr Sshoogr Gradle: https://github.com/aestasit/sshoogr-gradle PUnit: https://github.com/aestasit/puppet-unit Gramazon: https://github.com/aestasit/gramazon Imgr: https://github.com/aestasit/imgr • • • • • 75
  76. 76. Seeking contributors! 76
  77. 77. Questions? 77
  78. 78. Hvala! Thank you! 78
  79. 79. Links http://aetomation.com http://aestasit.com • • 79

×