Groovy DevOps in the
Cloud
01
About me
02
Andrey Adamovich
Bio: Developer, coach, speaker, author
Company: Aestas/IT (http://aestasit.com)
E-mail: andrey@aestasit.c...
What's this presentation about?
Our take on:
DevOps
Server Provisioning
Continuous Integration
Continuous Delivery
•
•
•
•...
Technologies
AWS - http://aws.amazon.com
Groovy - http://groovy.codehaus.org
Gradle - http://gradle.org
Jenkins - http://j...
Developers +
Operations =
?06
Silos
07
Conflicts
08
Risk
09
Agile
10
What is DevOps?
11
It's not about
tools!
12
It's about
culture and
process!13
But without
tools...
14
...it's
definitely
harder!15
DevOps imply
automation!
16
DevOps imply
structure!
17
Infrastructure as Code
Automate the provisioning and maintenance of servers:
Build from source control
Utilize Open Source...
Configuration propagation
19
Changes
No Manual Changes!
20
Building an automation toolkit
Automation is key
We are JVM hackers
Fragmented ecosystem
Full-stack approach
•
•
•
•
21
Sshoogr
22
Sshoogr Features
Remote command execution
File uploading/downloading
Tunneling
•
•
•
23
Sshoogr Example I
remoteSession {
url = 'user2:654321@localhost:2222'
exec 'rm -rf /tmp/*'
exec 'touch /var/lock/my.pid'
r...
Sshoogr Example II
remoteSession {
scp {
from { localDir "$buildDir/application" }
into { remoteDir '/var/bea/domain/appli...
Sshoogr Example III
def result = exec(command: '/usr/bin/mycmd',
failOnError: false, showOutput: false)
if (result.exitSta...
Why Groovy?
Groovy is perfect choice for scripting
Very mature, concise syntax
Extremely easy to produce DSL
We wrote a bo...
Shameless plug
28
Puppet
29
Why Puppet?
More mature than competition
Large community
Readable DSL
No need to learn Ruby ;)
•
•
•
•
30
Continuous Integration
31
Why Jenkins?
De-facto standard
Stable
There is a plugin for that!
•
•
•
32
Gradle
33
Gradle Example I
task uploadModules << {
remoteSession {
exec 'rm -rf /tmp/repo.zip'
scp {
from { localFile "${buildDir}/r...
Gradle Example I
...
exec 'rm -rf /etc/puppet/modules'
exec 'unzip /tmp/repo.zip -d /etc/puppet/modules'
}
}
01.
02.
03.
0...
Gradle Example II
task puppetApply(dependsOn: uploadModules) << {
remoteSession {
scp {
from { localFile "${buildDir}/setu...
In the meanwhile...
We started developing complex Puppet modules
Modules needs proper testing
...on different platforms
•
...
Do you test, right?
How to test this stuff?
How to reuse a JUnit approach to testing?
We wanted things to be SIMPLE!
•
•
•...
PuppetUnit
39
PUnit
Simple testing tool for verifying remote server state
Uses sshoogr and JUnit
Reuse reporting features of JUnit and J...
PUnit Example I
class DerbyInstallTest
extends BasePuppetIntegrationTest {
@Before
void installDerby() {
apply("include de...
PUnit Example II
@Test
def void ensureDerbyRunning() {
command('service derby status > derbystatus.log')
assertTrue fileTe...
PUnit Example III
@Test
def void ensureCanConnect() {
Thread.sleep(10000)
uploadScript()
command('/opt/derby/db-derby-10.9...
PUnit Example III
...
// Check if the log of the insert
// operation contains the word ERROR.
assertFalse(
"The script sho...
PUnit Example III
...
// Check on data that was inserted into a table.
assertTrue(
"The log should contain a SELECT result...
PUnit
46
More examples
session {
tunnel ('127.0.0.1', 8080) { int localPort ->
// Login to Jenkins.
def driver = new HtmlUnitDriver...
More examples
...
def input = driver.findElement(By.name('j_username'))
input.sendKeys('john')
input = driver.findElement(...
More examples
...
// Verify login.
def wait = new WebDriverWait(driver, 30)
wait.until ExpectedConditions.presenceOfElemen...
More examples
session {
tunnel ('127.0.0.1', 80) { int localPort ->
// Initilize repository connection data.
DAVRepository...
More examples
...
// Setup credentials.
def authManager = SVNWCUtil.createDefaultAuthenticationMa
repository.setAuthentica...
More examples
...
// Commit first revision.
ISVNEditor editor = repository.getCommitEditor("Initial c
editor.with {
openRo...
More examples
...
def checksum = deltaGenerator.sendDelta('dummy.txt', ne
closeFile('dummy.txt', checksum)
def commitInfo ...
More examples
...
// Verify repository is at revision 1 now.
assertEquals 1, repository.getLatestRevision()
}
}
01.
02.
03...
Next
problem?
55
Scalability
How do we test on different OS?
How do we run parallel tests on multiple architectures?
How do we avoid sellin...
Amazon Web
Services
57
Elastic Compute Cloud
Mature
Great API
Virtual hardware variety
OS variety
•
•
•
•
58
Gramazon
59
Gramazon
Groovy based API for interacting with EC2
Integrates with the rest of the stack
•
•
60
Gramazon Example I
task startInstance(type: StartInstance) {
keyName 'cloud-do'
securityGroup 'cloud-do'
instanceName 'gra...
Gramazon Example II
task terminateInstance(type: TerminateInstance) {
stateFileName 'cloud-do.json'
}
01.
02.
03.
62
Imgr
63
Imgr
A tool for building images
Inspired by Packer
•
•
64
Images?
65
Supports
Shell
Puppet
•
•
66
Configuration Example
67
The flow
Start instance(s)
Upload manifests
Run tests
Generate report
Terminate instance(s)
1.
2.
3.
4.
5.
68
Images, manifests, tasks
69
The big picture
70
Aetomation
71
Conclusions
Reuse your existing Java knowledge
...to build a bridge between DEVs and OPs
Reuse development best practices ...
One more
thing...
73
It's all Open
Source!
74
Source code
Sshoogr: https://github.com/aestasit/sshoogr
Sshoogr Gradle: https://github.com/aestasit/sshoogr-gradle
PUnit:...
Seeking
contributors!
76
Questions?
77
Hvala! Thank you!
78
Links
http://aetomation.com
http://aestasit.com
•
•
79
Upcoming SlideShare
Loading in …5
×

Groovy DevOps in the Cloud for DevOpsDays in Ljubljana 2014

966 views

Published on

This presentation focuses on a set of tools used to automate the provisioning of (cloud) servers, using Groovy libraries and Gradle plugins. We will explore how to leverage these to create an infrastructure for building, configuring and testing the provisioning of boxes in the cloud – elegant and groovy.

Java/Groovy developers interested in reusing their existing skills for infrastructure provisioning and learning more about problems encountered during system operations should come to this presentation.

Published in: Software, Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
966
On SlideShare
0
From Embeds
0
Number of Embeds
98
Actions
Shares
0
Downloads
5
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Groovy DevOps in the Cloud for DevOpsDays in Ljubljana 2014

  1. 1. Groovy DevOps in the Cloud 01
  2. 2. About me 02
  3. 3. Andrey Adamovich Bio: Developer, coach, speaker, author Company: Aestas/IT (http://aestasit.com) E-mail: andrey@aestasit.com Linkedin: http://www.linkedin.com/in/andreyadamovich Twitter: @aestasit • • • • • 03
  4. 4. What's this presentation about? Our take on: DevOps Server Provisioning Continuous Integration Continuous Delivery • • • • 04
  5. 5. Technologies AWS - http://aws.amazon.com Groovy - http://groovy.codehaus.org Gradle - http://gradle.org Jenkins - http://jenkins-ci.org Puppet - http://puppetlabs.com • • • • • 05
  6. 6. Developers + Operations = ?06
  7. 7. Silos 07
  8. 8. Conflicts 08
  9. 9. Risk 09
  10. 10. Agile 10
  11. 11. What is DevOps? 11
  12. 12. It's not about tools! 12
  13. 13. It's about culture and process!13
  14. 14. But without tools... 14
  15. 15. ...it's definitely harder!15
  16. 16. DevOps imply automation! 16
  17. 17. DevOps imply structure! 17
  18. 18. Infrastructure as Code Automate the provisioning and maintenance of servers: Build from source control Utilize Open Source tools Ensure testability • • • 18
  19. 19. Configuration propagation 19
  20. 20. Changes No Manual Changes! 20
  21. 21. Building an automation toolkit Automation is key We are JVM hackers Fragmented ecosystem Full-stack approach • • • • 21
  22. 22. Sshoogr 22
  23. 23. Sshoogr Features Remote command execution File uploading/downloading Tunneling • • • 23
  24. 24. Sshoogr Example I remoteSession { url = 'user2:654321@localhost:2222' exec 'rm -rf /tmp/*' exec 'touch /var/lock/my.pid' remoteFile('/var/my.conf').text = "enabled=true" } 01. 02. 03. 04. 05. 06. 24
  25. 25. Sshoogr Example II remoteSession { scp { from { localDir "$buildDir/application" } into { remoteDir '/var/bea/domain/application' } } } 01. 02. 03. 04. 05. 06. 25
  26. 26. Sshoogr Example III def result = exec(command: '/usr/bin/mycmd', failOnError: false, showOutput: false) if (result.exitStatus == 1) { result.output.eachLine { line -> if (line.contains('WARNING')) { throw new RuntimeException("Warning!!!") } } } 01. 02. 03. 04. 05. 06. 07. 08. 09. 26
  27. 27. Why Groovy? Groovy is perfect choice for scripting Very mature, concise syntax Extremely easy to produce DSL We wrote a book about it! • • • • 27
  28. 28. Shameless plug 28
  29. 29. Puppet 29
  30. 30. Why Puppet? More mature than competition Large community Readable DSL No need to learn Ruby ;) • • • • 30
  31. 31. Continuous Integration 31
  32. 32. Why Jenkins? De-facto standard Stable There is a plugin for that! • • • 32
  33. 33. Gradle 33
  34. 34. Gradle Example I task uploadModules << { remoteSession { exec 'rm -rf /tmp/repo.zip' scp { from { localFile "${buildDir}/repo.zip" } into { remoteDir "/root" } } ... 01. 02. 03. 04. 05. 06. 07. 08. 34
  35. 35. Gradle Example I ... exec 'rm -rf /etc/puppet/modules' exec 'unzip /tmp/repo.zip -d /etc/puppet/modules' } } 01. 02. 03. 04. 05. 35
  36. 36. Gradle Example II task puppetApply(dependsOn: uploadModules) << { remoteSession { scp { from { localFile "${buildDir}/setup.pp" } into { remoteDir "/tmp" } } exec 'puppet apply /tmp/setup.pp' } } 01. 02. 03. 04. 05. 06. 07. 08. 09. 36
  37. 37. In the meanwhile... We started developing complex Puppet modules Modules needs proper testing ...on different platforms • • • 37
  38. 38. Do you test, right? How to test this stuff? How to reuse a JUnit approach to testing? We wanted things to be SIMPLE! • • • 38
  39. 39. PuppetUnit 39
  40. 40. PUnit Simple testing tool for verifying remote server state Uses sshoogr and JUnit Reuse reporting features of JUnit and Jenkins As simple as ... • • • • 40
  41. 41. PUnit Example I class DerbyInstallTest extends BasePuppetIntegrationTest { @Before void installDerby() { apply("include derby") } ... } 01. 02. 03. 04. 05. 06. 07. 08. 41
  42. 42. PUnit Example II @Test def void ensureDerbyRunning() { command('service derby status > derbystatus.log') assertTrue fileText("/root/derbystatus.log") .contains('Derby') assertTrue fileText("/root/derbystatus.log") .contains('is running.') } 01. 02. 03. 04. 05. 06. 07. 08. 42
  43. 43. PUnit Example III @Test def void ensureCanConnect() { Thread.sleep(10000) uploadScript() command('/opt/derby/db-derby-10.9.1.0-bin/bin/ij ' + 'testDataScript.sql > derbytest.log') ... 01. 02. 03. 04. 05. 06. 07. 43
  44. 44. PUnit Example III ... // Check if the log of the insert // operation contains the word ERROR. assertFalse( "The script should return at least one error", fileText("/root/derbytest.log") .contains('ERROR') ) ... 01. 02. 03. 04. 05. 06. 07. 08. 09. 44
  45. 45. PUnit Example III ... // Check on data that was inserted into a table. assertTrue( "The log should contain a SELECT result", fileText("/root/derbytest.log") .contains('Grand Ave.') ) } 01. 02. 03. 04. 05. 06. 07. 08. 45
  46. 46. PUnit 46
  47. 47. More examples session { tunnel ('127.0.0.1', 8080) { int localPort -> // Login to Jenkins. def driver = new HtmlUnitDriver(false) driver.manage().timeouts().pageLoadTimeout(300, TimeUnit. driver.get("http://127.0.0.1:${localPort}/login") ... 01. 02. 03.04. 05. 06. 07. 08. 09. 47
  48. 48. More examples ... def input = driver.findElement(By.name('j_username')) input.sendKeys('john') input = driver.findElement(By.name('j_password')) input.sendKeys('123456') input.submit() ... 01. 02. 03. 04. 05. 06. 07. 48
  49. 49. More examples ... // Verify login. def wait = new WebDriverWait(driver, 30) wait.until ExpectedConditions.presenceOfElementLocated (B ... } } 01. 02. 03. 04. 05. 06. 07. 49
  50. 50. More examples session { tunnel ('127.0.0.1', 80) { int localPort -> // Initilize repository connection data. DAVRepositoryFactory.setup() def url = SVNURL.create('http', null, '127.0.0.1', localP def repository = SVNRepositoryFactory.create(url) println "Verifying SVN repository at ${url}" ... 01. 02. 03.04. 05. 06. 07. 08. 09. 50
  51. 51. More examples ... // Setup credentials. def authManager = SVNWCUtil.createDefaultAuthenticationMa repository.setAuthenticationManager(authManager) // Verify repository is at revision 0. assertEquals 0, repository.getLatestRevision() ... 01. 02. 03. 04. 05.06. 07. 08. 51
  52. 52. More examples ... // Commit first revision. ISVNEditor editor = repository.getCommitEditor("Initial c editor.with { openRoot(-1) addFile('dummy.txt', null, -1) applyTextDelta('dummy.txt', null) def deltaGenerator = new SVNDeltaGenerator() ... 01. 02. 03. 04. 05. 06. 07. 08. 09. 52
  53. 53. More examples ... def checksum = deltaGenerator.sendDelta('dummy.txt', ne closeFile('dummy.txt', checksum) def commitInfo = closeEdit() println commitInfo } ... 01. 02. 03. 04. 05. 06. 07. 53
  54. 54. More examples ... // Verify repository is at revision 1 now. assertEquals 1, repository.getLatestRevision() } } 01. 02. 03. 04.05. 06. 54
  55. 55. Next problem? 55
  56. 56. Scalability How do we test on different OS? How do we run parallel tests on multiple architectures? How do we avoid selling our houses? • • • 56
  57. 57. Amazon Web Services 57
  58. 58. Elastic Compute Cloud Mature Great API Virtual hardware variety OS variety • • • • 58
  59. 59. Gramazon 59
  60. 60. Gramazon Groovy based API for interacting with EC2 Integrates with the rest of the stack • • 60
  61. 61. Gramazon Example I task startInstance(type: StartInstance) { keyName 'cloud-do' securityGroup 'cloud-do' instanceName 'gramazon/cloud-do' stateFileName 'cloud-do.json' ami 'ami-6f07e418' instanceType 't1.micro' waitForStart true } 01. 02. 03. 04. 05. 06. 07. 08. 09. 61
  62. 62. Gramazon Example II task terminateInstance(type: TerminateInstance) { stateFileName 'cloud-do.json' } 01. 02. 03. 62
  63. 63. Imgr 63
  64. 64. Imgr A tool for building images Inspired by Packer • • 64
  65. 65. Images? 65
  66. 66. Supports Shell Puppet • • 66
  67. 67. Configuration Example 67
  68. 68. The flow Start instance(s) Upload manifests Run tests Generate report Terminate instance(s) 1. 2. 3. 4. 5. 68
  69. 69. Images, manifests, tasks 69
  70. 70. The big picture 70
  71. 71. Aetomation 71
  72. 72. Conclusions Reuse your existing Java knowledge ...to build a bridge between DEVs and OPs Reuse development best practices for OPs Don't be afraid to try new technologies Automate! • • • • • 72
  73. 73. One more thing... 73
  74. 74. It's all Open Source! 74
  75. 75. Source code Sshoogr: https://github.com/aestasit/sshoogr Sshoogr Gradle: https://github.com/aestasit/sshoogr-gradle PUnit: https://github.com/aestasit/puppet-unit Gramazon: https://github.com/aestasit/gramazon Imgr: https://github.com/aestasit/imgr • • • • • 75
  76. 76. Seeking contributors! 76
  77. 77. Questions? 77
  78. 78. Hvala! Thank you! 78
  79. 79. Links http://aetomation.com http://aestasit.com • • 79

×