1.
01

2.
About me
02

3.
Andrey Adamovich
Bio: Developer, coach, speaker, author
Company: Aestas/IT (http://aestasit.com)
E­mail: andrey@aestasit.com
Linkedin: http://www.linkedin.com/in/andreyadamovich
Lanyrd: http://lanyrd.com/profile/andrey­adamovich
GitHub: https://github.com/aadamovich
SO: http://stackoverflow.com/users/162792/andrey­adamovich
Twitter: @codingandrey, @aestasit
•
•
•
•
•
•
•
•
03

4.
What's this presentation about?
DevOps
Intrastructure Provisioning
Continuous Integration
Continuous Delivery
•
•
•
•
04

5.
Technologies
Groovy ­ http://groovy.codehaus.org
Gradle ­ http://gradle.org
Jenkins ­ http://jenkins­ci.org
Puppet ­ http://puppetlabs.com
AWS ­ http://aws.amazon.com
•
•
•
•
•
05

6.
Developers +
Operations =
?06

7.
First Blood
07

8.
Ant + Gradle
ant.taskdef(
name: 'scp',
classname: 'o.a.t.a.t.o.ssh.Scp',
classpath: configurations.secureShell.asPath)
ant.taskdef(
name: 'sshexec',
classname: 'o.a.t.a.t.o.ssh.SSHExec',
classpath: configurations.secureShell.asPath)
01.
02.
03.
04.
05.06.
07.
08.
09.
08

9.
Simple call
ant.sshexec(
host: host,
username: user,
password: password,
command: command,
trust: 'true',
failonerror: failOnError)
01.
02.
03.
04.
05.
06.
07.
09

10.
Next step: wrapper function
def ssh(String command,
Properties props,
boolean failOnError = false,
String suCommandQuoteChar = "'",
String outputProperty = null) {
...
}
01.
02.
03.
04.
05.
06.
07.
10

11.
Next step: wrapper function
def scp(String file,
String remoteDir,
Properties props) {
...
}
01.
02.
03.
04.
05.
11

12.
Task example I
task installFonts << {
forAllServers { props ‐>
ssh('yes | yum install *font*', props)
}
}
01.
02.
03.
04.
05.
12

13.
Task example II
task uninstallNginx << {
forAllServers { props ‐>
ssh('/etc/init.d/nginx stop', props)
ssh('yes | yum remove nginx', props, true)
ssh('rm ‐rf /etc/yum.repos.d/nginx.repo', props)
ssh('rm ‐rf /var/log/nginx', props)
ssh('rm ‐rf /etc/nginx /var/nginx', props)
}
}
01.
02.
03.
04.
05.
06.
07.
08.
09.
13

14.
Drawbacks
New connection each time
Excplicit repeating parameters
Complex scripts are hard to maintain
Tasks are not idempotent
•
•
•
•
14

15.
Sshoogr
15

16.
Sshoogr features
Groovy­based SSH DSL for:
Remote command execution
File uploading/downloading
Tunneling
•
•
•
16

17.
Sshoogr usage (import)
@Grab(
group='com.aestasit.infrastructure.sshoogr',
module='sshoogr',
version='0.9.16')
import static com.aestasit.ssh.DefaultSsh.*
01.
02.
03.
04.
05.
17

18.
Sshoogr usage (defaults)
defaultUser = 'root'
defaultKeyFile = new File('secret.pem')
execOptions {
verbose = true
showCommand = true
}
01.
02.
03.
04.
05.
06.
18

19.
Sshoogr usage (connection)
remoteSession {
url = 'user2:654321@localhost:2222'
exec 'rm ‐rf /tmp/*'
exec 'touch /var/lock/my.pid'
remoteFile('/var/my.conf').text = "enabled=true"
}
01.
02.
03.
04.
05.
06.
19

20.
Sshoogr usage (multi­line content)
remoteFile('/etc/yum.repos.d/puppet.repo').text = '''
[puppet]
name=Puppet Labs Packages
baseurl=http://yum.puppetlabs.com/el/
enabled=0
gpgcheck=0
'''
01.
02.
03.
04.
05.
06.
07.
20

21.
Sshoogr usage (file copying)
remoteSession {
scp {
from { localDir "$buildDir/application" }
into { remoteDir '/var/bea/domain/application' }
}
}
01.
02.
03.
04.
05.
06.
21

22.
Sshoogr usage (command result)
def result = exec(command: '/usr/bin/mycmd',
failOnError: false, showOutput: false)
if (result.exitStatus == 1) {
result.output.eachLine { line ‐>
if (line.contains('WARNING')) {
throw new RuntimeException("Warning!!!")
}
}
}
01.
02.
03.
04.
05.
06.
07.
08.
09.
22

23.
Sshoogr usage (shortcuts)
if (ok('/usr/bin/mycmd')) {
...
}
if (fail('/usr/bin/othercmd')) {
...
}
01.
02.
03.
04.
05.
06.
23

24.
Sshoogr usage (tunnels)
tunnel('1.2.3.4', 8080) { int localPort ‐>
def url = "http://localhost:${localPort}/flushCache"
def result = new URL(url).text
if (result == 'OK') {
println "Cache is flushed!"
} else {
throw new RuntimeException(result)
}
}
01.
02.
03.
04.
05.
06.
07.
08.
09.
24

25.
Sshoogr usage (prefix/suffix)
prefix('sudo ') {
exec 'rm ‐rf /var/log/abc.log'
exec 'service abc restart'
}
suffix(' >> output.log') {
exec 'yum ‐y install nginx'
exec 'yum ‐y install mc'
exec 'yum ‐y install links'
}
01.
02.
03.
04.
05.
06.
07.
08.
09.
25

26.
Still problems
Complex scripts are still not easy to maintain
Scripts are usually not idempotent
•
•
26

27.
Puppet
27

28.
Why Puppet?
More mature than competition
Large community
Readable DSL
Good acceptance from DEVs and OPs
No need to learn Ruby ;)
•
•
•
•
•
28

29.
Puppet example
29

30.
Puppet provisioning
30

31.
Puppet provisioning
31

32.
Puppet provisioning
32

33.
Puppet provisioning
33

34.
Puppet state management
34

35.
Puppet state management
35

36.
Puppet state management
36

37.
Puppet modules
37

38.
Puppet modules
38

39.
Puppet modules
39

40.
Deployment
vs.
Infrastructure
management40

41.
Push vs. Pull
41

42.
Pull
42

43.
Push
43

44.
Sshoogr +
Gradle +
Puppet44

45.
Upload modules
task uploadModules << {
remoteSession {
exec 'rm ‐rf /tmp/repo.zip'
scp {
from { localFile "${buildDir}/repo.zip" }
into { remoteDir "/root" }
}
...
01.
02.
03.
04.
05.
06.
07.
08.
45

46.
Upload modules
...
exec 'rm ‐rf /etc/puppet/modules'
exec 'unzip /tmp/repo.zip ‐d /etc/puppet/modules'
}
}
01.
02.
03.
04.
05.
46

47.
Apply manifests
task puppetApply(dependsOn: uploadModules) << {
remoteSession {
scp {
from { localFile "${buildDir}/setup.pp" }
into { remoteDir "/tmp" }
}
exec 'puppet apply /tmp/setup.pp'
}
}
01.
02.
03.
04.
05.
06.
07.
08.
09.
47

48.
What we solved?
Separated infrastructure state description and operations tasks
Scripts became more maintainable and idempotent
•
•
48

49.
In the meanwhile...
We started developing complex/generic Puppet modules
Modules need proper testing
...on different platforms
•
•
•
49

50.
Do you test, right?
How to test this stuff?
How to reuse a JUnit approach to testing?
We wanted things to be SIMPLE!
•
•
•
50

51.
PUnit
51

52.
PUnit
Simple testing tool for verifying remote server state
Uses Sshoogr and JUnit
Reuse reporting features of JUnit
As simple as ...
•
•
•
•
52

53.
PUnit example (derby)
class DerbyInstallTest
extends BasePuppetIntegrationTest {
@Before
void installDerby() {
apply("include derby")
}
...
}
01.
02.
03.
04.
05.
06.
07.
08.
53

54.
PUnit example (derby)
@Test
void ensureDerbyRunning() {
command('service derby status > derbystatus.log')
assertTrue fileText("/root/derbystatus.log")
.contains('Derby')
assertTrue fileText("/root/derbystatus.log")
.contains('is running.')
}
01.
02.
03.
04.
05.
06.
07.
08.
54

55.
PUnit example (derby)
@Test
void ensureCanConnect() {
Thread.sleep(10000)
uploadScript()
command('/opt/derby/db‐derby‐10.9.1.0‐bin/bin/ij ' +
'testDataScript.sql > derbytest.log')
...
01.
02.
03.
04.
05.
06.
07.
55

56.
Continuous integration
56

57.
Jenkins build
57

58.
Next
problem?
58

59.
Scalability
How do we test on different OS?
How do we run parallel tests on multiple architectures?
How do we avoid selling our houses?
•
•
•
59

60.
Amazon Web
Services
60

61.
Elastic Compute Cloud
Mature
Great API
Virtual hardware variety
OS variety
•
•
•
•
61

62.
Gramazon
62

63.
Gramazon
Groovy­based API for interacting with EC2
Integration with Gradle
•
•
63

64.
Gramazon example I
task startInstance(type: StartInstance) {
keyName 'cloud‐do'
securityGroup 'cloud‐do'
instanceName 'gramazon/cloud‐do'
stateFileName 'cloud‐do.json'
ami 'ami‐6f07e418'
instanceType 't1.micro'
waitForStart true
}
01.
02.
03.
04.
05.
06.
07.
08.
09.
64

65.
Gramazon example II
task terminateInstance(type: TerminateInstance) {
stateFileName 'cloud‐do.json'
}
01.
02.
03.
65

66.
The flow
Start instance(s)
Upload manifests
Run tests
Generate report
Terminate instance(s)
1.
2.
3.
4.
5.
66

67.
Next issue?
67

68.
Imgr
68

69.
Imgr
A tool for building images
Inspired by Packer
•
•
69

70.
Supports
Shell
Puppet
•
•
70

71.
Configuration example
71

72.
Summary
72

73.
Images, manifests, tasks
73

74.
The big picture
74

75.
Conclusions
Groovy/Gradle is an ultimate automation glue!
Reuse your existing Java knowledge
...to build a bridge between DEVs and OPs
Reuse development best practices for OPs
Infrastructure configuration is outsourcable now!
Automate!
•
•
•
•
•
•
75

76.
Source code
Sshoogr: https://github.com/aestasit/sshoogr
Sshoogr Gradle: https://github.com/aestasit/sshoogr­gradle
Groowin: https://github.com/aestasit/groowin
Groowin Gradle: https://github.com/aestasit/groowin­gradle
PUnit: https://github.com/aestasit/puppet­unit
Gramazon: https://github.com/aestasit/gramazon
Imgr: https://github.com/aestasit/imgr
•
•
•
•
•
•
•
76

77.
Questions?
77

78.
Thank you!
78