1.
01

2.
02

3.
Let's start!
03

4.
Background
04

5.
Background
Big projects built by Ant, Maven, and eventually Gradle
Teams composed mostly of Java developers
Complex (sometimes, over­engineered) architectures
Many environments (DEV, TEST, QA, SIT, UAT, PRE­PROD, PROD)
to support
•
•
•
•
05

6.
Problems I
Infrastructure is influenced by (relatively) frequent architecture
changes (components, versions, layers)
We want our environments to be the same (or at least quite similar) to
avoid any side effects during development, testing and production
We don't want to spend hours/days/weeks on configuring each and
every new server and keeping them in­sync
•
•
•
06

7.
Problems II
Operations guys are not always available (e.g. busy supporting
production systems or just not skilled enough)
Development infrastructure (Jenkins, Sonar, Version Control, Load
Testing etc.) also needs maintenance
We want to reuse experience available in our team and avoid throwing
in too many various trendy technologies that will fail our expectations
•
•
•
07

8.
First Blood
08

9.
Ant + Gradle
ant.taskdef(
name: 'scp',
classname: 'o.a.t.a.t.o.ssh.Scp',
classpath: configurations.secureShell.asPath)
ant.taskdef(
name: 'sshexec',
classname: 'o.a.t.a.t.o.ssh.SSHExec',
classpath: configurations.secureShell.asPath)
01.
02.
03.
04.
05.06.
07.
08.
09.
09

10.
Simple call
ant.sshexec(
host: host,
username: user,
password: password,
command: command,
trust: 'true',
failonerror: failOnError)
01.
02.
03.
04.
05.
06.
07.
10

11.
Sshoogr
11

12.
Sshoogr features
Groovy­based SSH DSL for:
Remote command execution
File uploading/downloading
Tunneling
•
•
•
12

13.
Sshoogr
0.9.25!
13

14.
Sshoogr use
cases
14

15.
UC: scripting
15

16.
UC: provisioning
16

17.
UC: testing
17

18.
UC: tunnelling
18

19.
UC: bridging
19

20.
UC: IOT
20

21.
Sshoogr
usage
21

22.
Import
@Grab(
group='com.aestasit.infrastructure.sshoogr',
module='sshoogr',
version='0.9.25')
import static com.aestasit.ssh.DefaultSsh.*
01.
02.
03.
04.
05.
22

23.
Defaults
defaultUser = 'root'
defaultKeyFile = new File('secret.pem')
execOptions {
verbose = true
showCommand = true
}
01.
02.
03.
04.
05.
06.
23

24.
Connection
remoteSession {
url = 'user2:654321@localhost:2222'
exec 'rm ‐rf /tmp/*'
exec 'touch /var/lock/my.pid'
remoteFile('/var/my.conf').text = "enabled=true"
}
01.
02.
03.
04.
05.
06.
24

25.
Multi­line content
remoteFile('/etc/yum.repos.d/puppet.repo').text = '''
[puppet]
name=Puppet Labs Packages
baseurl=http://yum.puppetlabs.com/el/
enabled=0
gpgcheck=0
'''
01.
02.
03.
04.
05.
06.
07.
25

26.
Appendable
remoteFile('/etc/motd') << 'Additional message'
new File('localFile') << remoteFile('/etc/motd')
localFile << remoteFile('/etc/motd') << 'msg'
01.
02.
03.
26

27.
File copying
remoteSession {
scp {
from { localDir "$buildDir/application" }
into { remoteDir '/var/bea/domain/application' }
}
}
01.
02.
03.
04.
05.
06.
27

28.
File copying
remoteSession {
scp {
from { remoteDir '/var/bea/domain/application' }
into { localDir "$buildDir/application" }
}
}
01.
02.
03.
04.
05.
06.
28

29.
File copying
remoteSession {
scp {
from {
remoteFile '/etc/init.d/service1'
remoteFile '/etc/init.d/service2'
}
into { localDir "$buildDir/application" }
}
}
01.
02.
03.
04.
05.
06.
07.
08.
09.
29

30.
File copying
remoteSession {
folders.each { folder ‐>
exec "zip ‐9 ‐q ‐r /tmp/${folder}.zip " +
"/usr/.../storage/${folder}"
scp {
from { remoteFile("/tmp/${folder}.zip") }
into { localDir('./repos/') }
}
}
}
01.
02.
03.
04.
05.
06.
07.
08.
09.
10.
30

31.
File copying
scpOptions {
uploadToDirectory = '/tmp'
postUploadCommand =
'sudo cp ‐R %from%/* %to% && sudo rm ‐rf %from%'
}
01.
02.
03.
04.
05.
31

32.
Command result
def result = exec(command: '/usr/bin/mycmd',
failOnError: false, showOutput: false)
if (result.exitStatus == 1) {
result.output.eachLine { line ‐>
if (line.contains('WARNING')) {
throw new RuntimeException("Warning!!!")
}
}
}
01.
02.
03.
04.
05.
06.
07.
08.
09.
32

33.
Shortcuts
if (ok('/usr/bin/mycmd')) {
...
}
if (fail('/usr/bin/othercmd')) {
...
}
if (commandOutput('ls /')) {
...
}
01.
02.
03.
04.
05.
06.
07.
08.
09.
33

34.
Tunnels
tunnel('1.2.3.4', 8080) { int localPort ‐>
def url = "http://localhost:${localPort}/flushCache"
def result = new URL(url).text
if (result == 'OK') {
println "Cache is flushed!"
} else {
throw new RuntimeException(result)
}
}
01.
02.
03.
04.
05.
06.
07.
08.
09.
34

35.
Prefix/suffix
prefix('sudo ') {
exec 'rm ‐rf /var/log/abc.log'
exec 'service abc restart'
}
suffix(' >> output.log') {
exec 'yum ‐y install nginx'
exec 'yum ‐y install mc'
exec 'yum ‐y install links'
}
01.
02.
03.
04.
05.
06.
07.
08.
09.
35

36.
More features
36

37.
ANSI colors
37

38.
Rainbow Demo
remoteSession(
'vagrant:vagrant@192.168.33.144:22'
) {
exec(
"bash ‐c 'yes "$(seq 231 ‐1 16)" | " +
"while read i; " +
" do printf "x1b[48;5;${i}mn"; " +
"sleep .02; " +
"done'")
}
01.
02.
03.
04.
05.
06.
07.
08.
09.
10.
38

39.
Download progress
39

40.
HTTP proxy
SSH over HTTP? Really? Yes!
defaultProxyHost=192.168.1.2
defaultProxyPort=8080
01.
02.
40

41.
Sshoogr executable
sshoogr ‐h 192.168.43.122
‐u ubuntu
‐l color
default.sshoogr
01.
02.
03.
04.
41

42.
*.sshoogr scripts
No need to install Groovy
Can ommit @Grab
Can ommit connection details
Scripts are more portable
Ansible for Groovy!
•
•
•
•
•
42

43.
*.sshoogr scripts
remoteSession {
exec('uname ‐a')
exec('date')
...
}
01.
02.
03.
04.
05.
43

44.
sdkman.io
44

45.
sdkman.io
curl ‐s http://get.sdkman.io | bash
sdk install sshoogr 0.9.25
sdk list sshoogr
01.
02.
03.
45

46.
Demo
46

47.
More goodies
47

48.
Intellij IDEA
DSL support
48

49.
Intellij IDEA DSL support
49

50.
Intellij IDEA DSL support
50

51.
Demo
51

52.
Intellij IDEA DSL support
https://confluence.jetbrains.com/display/GRVY/Scripting+IDE+for+DSL+awarenes•
52

53.
SSHD Mock
53

54.
SSHD Mock
MockSshServer.with {
command('^ls.*$') { inp, out, err, callback, env ‐>
out << '''total 20
drwxr‐xr‐x 3 1100 1100 4096 Aug 7 16:52 .
drwxr‐xr‐x 8 1100 1100 4096 Aug 1 17:53 ..
drwxr‐xr‐x 3 1100 1100 4096 Aug 7 16:49 examples
callback.onExit(0)
}
...
01.
02.
03.
04.
05.
06.
07.
08.
09.
54

55.
SSHD Mock
...
command('^whoami.*$') { inp, out, err, callback, env ‐>
out << "rootn"
callback.onExit(0)
}
command('^du.*$') { inp, out, err, callback, env ‐>
out << "100n"
callback.onExit(0)
}
...
01.
02.
03.
04.
05.
06.
07.
08.
09.
10.
55

56.
SSHD Mock
...
// Create file expectations.
dir('.')
dir('/tmp')
...
01.
02.
03.
04.
05.
56

57.
SSHD Mock
...
// Start server
startSshd(2233)
}
01.
02.
03.
04.
57

58.
Demo
58

59.
PUnit
59

60.
PUnit
Simple testing tool for verifying remote server state
Uses Sshoogr and JUnit
Reuse reporting features of JUnit
As simple as ...
•
•
•
•
60

61.
PUnit: example (derby)
class DerbyInstallTest
extends BasePuppetIntegrationTest {
@Before
void installDerby() {
apply("include derby")
}
...
}
01.
02.
03.
04.
05.
06.
07.
08.
61

62.
PUnit: example (derby)
@Test
void ensureDerbyRunning() {
command('service derby status > derbystatus.log')
assertTrue fileText("/root/derbystatus.log")
.contains('Derby')
assertTrue fileText("/root/derbystatus.log")
.contains('is running.')
}
01.
02.
03.
04.
05.
06.
07.
08.
62

63.
PUnit: example (derby)
@Test
void ensureCanConnect() {
Thread.sleep(10000)
uploadScript()
command('/opt/derby/db‐derby‐10.9.1.0‐bin/bin/ij ' +
'testDataScript.sql > derbytest.log')
...
01.
02.
03.
04.
05.
06.
07.
63

64.
PUnit: example (derby)
...
// Check if the log of the insert
// operation contains the word ERROR.
assertFalse(
"The script should return at least one error",
fileText("/root/derbytest.log")
.contains('ERROR')
)
...
01.
02.
03.
04.
05.
06.
07.
08.
09.
64

65.
PUnit: example (derby)
...
// Check on data that was inserted into a table.
assertTrue(
"The log should contain a SELECT result",
fileText("/root/derbytest.log")
.contains('Grand Ave.')
)
}
01.
02.
03.
04.
05.
06.
07.
08.
65

66.
PUnit: example (svn)
session {
tunnel ('127.0.0.1', 80) { int localPort ‐>
// Initilize repository connection data.
DAVRepositoryFactory.setup()
def url = SVNURL.create('http', null, '127.0.0.1',
localPort, 'repos/cafebabe', true)
def repository = SVNRepositoryFactory.create(url)
println "Verifying SVN repository at ${url}"
...
01.
02.
03.
04.
05.
06.
07.
08.
09.
66

67.
PUnit: example (svn)
...
// Setup credentials.
def authManager = SVNWCUtil.
createDefaultAuthenticationManager('joe', '123456')
repository.setAuthenticationManager(authManager)
// Verify repository is at revision 0.
assertEquals 0, repository.getLatestRevision()
...
01.
02.
03.
04.
05.
06.
07.
08.
09.
67

68.
PUnit: example (svn)
...
// Commit first revision.
ISVNEditor editor = repository.
getCommitEditor("Initial commit.", null)
editor.with {
openRoot(‐1)
addFile('dummy.txt', null, ‐1)
applyTextDelta('dummy.txt', null)
def deltaGenerator = new SVNDeltaGenerator()
01.
02.
03.
04.
05.
06.
07.
08.
09.
68

69.
PUnit: example (svn)
...
def checksum = deltaGenerator.sendDelta('dummy.txt',
new ByteArrayInputStream("data".getBytes()),
editor, true)
closeFile('dummy.txt', checksum)
def commitInfo = closeEdit()
println commitInfo
}
...
01.
02.
03.
04.
05.
06.
07.
08.
09.
69

70.
PUnit: example (svn)
...
// Verify repository is at revision 1 now.
assertEquals 1, repository.getLatestRevision()
}
}
01.
02.
03.
04.
05.
70

71.
PUnit: continuous integration
71

72.
PUnit: Jenkins build
72

73.
TAP: Test Anything Protocol
1..4
ok 1 ‐ Input file opened
not ok 2 ‐ First line of the input valid
ok 3 ‐ Read the rest of the file
not ok 4 ‐ Summarized correctly
01.
02.
03.
04.
05.
73

74.
TAP with Sshoogr
def tests = [
'sdkman should create candidates directory':
'test ‐d ~/.sdkman/candidates',
'7 groovies should be installed':
'test 7 ‐eq `ls ‐1 ~/.sdkman/candidates/groovy ' +
'| grep ‐v current | wc ‐l`'
]
01.
02.
03.
04.
05.
06.
07.
74

75.
TAP with Sshoogr
remoteSession {
connect()
println "1..${tests.size()}"
tests.eachWithIndex { name, command, index ‐>
if (ok(command)) {
println "ok ${index + 1} ‐ ${name}"
} else {
println "not ok ${index + 1} ‐ ${name}"
}}}
01.
02.
03.
04.
05.
06.
07.
08.
09.
75

76.
TAP with Sshoogr
>>> Connecting to 192.168.33.144
1..2
ok 1 ‐ sdkman should create candidates directory
not ok 2 ‐ 7 groovies should be installed
<<< Disconnected from 192.168.33.144
01.
02.
03.
04.
05.
76

77.
Demo
77

78.
Jenkins TAP plugin
78

79.
Gradle
integration
79

80.
Gradle integration
buildscript {
repositories { mavenCentral() }
dependencies {
classpath 'com.a....sshoogr:sshoogr‐gradle:0.9.18'
}
}
apply plugin: 'secureShell'
01.
02.
03.
04.
05.
06.
07.
80

81.
Gradle integration
task remoteTask << {
remoteSession("user:password@localhost:22") {
exec 'rm ‐rf /tmp/cache/'
scp "$buildDir/cache.content",
'/tmp/cache/cache.content'
}
}
01.
02.
03.
04.
05.
06.
07.
81

82.
Gradle integration
sshOptions {
defaultUser = remoteShellUser
defaultPassword = remoteShellPassword
}
01.
02.
03.
04.
82

83.
gradle.properties (project's home)
remoteShellUser=<PLEASE SET ME>
remoteShellPassword=<PLEASE SET ME>
01.
02.
83

84.
gradle.properties (user's home)
remoteShellUser=andrey
remoteShellPassword=yes_it_is_my_real_password
01.
02.
84

85.
Demo
85

86.
Little brother
86

87.
Groowin
@Grab('com.aestasit.infrastructure.groowin:groowin:0.1.8')
import static com.aestasit.winrm.DefaultWinRM.*
01.
02.
87

88.
Groowin
remoteManagement {
host = '127.0.0.1'
user = 'Administrator'
password = 'secret'
exec 'del', 'C:temp.txt'
remoteFile('C:my.conf').text = "enabled=true"
}
01.
02.
03.
04.
05.
06.
07.
88

89.
Summary
89

90.
Sshoogr is...
Battle­tested Groovy DSL for SSH connectivity
Executable and portable scripting tool
Easily integratable with any Java/Groovy library
•
•
•
90

91.
Sshoogr can be used for...
Provisioning your servers and IoT devices
Executing remote orchestration commands
Testing and monitoring infrastructure state
•
•
•
91

92.
Next steps
Resource definitions
Command rollbacks
Parallel execution
XSS utilities
Extend integration tests
Better documentation
•
•
•
•
•
•
92

93.
Seeking
contributors!
93

94.
Source code
Sshoogr: https://github.com/aestasit/sshoogr
Sshoogr Gradle: https://github.com/aestasit/sshoogr­gradle
Groowin: https://github.com/aestasit/groowin
Groowin Gradle: https://github.com/aestasit/groowin­gradle
•
•
•
•
94

95.
Source code
Sshd­mock: https://github.com/aestasit/groovy­sshd­mock
P­unit: https://github.com/aestasit/p­unit
WinRM client: https://github.com/aestasit/groovy­winrm­client
•
•
•
95

96.
Thank you!
96

97.
97