How do you deploy a traditional on-premise solution as Software as a Service? Here, I am presenting architectural decisions and challenges faced in building and maturing an orchestration service for automating and managing deployments of Dynatrace, an application monitoring solution at enterprise scale, into Amazon's AWS cloud. The focus of this presentation is on our experiences with Ansible, a radically simple, yet powerful, IT automation tool, whose agentless architecture and extensive modules library greatly simplified cloud service orchestration and integration of our solution with customer applications. Besides, automated testing of our deployment scripts with Test Kitchen and serverspec is discussed.

1. 1 #Dynatrace
QCon London 2015
Martin Etmajer, Technology Strategist @ Dynatrace
Deploying On-Prem as SaaS
Why we go with Ansible

2. 2 #Dynatrace
Martin Etmajer
Technology Strategist @Dynatrace
martin.etmajer@dynatrace.com
@metmajer
http://blog.dynatrace.com

3. 3 #Dynatrace
Agenda
» Dynatrace On-Prem
» Dynatrace SaaS Architecture
» Ansible as Orchestration Engine
» Test-Driven Infrastructure with Test Kitchen

4. 4 #Dynatrace
Dynatrace

5. 5 #Dynatrace
Dynatrace On-Prem Environment

6. 6 #Dynatrace
Dynatrace On-Prem Environment

7. 7 #Dynatrace
Dynatrace On-Prem Environment

8. 8 #Dynatrace
Dynatrace On-Prem Environment

9. 9 #Dynatrace
Dynatrace On-Prem Environment

10. 10 #Dynatrace
Dynatrace. So What?
Presentation Tier
Business Logic Tier

11. 11 #Dynatrace
Dynatrace. So What?
# Requests
Health
Timings

12. 12 #Dynatrace
Dynatrace. So What?
Frustrated Users

13. 13 #Dynatrace
Dynatrace. So What?
What happened?

14. 14 #Dynatrace
Dynatrace. So What?
Erroneous Call Username
SQL Statements
Exception

15. 15 #Dynatrace
Reasons for Going SaaS

16. 16 #Dynatrace
Reasons for Going SaaS
#1: Operational Risk
Moves from Customer to Service Provider

17. 17 #Dynatrace
Reasons for Going SaaS
#2: Customers Want Solutions
Without Buying Extra Infrastructure

18. 18 #Dynatrace
Reasons for Going SaaS
#3: Customers Already in the Cloud

19. 19 #Dynatrace
Dynatrace SaaS Architecture

20. 20 #Dynatrace
On-Prem Dynatrace Environment

21. 21 #Dynatrace
Dynatrace SaaS Environment

22. 22 #Dynatrace
Dynatrace SaaS Environment

23. 23 #Dynatrace
Dynatrace SaaS Environment

24. 24 #Dynatrace
for Dynatrace SaaS
Elastic Compute Cloud (EC2)

25. 25 #Dynatrace
for Dynatrace SaaS
Elastic Block Store (EBS)

26. 26 #Dynatrace
for Dynatrace SaaS
Relational Database Service (RDS)

27. 27 #Dynatrace
for Dynatrace SaaS
Elastic IP (EIP)

28. 28 #Dynatrace
for Dynatrace SaaS
Route 53 (DNS)

29. 29 #Dynatrace
for Dynatrace SaaS
Simple Storage Service (S3)

30. 30 #Dynatrace
for Dynatrace SaaS
CloudWatch

31. 31 #Dynatrace
Design Decisions

32. 32 #Dynatrace
#1 – Dedicated Customer Resources
Clean Separation of Customers & Data

33. 33 #Dynatrace
#1 – Dedicated Customer Resources
Customizable Security
IP-Based Access Rules

34. 34 #Dynatrace
#1 – Dedicated Customer Resources
Grow with the Customer
Resize Resources on Demand

35. 35 #Dynatrace
#2 – Data Persistence
Prevent Data Loss
When Resources Go Down

36. 36 #Dynatrace
#3 – Regular Backups
Enable < Daily Data Recovery
All Configuration and Data

37. 37 #Dynatrace
Orchestration Platform

38. 38 #Dynatrace
Orchestration Platform

39. 39 #Dynatrace
Orchestration Platform

40. 40 #Dynatrace
Orchestration Platform

41. 41 #Dynatrace
How We Did It
Lessions Learned

42. 42 #Dynatrace
We Coded Our Own Solution
#1: Lifecycle Management of Resources

43. 43 #Dynatrace
We Coded Our Own Solution
#2: Use-Cases

44. 44 #Dynatrace
We Coded Our Own Solution
#3: Monitoring

45. 45 #Dynatrace
We Coded Our Own Solution
Extensive Testing
Too much Manual Work Involved
Large Codebase, Tied to Single Vendor

46. 46 #Dynatrace
We Coded Our Own Solution

47. 47 #Dynatrace
Hey, Ansible!

48. 48 #Dynatrace
Ansible is a radically simple IT automation engine for
» environment and infrastructure provisioning
» configuration management
» application deployment
» etc.
What is Ansible?

49. 49 #Dynatrace
“I wrote Ansible because none of the existing tools fit
my brain. I wanted a tool that I could not use for 6
months, come back later, and still remember how it
worked.”
Michael DeHaan, Ansible Founder
“We need to do a rolling deployment of changes that
have certain dependencies (including external
services).
With Ansible this becomes trivial.
Puppet on the other hand feels like the Wild West.”
User IUseRhetoric on reddit.com
Why Yet Another Tool?

50. 50 #Dynatrace
» No Agents ✓
» No Scripting ✓
» Simple and Powerful ✓
Ansible Design Principles

51. 51 #Dynatrace
Agent-Based Architecture

52. 52 #Dynatrace
Agent-Based Architecture

53. 53 #Dynatrace
Agent-Based Architecture

54. 54 #Dynatrace
Ansible’s Agentless Architecture

55. 55 #Dynatrace
Ansible’s Agentless Architecture

56. 56 #Dynatrace
Ansible’s Agentless Architecture

57. 57 #Dynatrace
Ansible’s Agentless Architecture

58. 58 #Dynatrace
Ansible is an Orchestration Engine. So What?

59. 59 #Dynatrace
Ansible is an Orchestration Engine. So What?

60. 60 #Dynatrace
Ansible is an Orchestration Engine. So What?

61. 61 #Dynatrace
Ansible is an Orchestration Engine. So What?

62. 62 #Dynatrace
Ansible is an Orchestration Engine. So What?

63. 63 #Dynatrace
Ansible is an Orchestration Engine. So What?

64. 64 #Dynatrace
Ansible is an Orchestration Engine. So What?

65. 65 #Dynatrace
Ansible for the Cloud

66. 66 #Dynatrace
Ansible Cloud Modules

67. 67 #Dynatrace
Ansible Cloud Modules

68. 68 #Dynatrace
Ansible Cloud Modules for Amazon

69. 69 #Dynatrace
Ansible Concepts: Inventories

70. 70 #Dynatrace
» Ansible provisions groups of servers at once
» Groups and hosts are defined in inventories
» Use inventories for staging, production, etc.
Static vs. Dynamic Inventories
» Static: text files expressed in an INI-like format
» Dynamic: Python scripts for dynamic environments (cloud)
» Static + Dynamic: combine multiple inventories (hybrid cloud)
Ansible Inventories

71. 71 #Dynatrace
Example: Static Inventory
# file: production
[balancers]
www.example.com
[webservers]
www[0-9].example.com
[dbservers]
db[a:f].example.com
[monitoring]
dynatrace.example.com
Group
Host
Numeric Range
Alphabetic Range

72. 72 #Dynatrace
Python scripts that get data from dynamic sources such as:
» Cloud: Amazon, DigitalOcean, Google, OpenShift, OpenStack, etc.
» Distributed Information Services: LDAP, etcd, etc.
Dynamic Inventories

73. 73 #Dynatrace
Ansible Concepts: Playbooks

74. 74 #Dynatrace
Ansible Playbooks
$> ansible-playbook [–i <inventory>] <playbook.yml>
What is a Playbook?
» Describes policies your managed machines shall enforce
» Consist of vars, tasks, handlers, files, templates and roles
» Expressed in the YAML format (dictionaries, lists and scalars)

75. 75 #Dynatrace
Example: Ansible Playbook
--- # file: webservers.yml
- hosts: webservers
handlers:
- name: reload apache2
service: name=apache2 state=reloaded
tasks:
- name: Install Apache HTTP Server
apt: name=apache2 update_cache=yes
- name: Install Apache Modules
apache2_module: name={{ item }} state=present
with_items:
- proxy
- proxy_httpd
notify: reload apache2
remote_user: deploy
sudo: yes
Play
Module Arguments
Variable
Notify Handler

76. 76 #Dynatrace
Example: Ansible Playbook
--- # file: playbook.yml
- include: balancers.yml
- include: webservers.yml
- include: dbservers.yml
- include: monitoring.yml

77. 77 #Dynatrace
Example: Ansible Playbook
$> ansible-playbook –i production webservers.yml
PLAY [webservers]
*******************************************************
TASK: [Install Apache HTTP Server]
*******************************************************
changed: [www0.example.com]
changed: [www1.example.com]
...
PLAY RECAP
*******************************************************
web0.example.com: ok=3 changed=3 unreachable=0 failed=0
web1.example.com: ok=3 changed=3 unreachable=0 failed=0
...

78. 78 #Dynatrace
Ansible Concepts: Roles

79. 79 #Dynatrace
Ansible Roles
» Are the best way to organize a playbook!
» Structure content into related vars, tasks, files, handlers, etc.
» File structure for automated inclusion of role-specific content
» Roles can be shared and pulled from Ansible Galaxy, GitHub, etc.

80. 80 #Dynatrace
Ansible Roles: File Structure
ansible.cfg
production
staging
webservers.yml
roles/
common/
defaults/
files/
handlers/
meta/
tasks/
templates/
vars/
apache2/
...

81. 81 #Dynatrace
Example: Ansible Playbook
--- # file: webservers.yml
- hosts: webservers
roles:
- { role: common }
- { role: apache2 }
remote_user: deploy
sudo: yes

82. 82 #Dynatrace
Ansible: Our Use-Cases

83. 83 #Dynatrace
Use-Case #1
Create Dynatrace AMI

84. 84 #Dynatrace
#1 – Create Dynatrace AMI
#1: Remove Existing Dynatrace AMI

85. 85 #Dynatrace
#1 – Create Dynatrace AMI
#2: Launch Builder Instance

86. 86 #Dynatrace
#1 – Create Dynatrace AMI
#3: Add Persistent Volumes

87. 87 #Dynatrace
#1 – Create Dynatrace AMI
#4: Install Dynatrace

88. 88 #Dynatrace
#1 – Create Dynatrace AMI
#5: Bundle Dynatrace AMI

89. 89 #Dynatrace
#1 – Create Dynatrace AMI
#6: Register Dynatrace AMI

90. 90 #Dynatrace
#1 – Create Dynatrace AMI
#7: Clean Up Temporary Resources

91. 91 #Dynatrace
Use-Case #2
Create Dynatrace Instance

92. 92 #Dynatrace
#2 – Launch Dynatrace Instance
#1: Get Dynatrace AMI ID

93. 93 #Dynatrace
#2 – Launch Dynatrace Instance
#2: Launch Dynatrace Instance

94. 94 #Dynatrace
#2 – Launch Dynatrace Instance
#3: Associate Static IP

95. 95 #Dynatrace
#2 – Launch Dynatrace Instance
#4: Associate Domain Name

96. 96 #Dynatrace
#2 – Launch Dynatrace Instance
#5: Launch Database Instance

97. 97 #Dynatrace
Use-Case #3
Deploy Dynatrace Agents

98. 98 #Dynatrace
Dynatrace On-Prem Environment

99. 99 #Dynatrace
Test-Driven Infrastructure
with Test Kitchen
Because your Infrastructure deserves tests, too!

100. 100 #Dynatrace
What is Test Kitchen?
Test Kitchen allows you to test your infrastructure as code
» on multiple platforms in isolation
» supporting a wide range of drivers
» with support for various provisioners
» by using a variety of testing frameworks
» in a pluggable architecture
CentOS, Debian, Fedora, etc.
Vagrant, Docker, EC2, etc.
Ansible, Chef, Puppet
bash, Cucumber, Serverspec, etc.
extend as desired!

101. 101 #Dynatrace
Test Kitchen: Configure
--- # file: .kitchen.yml
provisioner:
name: ansible_playbook
require_ansible_repo: true
driver:
name: vagrant
customize:
cpus: 2
memory: 2048
cpuexecutioncap: 50
platforms:
# Opscode vagrant boxes: http://www.vagrantbox.es/
- name: centos-6.5
- name: ubuntu-12.04
suites:
- name: dynatrace-agents
- name: dynatrace-server
your test suites here!

102. 102 #Dynatrace
Test Kitchen: Ansible Playbook
# file: test/integration/dynatrace-server/default.yml
---
- hosts: all
roles:
- role: Dynatrace-Server
foo: bar
remote_user: vagrant
define a particular role under test

103. 103 #Dynatrace
Test Kitchen: Serverspec
# file: test/integration/dynatrace-server/serverspec/spec.rb
require 'serverspec’
...
describe user('dynatrace') do
it { should exist }
it { should belong_to_group 'dynatrace' }
end
describe service('dynaTraceServer') do
it { should be_enabled }
it { should be_running }
end
describe port(2021) do
it { should be_listening }
end
declare your assumptions

104. 104 #Dynatrace
Test Kitchen: List Instances
$> kitchen list
Instance Driver Provisioner
dynatrace-agents-centos-65 Vagrant AnsiblePlaybook
dynatrace-agents-ubuntu-1204 Vagrant AnsiblePlaybook
dynatrace-server-centos-65 Vagrant AnsiblePlaybook
dynatrace-server-ubuntu-1204 Vagrant AnsiblePlaybook
...
let’s test dynatrace-server* !

105. 105 #Dynatrace
Test Kitchen: Test Instances
$> kitchen test dynatrace-server
-----> Starting Kitchen (v1.3.1)
-----> Cleaning up any prior instances of <dynatrace-server-centos-65>
...
-----> Creating <dynatrace-server-centos-65>...
Bringing machine 'default' up with 'virtualbox' provider...
...
Finished in 0.6908 seconds (files took 0.74047 seconds to load)
24 examples, 0 failures
-----> Starting Kitchen (v1.3.1)
-----> Cleaning up any prior instances of <dynatrace-server-ubuntu-1204>
...
-----> Creating <dynatrace-server-ubuntu-1204>...
Bringing machine 'default' up with 'virtualbox' provider...
...
Finished in 0.6947 seconds (files took 0.82016 seconds to load)
24 examples, 0 failures

106. 106 #Dynatrace
Questions?

107. 107 #Dynatrace

108. 108 #Dynatrace