VMWare is a common hypervisor choice in large organizations, and it comes with a zoo of additional tools, options and licenses. But once you add a little bit of OpenSource to the mix, things start to get interesting. There are open APIs which integrate well with Tools like Ansible and InSpec. So it is easy to write your own scripts to verify a configuration or harden a ESX host. We will show you, how we automated our deployment, what problems we encountered and how we added some nice features.

1. Spicing up VMware with
Ansible and InSpec
T-Systems Multimedia Solutions GmbH

2. Martin Schurz
Sebastian Gumprich
T-Systems MMS
T-Systems Multimedia Solutions GmbH

3. Ops: the old days (tm)
T-Systems Multimedia Solutions GmbH

4. Ops: the old days (tm)
T-Systems Multimedia Solutions GmbH

5. Ops: the old days (tm)
handcrafted and sometimes arcane con guration
clusters
parameters for Oracle
the "one" server someone installed
virtualization is just lift and shift
T-Systems Multimedia Solutions GmbH

6. Ops: slowly improving
reliance on enterprise tools
vSphere / vRealize / vCloud
T-Systems Multimedia Solutions GmbH

7. Ops: slowly improving
reliance on enterprise tools
vSphere / vRealize / vCloud
T-Systems Multimedia Solutions GmbH

8. VMware
T-Systems Multimedia Solutions GmbH

9. We have a lot of pets,
but we need more cattle
T-Systems Multimedia Solutions GmbH

10. Mantra:
manual work is a bug!
T-Systems Multimedia Solutions GmbH

11. T-Systems Multimedia Solutions GmbH

12. Why Ansible?
because we don't like Puppet
Ansible is simple, agent-less
easy to learn
straight-forward in task execution
Not written in Ruby (looking @ you, Puppet)
T-Systems Multimedia Solutions GmbH

13. Ansible - quick rundown
T-Systems Multimedia Solutions GmbH

14. Ansible modules - many of them
T-Systems Multimedia Solutions GmbH

15. ... except Oracle
pet, not cattle.
T-Systems Multimedia Solutions GmbH

16. Jenkins incoming
already reliable application deployments
now reliable con guration of servers, too
T-Systems Multimedia Solutions GmbH

17. Automation is fun!
... or so they say ...
T-Systems Multimedia Solutions GmbH

18. Automation is fun!
... or so they say ...
due to "unfortunate circumstances" we lost half
our servers
“
“
T-Systems Multimedia Solutions GmbH

19. Automation is fun!
... or so they say ...
... and we did recover easily
due to "unfortunate circumstances" we lost half
our servers
“
“
T-Systems Multimedia Solutions GmbH

20. Automation is fun!
... or so they say ...
... and we did recover easily
due to "unfortunate circumstances" we lost half
our servers
“
“
T-Systems Multimedia Solutions GmbH

21. Managing ESX Hosts
Prebuild modules for all basic tasks:
Network
Storage
Cluster
vCenter
VM tasks
T-Systems Multimedia Solutions GmbH

22. Managing ESX Hosts (example)
I want to con gure all VLANs for my ESX Cluster
All Hosts should have correct VLAN con guration
All Hosts should be con gured from one source
Adding Hosts and VLANs should be easy
(like a distributed vSwitch)
T-Systems Multimedia Solutions GmbH

23. Managing ESX Hosts (example)
create a host group (e.g. esx-servers )
Add group_vars:
vlans:
customer1-vlan:
tag: 4006
vswitch: vSwitch0
customer2-vlan:
tag: 4007
vswitch: vSwitch0
...
T-Systems Multimedia Solutions GmbH

24. Managing ESX Hosts (example)
Add a playbook task:
- hosts: esx-servers
tasks:
- name: "Add VLANs"
local_action:
module: vmware_portgroup
hostname: '{{ ansible_hostname }}'
username: root
password: '{{ esxi_pass }}'
switch_name: "{{ item.value.vswitch }}"
portgroup_name: "{{ item.key }}"
vlan_id: "{{ item.value.tag }}"
validate_certs: false
with_dict: "{{ vlans }}"
T-Systems Multimedia Solutions GmbH

25. Managing ESX Hosts (example)
rst Ansible run
TASK [Add VLANs] *****************************************
ok: [esx_server] => (item={'value':
{u'vswitch': u'vSwitch0', u'tag': 4006},
'key': u'customer1-vlan'})
changed: [esx_server] => (item={'value':
{u'vswitch': u'vSwitch0', u'tag': 4007},
'key': u'customer2-vlan'})
PLAY RECAP ***********************************************
esx_server : ok=1 changed=1 unreachable=0 failed=0
T-Systems Multimedia Solutions GmbH

26. Managing ESX Hosts (example)
second Ansible run
TASK [Add VLANs] *****************************************
ok: [esx_server] => (item={'value':
{u'vswitch': u'vSwitch0', u'tag': 4006},
'key': u'customer1-vlan'})
ok: [esx_server] => (item={'value':
{u'vswitch': u'vSwitch0', u'tag': 4007},
'key': u'customer2-vlan'})
PLAY RECAP ***********************************************
esx_server : ok=1 changed=0 unreachable=0 failed=0
T-Systems Multimedia Solutions GmbH

27. Managing ESX Hosts (example)
I want to con gure all VLANs for my ESX Cluster
All Hosts should have correct VLAN con g
All Hosts should be con gured from one source
Adding Hosts and VLANs should be easy
(like a distributed vSwitch)
T-Systems Multimedia Solutions GmbH

28. Creating VMs - Host variables
vm_cpu: 8
vm_ram: 8
vm_storage: srv_live_vmdata1
vm_host: srv-live-vh07
vm_disksize: 80
default_gateway: 172.31.225.1
network_ether_interfaces:
- vm_net: srv-lgen-app
device: eth0
bootproto: static
address: 172.31.225.36
netmask: 255.255.255.128
onboot: "yes"
dns1: "{{ srv_dns1 }}"
dns2: "{{ srv_dns2 }}"
domain: "{{ srv_domain }}"
T-Systems Multimedia Solutions GmbH

29. Creating VMs - the Ansible task
- name: Create new VM
vmware_guest:
hostname: "{{ vcenter_host }}"
username: "{{ vcenter_user }}"
password: "{{ vcenter_pass }}"
datacenter: "{{ vcenter_dc }}"
name: "{{ item }}"
template: "{{ vm_template }}"
state: poweredon
wait_for_ip_address: yes
hardware:
memory_mb: "{{hostvars[item]['vm_ram']}}"
num_cpus: "{{hostvars[item]['vm_cpu']}}"
disk:
- size_gb: "{{hostvars[item]['vm_disksize']}}"
datastore: "{{hostvars[item]['vm_storage']}}"
T-Systems Multimedia Solutions GmbH

30. Adding Security to the mix
Telekom security guideline requires all servers to
be hardened
also VMware security guideline:
https://www.vmware.com/security/hardening-
guides.html (beware Excel!)
T-Systems Multimedia Solutions GmbH

31. Hardening an ESX host (example)
VMware Requirement:
Guideline ID: ESXi.disable-mob:
The managed object browser (MOB) provides a
way to explore the object model used by the
VMkernel to manage the host; it enables
con gurations to be changed as well. This
interface is meant to be used primarily for
debugging the vSphere SDK. In Sphere 6.0 this
is disabled by default
T-Systems Multimedia Solutions GmbH

32. Hardening an ESX host (example)
Ansible implementation:
# Guideline ID: ESXi.disable-mob
- name: get | disable MOB
shell: "vim-cmd hostsvc/advopt/view
Config.HostAgent.plugins.solo.enableMob
| grep value | cut -d ' ' -f 9"
register: mob_status
changed_when: mob_status.rc > 0
- name: set | disable MOB
shell: "vim-cmd hostsvc/advopt/update
Config.HostAgent.plugins.solo.enableMob
bool {{ mob }}"
when: mob not in mob_status.stdout
T-Systems Multimedia Solutions GmbH

33. Hardening VMs - nding them all!
- name: Find all .vmx files on local store
shell: |
find /vmfs/volumes/datastore/ -name *.vmx
register: found_vms
changed_when: False
T-Systems Multimedia Solutions GmbH

34. Hardening VMs - changing them
- name: Set VM parameters
lineinfile:
path: "{{ item[1] }}"
regexp: "{{ item[0].key }}"
backrefs: yes
line: "{{ item[0].key }} = "{{ item[0].value }}""
with_nested:
- "{{ parameters_add }}"
- "{{ found_vms }}"
parameters_add:
- { key: isolation.tools.copy.disable, value: TRUE }
- { key: isolation.tools.paste.disable, value: TRUE }
T-Systems Multimedia Solutions GmbH

35. Managing VMs - deleting them
- name: delete VM
vmware_guest:
vcenter_hostname: "{{ vcenter_host }}"
username: "{{ vcenter_user }}"
password: "{{ vcenter_pass }}"
validate_certs: false
guest: "{{ item }}"
force: true
state: absent # deletion!
with_items: "{{ vm_name }}"
T-Systems Multimedia Solutions GmbH

36. Managing VMs - making snapshots
- name: Create snapshot of {{vm_name}}
vmware_guest_snapshot:
folder: "/vm/"
name: "{{ vm_name }}"
state: present
snapshot_name: "snap_{{ '%Y-%m-%d-%M' | strftime }}"
T-Systems Multimedia Solutions GmbH

37. Not everything out of the box
moving VMs not implemented in Ansible :(
but Ansible is extensible with Python code
so just write your own module
VMware vSphere API Bindings for Python
(https://github.com/vmware/pyvmomi)
VMware API Docs
Python + API =
T-Systems Multimedia Solutions GmbH

38. Not everything out of the box
we started with Ansible code:
- name: Move VM to target host and DS
delegate_to: localhost
vm_move:
vc_host: "{{ vcenter_host }}"
vc_pass: "{{ vcenter_pass }}"
vc_user: "{{ vcenter_user }}"
vm_name: "{{ inventory_hostname }}"
ds_name: "{{ vm_storage }}"
esx_host: "{{ vm_host }}"
T-Systems Multimedia Solutions GmbH

39. Not everything out of the box
then think about implementation
what needs to be done:
Locate the VM we want to move
T-Systems Multimedia Solutions GmbH

40. Not everything out of the box
then think about implementation
what needs to be done:
Locate the VM we want to move
Locate the target ESX host / storage
T-Systems Multimedia Solutions GmbH

41. Not everything out of the box
then think about implementation
what needs to be done:
Locate the VM we want to move
Locate the target ESX host / storage
check what needs to be changed
T-Systems Multimedia Solutions GmbH

42. Not everything out of the box
then think about implementation
what needs to be done:
Locate the VM we want to move
Locate the target ESX host / storage
check what needs to be changed
move the VM
T-Systems Multimedia Solutions GmbH

43. Not everything out of the box
some boilerplate is needed:
def main():
module = AnsibleModule(
argument_spec=dict(
vc_host = dict(required=True, type='str'),
...
esx_host = dict(required=False, type='str'),
),
)
result = dict(
changed=False, original_message='', message=''
)
# do something
module.exit_json(**result)
T-Systems Multimedia Solutions GmbH

44. Not everything out of the box
vm = get_obj(content, [vim.VirtualMachine], vm_name)
vm_datastore = get_obj(content, [vim.Datastore], ds_name)
dest_host = get_obj(content, [vim.HostSystem], esx_host)
vm_relocate_spec = vim.vm.RelocateSpec()
T-Systems Multimedia Solutions GmbH

45. Not everything out of the box
vm = get_obj(content, [vim.VirtualMachine], vm_name)
vm_datastore = get_obj(content, [vim.Datastore], ds_name)
dest_host = get_obj(content, [vim.HostSystem], esx_host)
vm_relocate_spec = vim.vm.RelocateSpec()
if vm.datastore[0] != vm_datastore:
result['changed'] = True
vm_relocate_spec.datastore = vm_datastore
T-Systems Multimedia Solutions GmbH

46. Not everything out of the box
vm = get_obj(content, [vim.VirtualMachine], vm_name)
vm_datastore = get_obj(content, [vim.Datastore], ds_name)
dest_host = get_obj(content, [vim.HostSystem], esx_host)
vm_relocate_spec = vim.vm.RelocateSpec()
if vm.datastore[0] != vm_datastore:
result['changed'] = True
vm_relocate_spec.datastore = vm_datastore
if vm.runtime.host != dest_host:
result['changed'] = True
vm_relocate_spec.host = dest_host
T-Systems Multimedia Solutions GmbH

47. Not everything out of the box
vm = get_obj(content, [vim.VirtualMachine], vm_name)
vm_datastore = get_obj(content, [vim.Datastore], ds_name)
dest_host = get_obj(content, [vim.HostSystem], esx_host)
vm_relocate_spec = vim.vm.RelocateSpec()
if vm.datastore[0] != vm_datastore:
result['changed'] = True
vm_relocate_spec.datastore = vm_datastore
if vm.runtime.host != dest_host:
result['changed'] = True
vm_relocate_spec.host = dest_host
if result['changed']:
task = vm.Relocate(spec=vm_relocate_spec)
wait_for_task(module, task, si)
T-Systems Multimedia Solutions GmbH

48. Not everything out of the box
VMWare has a tool called govc
https://github.com/vmware/govmomi/tree/mast
er/govc
pretty easy to use from the command line
this can also be included in Ansible scripts
but do I really need to write all this python code?
I'm not a programmer!
“
“
T-Systems Multimedia Solutions GmbH

49. Testing
T-Systems Multimedia Solutions GmbH

50. Testing with inSpec
written by Chef guys
originally a fork of serverspec
diverged since then and has gotten many new
features
T-Systems Multimedia Solutions GmbH

51. Testing with inSpec - the test
control 'VM.disable-console-drag-n-drop' do
title 'Explicitly disable copy/paste operations'
vsphere.datacenters.each { |dc|
dc.vms.each { |vm|
describe vm_advancedsetting) do
its(['isolation.tools.dnd.disable'])
{ should eq true }
end
}
}
end
T-Systems Multimedia Solutions GmbH

52. Testing with inSpec - results
VM.disable-console-drag-n-drop
isolation.tools.dnd.disable should eq true
Profile Summary: 136 successful controls, 0 failures
Test Summary: 136 successful, 0 failures, 0 skipped
T-Systems Multimedia Solutions GmbH

53. Bonus - ansible-cmdb
T-Systems Multimedia Solutions GmbH

54. The End
Now grab some food!
T-Systems Multimedia Solutions GmbH

55. Ansible logo from redbubble.com
VMWare logo from fujitsu
InSpec logo from sdtimes
Fry from ickr user liliana_von_k
success kid from instagram user laneymg
automate from ickr user Amber Case
Ansible works image from tutorialspoint.com
T-Systems Multimedia Solutions GmbH