[HKOSCON][20190615][DevOps with Ansible, From Native to Kubernetes]

1. Hong Kong Open Source
Conference 2019
DevOps with Ansible
From Native to Kubernetes
Edison Wong
2019-06-15

2. Wong Hoi Sing, Edison
●
2005 - Drupal Developer & Contributor
– https://drupal.org/user/33940
●
2008 - HKDUG Co-founder
– https://groups.drupal.org/drupalhk
●
2010 - CEO, PantaRei Design
– hswong3i@pantarei-design.com

7. PantaRei Design
●
Everything Changes and Nothing Remains Still
●
Reinvent Enterprise with Open Source Software and Cloud Computing
●
Hong Kong based FOSS service provider
– Content Management System (CMS) with Drupal
– Cloud Hosting Solution with Amazon Web Services (AWS)
– Team collaborate solution with Atlassian
●
Business Partner with industry leaders
– 2012, AWS Consulting Partner
– 2013, Acquia Partner
– 2013, Atlassian Experts
– 2014, Rackspace Hosting Partner
●
http://pantarei-design.com

9. Outline
●
Why DevOps with Ansible?
●
Hello World with Ansible
●
Ansible Role with Molecule
●
Docker Build with Ansible (!?)
●
Molecule + Vagrant + Ceph + Kubernetes
●
Tips & Tricks
●
Roadmap
●
Q&A

10. Why DevOps with Ansible?
●
HKOSCON 2018
●
SysAdmin Daily Difficulties
●
Why DevOps?
●
Why Ansible?
●
Demo: alvistack-ansible

11. HKOSCON 2018
●
Virtual Hosting with Kubernetes
– Ansible: deployment management
– Docker: application pre-packaging
– CephFS: centralized network storage
– Kubernetes: runtime service lifecycle
management

15. HKOSCON 2018 (cont.)
●
Each talent (i.e. domain) goes into
individual Kubernetes namespace
●
Each namespace have its own Apache,
PHP, MySQL, SSHD, etc
●
All dynamic data store outside
Kubernetes, goes into CephFS

16. SysAdmin Daily Difficulties
●
Different deployment target
●
Test logic before deploy
●
Complex cluster management
●
Documentation
●
No time for learning

17. SysAdmin Daily Difficulties
(cont.)
●
Write-once for all cases
– Native
– Docker/LXD/Vagrant
– AWS/GCE/Azure

19. Why DevOps?
●
Manual install
– Non-repeatable
●
Manual install with document
– Difficult to manage (Docs to Action)
– Always async with production
●
Manual scripting
– Difficult for everything: learn, write, error detection,
debug, etc…

20. Why DevOps? (cont.)
●
DevOps
– Deployment logic as code (i.e. revision
with GIT)
– With error detection and debug tools
– Manage multiple deployment target at
once (e.g. data center, clustering)

21. Why Ansible?
●
Writing “tasks” in YAML
– Human readable == minimize
documentation
– Easy to learn, when compare with
Ruby for Chef or Puppet

22. Why Ansible? (cont.)
A lot of reusable modules
– Simplify complicated logic with error
detection
– Or running “shell” command directly

23. Why Ansible? (cont.)
●
Simple requirement
– Python and Password-less SSH
– Agent-less for managed node

26. Demo: alvistack-ansible
●
https://github.com/alvistack/alvistack-ansibl
e
– Ceph + Kubernetes + Addon
– Support Ubuntu 18.04/16.04 + CentOS 7 +
openSUSE 15.0
– All Roles tested with LXD individually
– Overall Playbook tested with Vagrant
– Simply clone-and-play

28. Hello World with Ansible
●
Install and Setup Ansible
●
Ansible in CLI
●
Ansible in Playbook
●
Ansible in Role
●
Demo: ansible-playbook-ubuntu-desktop

29. Install and Setup Ansible
●
Control Node Requirements
– sudo apt-get install python python-dev openssh-client
– pip install ansible molecule
●
Managed Node Requirements
– sudo apt-get install python-minimal openssh-server
– Password-less SSH login by public-private key pair

30. Ansible in CLI
●
Running command on remote guest is
simple
– ansible -i guest1,guest2, -m ping
– ansible -i guest1,guest2, -m apt -a ‘name=vim
state=present’
– ansible -i guest1,guest2, -m shell -a ‘uname -a’

31. Ansible in Playbook
●
Running multiple “task” once together
●
Finer control than running with CLI
●
Define your inventory then play with it
– ansible-playbook -i inventory/all/hosts
playbooks/setup-everything.yml

34. Ansible in Role
●
Not just “Tasks”, but also:
– Default over-writable variables
– Internal static variables
– Static files for copy
– Template files
– Event handlers
●
A basic build block for complex architecture
– Use Playbook to include different Roles

35. Ansible in Role (cont.)
●
Create a new role with ansible-galaxy
– mkdir ~/.ansible/roles
– cd ~/.ansible/roles
– ansible-galaxy init dummy
●
You could now test it (run via your localhost)
– cd ~/.ansible/roles/dummy
– ansible-playbook -i tests/inventory tests/test.yml

37. Demo: ansible-playbook-
ubuntu-desktop
●
https://github.com/pantarei/ansibl
e-playbook-ubuntu-desktop
– Deployment for Ubuntu Desktop
19.04 + Lot of daily used applications
– Well… Also install `vanilla-gnome-
desktop`…
– Simply Ansible Playbook-only

39. Ansible Role with Molecule
●
Install and Setup Molecule
●
Why Molecule?
●
Multiple Test Cases
●
Multiple Test Drivers
●
Demo: ansible-role-sshd

40. Install and Setup Molecule
●
Molecule could be install with PIP
– pip install ansible molecule
●
Also install with Docker and Vagrant
support
– pip install molecule[docker]
– pip install molecule[vagrant]

41. Why Molecule?
●
DevOps R&D need test
– Role created by ansible-galaxy run via localhost
●
Each test should run in a clean environment
– Role created by molecule run inside
Docker/LXD/Vagrant/etc...
●
Manage test environment(s) is boring
●
Code lint

42. Why Molecule? (cont.)
●
Molecule
– Testing framework for Ansible
– Written in Ansible and Python style
– Write your test case in standard Ansible style
– Manage test environment life-cycle for you
– Code lint
– Idempotence (i.e. run twice to confirm no extra
changes)

43. Why Molecule? (cont.)
●
Create a new Role with molecule
– cd ~/.ansible/roles
– molecule init role -r dummy2 -d docker
– molecule init role -r dummy3 -d lxd
– molecule init role -r dummy4 -d vagrant
●
Now you could run test inside Docker
– cd ~/.ansible/roles/dummy2
– molecule test

46. Multiple Test Cases
●
Molecule support multiple test cases
definition, e.g.
– molecule/centos-7/molecule.yml
– molecule/suse-15/molecule.yml
– molecule/ubuntu-16.04/molecule.yml
– molecule/ubuntu-18.04/molecule.yml

47. Multiple Test Drivers
●
Molecule support multiple drivers
– Docker (default)
– LXD
– Vagrant (i.e. VirtualBox)
– EC2/GCE/Delegated/etc...

51. Demo: ansible-role-sshd
●
https://github.com/alvistack/ansi
ble-role-sshd
– Ansible Role for SSHD installation
– Molecule + LXD + Travis CI
– Support Ubuntu 18.04/16.04 +
CentOS 7 + openSUSE 15.0

54. Docker Build with Ansible (!?)
●
Why NOT Dockerfile?
●
Why NOT ansible-bender?
●
How Dockerfile with Ansible?
●
Demo: docker-jira

55. Why NOT Dockerfile?
●
Back to the origin: why still
custom shell scripting?
– Difficult for everything: learn, write,
error detection, debug, etc…

56. Why NOT ansible-bender?
●
https://github.com/ansible-comm
unity/ansible-bender
– Build Docker Image with standard
Ansible Playbook
– Just need basic Python support
inside target container

57. Why NOT ansible-bender?
(cont.)
●
PROS
– Could integrate with Travis CI
– Push image to DockerHub once build successful
●
CONS
– Can’t trigger auto build from DockerHub if
upstream base image get update (i.e. not
Dockerfile-based)

60. How Dockerfile with Ansible?
●
Kickstart with Dockerfile
●
Install Python, PIP then Ansible
inside Docker image
●
Run Molecule via localhost inside
Docker image (i.e. Delegated)

61. How Dockerfile with Ansible?
●
PROS
– All Roles could be pre-tested
– A simple Playbook via localhost
– Single layer for Docker Image
– Support auto build via DockerHub
●
CONS
– Size bigger than purely Dockerfile (i.e. Python and
Ansible)

63. Demo: docker-jira
●
https://github.com/alvistack/docker-j
ira
– Docker Image packaging for Atlassian
JIRA
– Molecule + Delegated
– All used Roles are LXD tested
– Support DockerHub auto build

66. Molecule + Vagrant + Ceph +
Kubernetes
●
Roles Pre-tested with Molecule
●
Links Roles with GIT Submodule
●
Test Complex Cluster with Vagrant
●
Demo: alvistack-ansible

67. Roles Pre-tested with Molecule
●
Similar as OOP, writing each
Ansible Roles with test cases
●
Simpler for R&D, debug,
maintenance, etc...

68. Links Roles with GIT
Submodule
●
Easy update
●
Easy management

69. Test Complex Cluster with
Vagrant
●
In case of Ceph OSD, truth block device is
required
– Not support file-based loop device
●
In case of Weave, each Kubernetes node
must have unique machine ID
– With LXD all instance get the same host
machine ID

70. Test Complex Cluster with
Vagrant (cont.)
●
Molecule + Docker
– Default, VERY FAST
– NO, (simple) systemd support
– NO, (simple) block device support
– YES, Travis CI support

71. Test Complex Cluster with
Vagrant (cont.)
●
Molecule + LXD
– Fast
– YES, systemd support
– NO, (simple) block device support
– YES, Travis CI support

72. Test Complex Cluster with
Vagrant (cont.)
●
Molecule + Vagrant
– VERY SLOW (i.e. booting VM)
– YES, systemd support
– YES, block device support
– NO, local test only (i.e. VT-x/AMD-v
required)

73. Test Complex Cluster with
Vagrant (cont.)
●
Molecule + LXD for most Ansible
Roles R&D
●
Molecule + Vagrant for final stage
local integration test

74. Demo: alvistack-ansible
●
Fetch source
– git clone https://github.com/alvistack/
alvistack-ansible.git
– cd alvistack-ansible && git submodule
update --init –recursive

75. Demo: alvistack-ansible (cont.)
●
Setup inventory
– cp -rfp inventory/sample inventory/all
– vi inventory/all/hosts

76. Demo: alvistack-ansible (cont.)
●
Run the playbook
– ansible-playbook -i inventory/all/hosts
playbooks/setup-everything.yml

77. Demo: alvistack-ansible (cont.)
●
Manually create Ceph OSD and CephFS, e.g.
– ceph-volume lvm create --blue-store --data /dev/sdc
– ceph osd pool create cephfs_metadata 32 32
– ceph osd pool create cephfs_data 128 128
– ceph fs new cephfs cephfs_metadata cephfs_data

81. Tips & Tricks
●
Always Start with Test Cases
●
Simple Tests Goes Docker
●
Complex Tests Goes LXD
●
Cluster Tests Goes Vagrant
●
Roles Only Handle Single Group
●
Playbook Handle Cross Group
●
Cross Target File Copy with `base64`

82. Roadmap
●
Automatic upgrade plan for each Ansible Role, e.g.
– Save last run schema version (e.g. 1001) to remote with
Local facts (facts.d)
– Fetch and compare with current schema version from
code (e.g. 1004)
– Run the in between upgrade procedures (e.g. 1002.yml,
1003.yml, 1004.yml)
– Update the saved schema version

83. Q&A

84. Contact Us
●
Address: Unit 326, 3/F, Building 16W, No.16
Science Park West Avenue, Hong Kong Science
Park, Shatin, N.T.
●
Phone: +852 3576 3812
●
Fax: +852 3753 3663
●
Email: sales@pantarei-design.com
●
Web: http://pantarei-design.com