Trillian is an Ansible-based tool that automates the deployment of CloudStack environments for testing purposes. It supports building environments with various hypervisor and CloudStack versions across multiple concurrent tests. Jenkins integrates with Trillian to run test suites after package builds and clean up environments. The tools provide automated testing capabilities for CloudStack improvements and integrations.

1. The Cloud Specialists
THE HITCH-HIKERS GUIDE
TO CI/CD
ShapeBlue.com • @ShapeBlue
Paul Angus, VP Technology • @CloudyAngus

2. The Cloud Specialists ShapeBlue.com @ShapeBlue
“ShapeBlue are expert builders of public
& private clouds. They are the leading
independent global CloudStack services
company”
A b o u t S h a p e B l u e

3. C l i c k t o e d i t
The Cloud Specialists ShapeBlue.com @ShapeBlue
Paul Angus – VP Technology
• Global authority on CloudStack & cloud infrastructure design.
• 10+ years C-Level experience.
• Apache CloudStack project committer
• Specialising in deployment of CloudStack
and surrounding infrastructure
• USP, Georgian Ministry of Justice, Orange, TomTom,
PaddyPower, Ascenty, BSkyB
A b o u t M e

4. C l i c k t o e d i t
The Cloud Specialists ShapeBlue.com @ShapeBlue
S h a p e B l u e c u s t o m e r s

5. C l i c k t o e d i t
The Cloud Specialists ShapeBlue.com @ShapeBlue
S h a p e B l u e c u s t o m e r s

6. C l i c k t o e d i t
The Cloud Specialists ShapeBlue.com @ShapeBlue
S h a p e B l u e c u s t o m e r s

7. The Cloud Specialists ShapeBlue.com @ShapeBlue
Tr i l l i a n
Why?

8. C l i c k t o e d i t
The Cloud Specialists ShapeBlue.com @ShapeBlue
W h y
“Building CloudStack
environments using CloudStack
and Ansible? Aren’t you just
having fun?”
CEO

9. C l i c k t o e d i t
The Cloud Specialists ShapeBlue.com @ShapeBlue
W h y
“Building CloudStack environments
using CloudStack?
Are you just having fun?”
Talented Cloud Architect
“If we can automate the
building of environments
including the networking, and
generate any topology we want
quickly, we can achieve the tasks
that I’ve highlighted in these
slides that I have with me, just
in case you asked.”

10. C l i c k t o e d i t
The Cloud Specialists ShapeBlue.com @ShapeBlue
W h y
building of environments
including the networking, and
generate any topology we want
quickly, we can achieve the tasks
that I’ve highlighted in these
slides that I have with me, just
in case you asked.”
CEO
“That would be excellent.
Go ahead.
Oh, and here’s a pay rise”

11. C l i c k t o e d i t
The Cloud Specialists ShapeBlue.com @ShapeBlue
We need to test:
• New features
• Hotfixes/patches
• Full Versions
• Physical integrations
• New hypervisors
• Upgrades
• Against a broad range of deployment architectures
• Against a broad range of hypervisor types and
versions
W h y

12. C l i c k t o e d i t
The Cloud Specialists ShapeBlue.com @ShapeBlue
Many, many, MANY permutations of a
CloudStack deployment….
• Basic / Advanced
• Local / shared / mixed storage
• More than 8 common hypervisor types/versions
• 4 or 5 Management server OS possiblilities
• That’s 144 combinations only looking the
basics.
• Previous attempts at test automation haven’t
been able to get coverage or capacity.
W h y

13. C l i c k t o e d i t
The Cloud Specialists ShapeBlue.com @ShapeBlue
Discovery projects:
• New Features
• CloudStack integrations (ie Ceph)
• CloudStack ecosystem components (ie ELK
stack)
W h y

14. C l i c k t o e d i t
The Cloud Specialists ShapeBlue.com @ShapeBlue
Lab v1.0
• Templates to build nested hypervisor hosts
• Cloudmonkey scripts to configure zone
• Manual creation of storage mounts
Simple but pretty effective
H u m b l e B e g i n n i n g s

15. C l i c k t o e d i t
The Cloud Specialists ShapeBlue.com @ShapeBlue
Lab v1.0
• A lot of small edits scripts to fit use cases.
• Human error still a big factor
• Can’t ‘set and forget’
• A lot of individual knowledge required
• Loose definition of ‘testing’.
H u m b l e B e g i n n i n g s

16. The Cloud Specialists ShapeBlue.com @ShapeBlue
Trillian Tenets

17. C l i c k t o e d i t
The Cloud Specialists ShapeBlue.com @ShapeBlue
Fully automated deployment of CloudStack
environments from any CloudStack package repo
Tr i l l i a n Te n e n t s

18. C l i c k t o e d i t
The Cloud Specialists ShapeBlue.com @ShapeBlue
• Ability to support multiple concurrent environments
• No hardcoding to a specific (our) lab/environment
• As few separate components as possible
• Separation of ‘code’ and ‘configuration’
• Scripted command-line use fully supported
• Ability to support as many hypervisors as possible
• As flexible as possible
• Best ‘performance’ possible
• Best utilization of hardware resources possible
Tr i l l i a n Te n a n t s

19. The Cloud Specialists ShapeBlue.com @ShapeBlue
Meet The Family

20. C l i c k t o e d i t
The Cloud Specialists ShapeBlue.com @ShapeBlue
T h e C I F a m i l y
Trillian
Marvin
Winston
Jenkins
Blue
Orangutan

21. C l i c k t o e d i t
The Cloud Specialists ShapeBlue.com @ShapeBlue
Trillian
Ansible based environment builder
• Multi-tenant (and multiple concurrent environments)
• CloudStack builds the Virtual Infrastructure
• Hypervisor hosts are CloudStack templates
• Supports multiple and mixed hypervisors
• Supports Advanced and Basic networks
• Open source
T h e C I F a m i l y

22. C l i c k t o e d i t
The Cloud Specialists ShapeBlue.com @ShapeBlue
Trillian
Installs & Configures:
• CloudStack
• MySQL
• XenServer (configures)
• vCenter (configures)
• ESXi (configures)
• KVM
• Marvin hosts and Marvin test suites
T h e C I F a m i l y

23. C l i c k t o e d i t
The Cloud Specialists ShapeBlue.com @ShapeBlue
Marvin
Python based test framework based on ‘nose’ test platform.
Configuration files define environment to be tested.
• Integral part of the CloudStack codebase
• Recently given its own install package
• Smoke and Component tests
• 53 smoke test suites
• 154 component test suites
• Additional tests for plugins such as Nuage and SolidFire
T h e C I F a m i l y

24. C l i c k t o e d i t
The Cloud Specialists ShapeBlue.com @ShapeBlue
Jenkins
CI Platform
• Runs build jobs to produce RPMs from source code
• Presents an integration point to wrap up the creation of
environments using Trillian and the running of tests using
Marvin.
• Gives a UI to Trillian.
T h e C I F a m i l y

25. C l i c k t o e d i t
The Cloud Specialists ShapeBlue.com @ShapeBlue
Blue Orangutan
Build-bot
• Automation endpoint to send build jobs to Jenkins for CentOS
6/7 and Debian packages from CloudStack source
• Provides an integrated way to create packages from pull
requests in Github.
• Can kick-off Jenkins jobs to build and test environments based
on packages which it builds.
T h e C I F a m i l y

26. C l i c k t o e d i t
The Cloud Specialists ShapeBlue.com @ShapeBlue
Winston
Test Orchestrator (Future project)
• Receives ‘offers’ of test type/capacity from remote test labs
• Takes request from Blue Orangutan to test packages and
distributes tests to suitable remote labs.
Offers might include:
• Use of local storage
• Use of basic vs advanced networks
• Use of particular hypervisors/versions
T h e C I F a m i l y

27. The Cloud Specialists ShapeBlue.com @ShapeBlue
How its used

28. C l i c k t o e d i t
The Cloud Specialists ShapeBlue.com @ShapeBlue
P r o c e s s u s e d i n A p a c h e C l o u d S t a c k P r o j e c t

29. C l i c k t o e d i t
The Cloud Specialists ShapeBlue.com @ShapeBlue
• Test Pull Requests to add XenServer 7 support
• XenServer won’t run on ESXi 5.5 
• Create XenServer 7 VMs on stand-alone ESXi 6.0 host
and ‘point’ Trillian at them. 
• vSphere 6.5 – Create vCenter and ESXi templates and
run tests. 
X e n S e r v e r 7 & v S p h e r e 6 . 5

30. C l i c k t o e d i t
The Cloud Specialists ShapeBlue.com @ShapeBlue
• Build base CloudStack environment quickly
• Leverage CloudStack to create other infrastructure
components just as quickly.
ie
• HostBill server
• Cloudian manager server and storage nodes
• Leverage CloudStack networking to give external access
to environments – Static NAT, port forwarding or VPN.
C l o u d i a n & H o s t B i l l

31. C l i c k t o e d i t
The Cloud Specialists ShapeBlue.com @ShapeBlue
• Build required configuration with required components
• Specific versions of CloudStack
• Specific hypervisor types
• Specific hypervisor versions
• Manual creation of additional components still allowed
• ie second zone with different hypervisor type or version
S o l i d F i r e P l u g i n

32. C l i c k t o e d i t
The Cloud Specialists ShapeBlue.com @ShapeBlue
We can do all of these things
AT THE SAME TIME
C o u p D e G r â c e

33. The Cloud Specialists ShapeBlue.com @ShapeBlue
Trillian

34. C l i c k t o e d i t
The Cloud Specialists ShapeBlue.com @ShapeBlue
https://github.com/shapeblue/Trillian
Tr i l l i a n
• Single zone
• Use pre-set or custom Repos
• Local or shared storage
• ESXi, KVM (C6,C7,U14.04),
XenServer hosts
• Customisable number of primary
and secondary storage pools
• Multiple Management servers
• Selectable Management server
OS (C6.x, C7, U14.04)
• Multiple hypervisors
• dvSwitch (public/guest traffic)
• Configurable Java versions
• Option to use pre-built hosts
• Multiple MySQL configurations

35. C l i c k t o e d i t
The Cloud Specialists ShapeBlue.com @ShapeBlue
A n s i b l e i n Tr i l l i a n
• All driven from Ansible
• A global group_vars file stores all
local configuration data such as:
• Passwords
• All defaults ie basic vs
advanced or network naming
• EVERYTHING can be customised at
runtime.
(except that)
(and that)

36. C l i c k t o e d i t
The Cloud Specialists ShapeBlue.com @ShapeBlue
ansible-playbook generate-
cloudconfig.yml -i localhost --extra-vars
"env_name='localstoragetest‘
env_version=cs49
mgmt=1
db=0
hvtype=x
xs_ver=xs65sp1
hv=2
env_accounts=all
pri=2
sec=1
build_marvin=yes
mgmt_os=6
use_local_storage=true
use_shared_storage_systemvms=true
use_shared_storage=false"
ansible-playbook deployvms.yml -i
./hosts_localstoragetest
A n s i b l e i n Tr i l l i a n

37. C l i c k t o e d i t
The Cloud Specialists ShapeBlue.com @ShapeBlue
Tr i l l i a n i n J e n k i n s
• Different Jenkins’ jobs with specific
‘assumptions’.
• Pull Requests receive URL from
Blue Orangutan and automatically
Run test suites, then clean up.
• Reference_Trillian is a reference
implementation of Jenkins and has
more corner-case options exposed.

38. C l i c k t o e d i t
The Cloud Specialists ShapeBlue.com @ShapeBlue
Tr i l l i a n i n J e n k i n s
• Allows user to force a version of
Java
• Allows use of ‘pre-built’
hypervisors and hypervisors which
cannot themselves be virtualised.

39. C l i c k t o e d i t
The Cloud Specialists ShapeBlue.com @ShapeBlue
Tr i l l i a n i n J e n k i n s
• If environment requires the
hypervisor to use local storage,
then CloudStack adds a data
volume to hypervisor and
configures hypervisor to use it for
local storage. This keeps template
size down
• Mixed local and shared storage is
supported

40. C l i c k t o e d i t
The Cloud Specialists ShapeBlue.com @ShapeBlue
Internal
ShapeBlue
summary
page UI
plugin.

41. C l i c k t o e d i t
The Cloud Specialists ShapeBlue.com @ShapeBlue
Tr i l l i a n & J e n k i n s i n C l o u d S t a c k
• UI gives overall status/health of
‘Lab’ and environments
• UI links to a job’s Jenkins output as
well as UI of built environment and
SSH session onto specific VMs

42. C l i c k t o e d i t
The Cloud Specialists ShapeBlue.com @ShapeBlue
I n S u m m a r y

43. The Cloud Specialists
THE HITCH-HIKERS GUIDE
TO CI/CD
ShapeBlue.com • @ShapeBlue
Paul Angus, VP Technology • @CloudyAngus