I would like to present our CI Provisioning with Openstack solution and how it improved our development. The CI provisioning is meant to replace your static CI env servers to a auto provisioned during your build stage for every commit. Abstract: 1. CI Flow Quick view 2. Openstack CI integration maven plugin 3. Orchestration using facters 4. Openstack instance queue for faster provisioning

1. Gidi Samuels
CI - DevOps
CI Provisioning
with Openstack

2. WHO AM I?
CI-DevOps Tech Leader
10 years in the field of SW development and CI
5 years @ Liveperson building the CI/CD pipeline

3. WHO ARE WE

4. About our CI
1600
Releases
~160
Micro Services
10K
Weekly Builds
1400
Git Repos
70
BuildAgents
1000
TeamCity jobs
30
Dev Teams
4
CI Members

5. Agenda
The resource PROBLEM we are trying to solve
FASTER DEVELOPMENT with Openstack
Developing OPENSTACK MAVEN PLUGIN
2
3
4
Using pre-defined servers from POOL5
Openstack Numbers @ Liveperson1

6. 20,000 physical
cores
8000 VM’s in
7 DC’s
Upgrading to
Kilo in progress
40,000
virtual cores
Web&App tiers 100%
on the Openstack
Openstack @ LivePerson

7. LIVEPERSON DEVELOPMENT TOOLS

8. Make a
change
TeamCity CI
Server
Build & Unit Test
GitHub Packaging
Deploy
System
Test
Maven lifecycle
Security
Analysis
RELEASE &
APPROVAL
CI Flow

9. Make a
change
TeamCity CI
Server
Build & Unit Test
GitHub Packaging
Deploy
System
Test
Maven lifecycle
Security
Analysis
RELEASE &
APPROVAL

10. THE PROBLEM

11. We need ~600 dedicated VM’s for the CI flow
Long time in queue (no free resources)
Dirty and unmanaged VM nodes
Not like Production

12. The Goal:
Run build in parallel

13. The Solution:
On-the-fly Openstack provisioning

14. Extending Maven by developing
Maven ci-provisioning Plugin

15. Openstack Toolkits
http://www.openstack4j.com/
Identity, Compute, Image, Network, Storage
Open source Java client which allows provisioning and control
of an OpenStack system

16. 1.Install maven from https://maven.apache.org/
1.mvn archetype:generate
-DgroupId=ci.plugins
-DartifactId=openstack-maven-plugin
-DarchetypeGroupId=org.apache.maven.archetypes
-DarchetypeArtifactId=maven-archetype-plugin
Step 1

17. Add the dependency to your maven pom.xml file
<dependency>
<groupId>org.pacesys</groupId>
<artifactId>openstack4j</artifactId>
<version>2.0.0</version>
</dependency>
Step 2

18. // authenticate
OSClient authenticate(String tanent) {
OSClient os = OSFactory.builder()
.endpoint(OS_AUTH_URL)
.credentials(OS_USERNAME, OS_PASSWORD)
.tenantName(tanent)
.authenticate();
return os;
}
Step 3

19. Develop the Maven plugin following goals:
Compute:
creates instances on the fly
DeleteInstance:
delete instance if build is successful
otherwise leave for debugging a period of
time
Step 4

20. TOMCAT
APP
TOMCAT
APP

21. How to use it in your pom.xml
<roles>
<role1>ci-app1</role1>
<role2>ci-app2</role2>
<role3>ci-nginx</role3>
</roles>
Step 4
● 3 Openstack
instances will be
assigned
● Facter Role will be
created on each
instance

22. Now that we can auto provision instances
How do we configure them
as a service?

23. Facter is a cross-platform library for retrieving
simple operating system facts, like operating
system, linux distribution, or MAC address
PUPPET FACTER
Available during puppet execution

24. Custom facts located @ /etc/facter/facts.d/
PUPPET FACTER
{
"role" : "ci-app1"
}
balancermember: node1,node2
JSON style YAML style

25. role1 => ci-app1
role2 => ci-app2
role3 => ci-nginx
balancermember => ctor-ci-q10-1,ctor-ci-q11-2
GENERATE CUSTOM FACTS

26. <web>ci-nginx</web>
<webdns>ci-web-${build.number}</webdns>
Step 5
Create a unique DNS for the service so it
can be accessed from the system test

27. node /qtvr-ajb4(2|3).*/ inherits default {
class{nginx::web':
upstream_members => ['ctvr-wap004', 'ctvr-wap005'],
}
if ($role1 == ci-app1) {
class{nginx::web':
upstream_membersn => split($balancermember, ',')
}
}
FROM STATIC TO DYNAMIC DECLERATION

28. What are the main
ADVANTAGES?

29. ADVANTAGES
CI is running in parallel
A dedicated provisioned env for each commit

30. ADVANTAGES
Each instance is fresh
A new fresh instance created for each commit

31. ADVANTAGES
Production-like environment
Each instance reflects prod-like env

32. ADVANTAGES
No resources waste
No need to allocated redundant resources

33. ADVANTAGES
Easy to scale

34. PROBLEM:
CI-provisioning add additional 5 minutes
the the CI Process …
SOLUTION:
Develop additional Maven goal that created
Openstack VM’s pool in the background
NOW WHAT ...

35. Added to the Maven plugin following goals:
Qmanager: instance Q manager that
prepare ready instances
ComputeCIQ: use instances from pull
Step Improvement

36. THANK YOU

37. WHO ARE WE - CUSTOMERS
More than 18,000 customers