This document discusses the infrastructure and automation tools used by mobile.de and eBay Kleinanzeigen to provision and deploy their integration environments. It describes the Autoconf system used to automatically provision virtual machines from a database, the Genesis "master Autoconf instance", and the Autodeploy continuous deployment system. Autodeploy uses a decentralized architecture with agents on hosts and has deployed over 2000 applications changes per day. Virtualization has increased over time and now the test environments mainly use virtual machines. The future may involve moving more to microservices and public cloud platforms.

3. Integration Systems @ mobile.de
Karen Almog
Sr. Web Operations Engineer, Mobile International GmbH
kalmog@team.mobile.de
!

4. 4
Our Platform "
mobile.de
Over 2.7M ads
Approx. 50k ads per
day
eBay
Kleinanzeigen
Nearly 12M ads
Over 200k ads per day

5. +50
Developers/QA
11
Teams
1826
VMs
Our Platform

6. 6
What Does PD Want? "
# $
%&
'
Requirements
REPRODUCTION CONSISTENT TEST
DATA
SIMPLE SETUP ROLLBACK

7. A Standard Integra setup @ mobile
7
What Does PD Need? (
!O/S Installation Debian/Solaris
"
#
Server Configuration
Tomcats, Databases,
Search Indexers, DNS,
Proxy Servers and more
Data Initialization Initial Test Data Sets
Application DeploymentWAR/JAR
Approx. 60 Virtual
machines per
environment Ñ VM Install
$

8. 8
What Does SiteOps Want? "
[ ©

9. 9
The Solution: Autoconf "
*
%
'
A Server Provisioning system and CMDB
Automated HW and VM server provisioning
Servers inventoried as a resource in a database%
'
Reduces provisioning time from hours to minutes)

10. 10
Inventory Database Overview "

11. 11
Inventory Database Overview "
insert into server set manufacturer='VMware, Inc.', productname='VMware Virtual Platform';
select last_insert_id() into @var_server_id;
insert into serverdetails set server_id=@var_server_id,detailkey='datacenter_number', detailvalue='44';
insert into serverdetails set server_id=@var_server_id,detailkey='virtual', detailvalue='vmware';
insert into serverdetails set server_id=@var_server_id,detailkey='vmware_createopts_desire_hdd', detailvalue='12';
insert into serverdetails set server_id=@var_server_id,detailkey='vmware_createopts_desire_mem', detailvalue='3072';
insert into serverdetails set server_id=@var_server_id,detailkey='vmware_createopts_desire_cpu', detailvalue='1';
insert into assignment_queue set server_id=@var_server_id, serverclass_id=477, sequence_number=48, hostsuffix=2;
select * from view_assignment_queue_hostdetails where server_id=@var_server_id;

12. 12
There’s No Shame in Using A Mouse! "

13. 13
The Solution: Genesis "
*
%
'
A “Master” Autoconf Instance for Integras
Based on Autoconf
Includes a Web-­‐UI and API%
'
DB Resource for other Tools*
DNS, Deployment, Monitoring, Puppet
Joined Project between PD & SiteOps+

14. 14
Genesis: Web-­‐UI "

15. 15
Inventory Database: Integra Reference List "

16. 16
Genesis: API "

17. 17
Genesis: API "

18. root@genesis44-219:~# vm_creator.py -i graphite
Assignment Queue entry for "graphite":
{
"serverClassId": 352,
"sequenceNumber": 17,
"hostSuffix": "1",
"hostPrefix": "graphite",
"modificationTime": null,
"state": "newrequest",
"serverId": 243,
"id": 335
}
18
Genesis: API "

19. 19
Genesis: Data Initialization "
*
%
'
Set of BASH Scripts
Initializes Integra’s Database according to use case%
'
Sets up database replication & Initialization)
Can be launched from the Genesis GUI*

20. 20
Application Deployment: Autodeploy "
*
%
'
Continuous Deployment System
Web interface & API
Decentralized architecture%
'
Full Deployment in 20 minutes)
Instant feedback & Logging*
Collaborative effort between PD & SiteOps+

21. 21
Autodeploy Architecture "
eBay Inc. confidential
Architecture Overview
7
Autodeploy UI
AutoDeploy
Web App
AutoDeploy
DB (infradb)
Apache ZooKeeper
Server running
Autodeploy
Agent
Server running
Autodeploy
Agent
Server running
Autodeploy
Agent

22. 22
Autodeploy Architecture: Host Agent "
eBay Inc. confidential
AutoDeploy Agent - What happens during a deployment
8
Text
XML
DEPLOY
PLAN
Autodeploy
Agent
TARGET HOST
1. Autodeploy sends deploy instructions to autodeploy agent
running on target host.
2. Autodeploy agent executes local deployment script on
target host.
3. Deployment script executes deployment steps
(autonomous)
deployment
script
• Call LBAPI
• Call Nagios
• Download software
• Create properties
• Test app
• update releases table on
infradb
• exit with status
4. Script output and exit code sent back to autodeploy agent
Logging
back to
AutoDeploy
1.
2.
3.
4.
Script output
sent to
Agent
5. Logging and status reported back to Autodeploy by agent
AutoDeploy Web
Application
5.

23. 23
Autodeploy: GUI "

24. 24
Autodeploy: GUI "

25. 2014
25
Autodeploy: Stats ,
Production
Pre-­‐Production
(Integra)
Rollouts Per Day 8 21
Applica2ons Changes
per day (avg) 29 2017
Deployable Ar2facts (Total) 279 279
Deployable Hosts (Total) 555 1621

26. 26
Virtualization in Test Environment ,
0!
450!
900!
1350!
1800!
2006! 2007! 2009! 2010! 2011! 2013! 2014!
Physical! Virtual!

27. 27
Today ,
CURRENT STATUS
• We know it well, it sbll consumes bme and there is very
licle led to automate.
• IaaS is becoming ubiquitous, PaaS is our new challenge.
• Several IaaS clouds are already available at eBay.
• We already have a sort of IaaS (VMWare/Autoconf) of our own
and it works well enough for the short term.

28. 28
What Does The Future Hold? ,
,
Development teams and processes
are moving towards Micro-­‐Services

29. 29
What Does The Future Hold? ,

30. THANK YOU! :-­‐)-
Karen Almog
Sr. Web Operations Engineer, Mobile International GmbH
kalmog@team.mobile.de