I gave this talk to the Continuos Delivery Conference 2016 in Bossum. It is showing how HolidayCheck migrated its infrastructure to a microservices and how this enabled the Continuos Delivery.

1. Continuous Delivery in a microservices
environment
A success story
Luca Mazzaferro
luca.mazzaferro@holidaycheck.com

2. Outline:
• About me
• About HolidayCheck
• CD and micro-services at
HolidayCheck

3. ▪ I was born in Rome where I spent most of my
life until 2013 when I moved to Munich.
▪ Former physicist, member of ATLAS
experiment @CERN during the roaring master
degree and Ph.D. years. Mainly focusing in data
analysis and grid computing.
▪ In June 2015, I sold my soul to the business,
leaving the public research for happily joining
HolidayCheck as devops engineer
About Me

4. About HolidayCheck

5. 1
Holidaycheck means…
Holidays and travel bookings

6. 1
Trustable reviews
Holidaycheck means…
Holidays and travel bookings
2

7. 1
Trustable reviews
Holidaycheck means…
Finding the best price
2
3
Holidays and travel bookings

8. 1
Trustable reviews
Holidaycheck means…
… and much more.
2
3
4
Finding the best price
Holidays and travel bookings

9. Hotels:
659.000
Dests:
195 Countries
52880 Towns
Reviews:
6.900.000
Photos:
7.470.000
Visit/Month:
13.66 Million
Tour operators:
100
About HolidayCheck
Some numbers
HolidayCheck is the largest review, booking and price
comparing portal for travel and holidays in German language.

10. CD and micro-services at HolidayCheck
A bit of History

11. Pre HC microservices era:
Not so much time ago…
The Deployment Day

12. Pre HC micro-services era:
Not so much time ago…
ABC
1 Day1 Week2 Weeks
C
B A
ABC
Development
Collect
&
Merge
Test & Deploy

13. Pre HC micro-services era:
Not so much time ago…
Results:
● Low deployment rate
● Frequent full roll backs
● Many inter teams
dependencies
● Hard team planning
● High stress

14. Pre HC micro-services era:
Not so much time ago…
The Elephant Release

15. Pre HC micro-services era:
Not so much time ago…
Dev
Test
Merge
Deploy
Fback

16. Pre HC micro-services era:
Not so much time ago…
Results:
● Low deployments rate
● Teams queued
● Inter teams dependencies
● Still monolithic application

17. Pre HC micro-services era:
Not so much time ago…
Continuous Delivery is the ability to get changes of all
types—including new features, configuration changes, bug
fixes and experiments—into production, or into the hands
of users, safely and quickly in a sustainable way.*
A bit better but still far away from here:
* continuousdelivery.com

18. CD and micro-services at HolidayCheck
Our solution

19. The migration to micro-services
The ingredients
● Migration plan
● New teams structure and responsibilities
● New technology stack

20. 1st Ingredient:
Migration plan

21. The 1st ingredient
1. Slow migration: from mobile to desktop
2. One section after another: less traffic to higher
3. Getting rid of old legacy software (when possible)
Migration Plan:

22. 2nd Ingredient:
New teams structure &
responsibilities

23. • Preferred small cross functional teams (5-6 people)
• Devops Culture adopted via in-team devops engineer
• Teams responsible for their own products (microservices)
• Inter-teams dependencies removed and autonomy enforced
2nd ingredient
PO/TO SM Devs Devops
Eng.
Ops Team
Teams structure & reposibilities
Organiza(ons which design systems ... are constrained to produce designs which are
copies of the communica(on structures of these organiza(ons [M. Conway]
*

24. 3rd Ingredient:
New technology stack
- The Dust project
- The Alerts and
Monitoring
- CD at HC

25. The new technology stack
The Dust project
Creation of Dust Team (Ops + Dev):
• Setup the new infrastructure
focused to run micro-services
• Develop specific tools for micro-
services deployments
• Support teams for services
“containerization”
Dust
Environment

26. The new technology stack
The Dust project
Dust
Environment
Main advantages:
• Easily scalable
• Ready for Continuous
Deployments
• Services easy to monitor
• Isolated environment
• Enforce team autonomy

27. 3rd Ingredient:
New technology stack
- The Dust project
- The Alerts and
Monitoring
- CD at HC

28. The new technology stack
• Metric basedAlerts
• Logs
• System metrics
Default
Monitoring
• Teams defined
dashboards
Custom
Monitoring
New approach to monitoring and alerting

29. • Required for each service
• Based on the user experience
(mainly error rate and response
time)
• Customizable by the teams
• Automatically deployed
Alerts
You can’t survive without

30. Alerts
A simple overview
A
B C
DUST
NGINX metrics
Customizable Metrics
Alerts defined on user
experience:
- Error rate (5XX, 4XX)
- Response Time
On Call
Duty
Teams
Traffic

31. • Automatically deployed
• Based on logs and metrics
• Useful for
debugging
criticalities discovery
performances issue discovery
• Accessible by devs and ops.
Default Monitoring
Makes your life easier

32. • One centralize system for
all the logs
• Search and visualize
• No configuration needed
• Easy to integrate in GCE
Default Monitoring
Logs
Servers
Microservices
GCE

33. • Grafana shows all the
metrics collected by
Prometheus
• Custom metrics
• Custom dashboards
Default Monitoring
Custom Metrics
Servers
Microservices
GCE

34. Default Monitoring
Custom Metrics
NGINX Grafana dashboard

35. • External service mainly used
by micro-services
• Error analysis
• Services maps
• Metrics monitoring
Default Monitoring
NewRelic
Servers
Microservices
GCE

36. Default Monitoring
NewRelic Services Maps
Services Maps

37. • Team defined metrics
and dashboards
• Freedom to chose tools
and technologies
• Mainly for service quality
and performances
monitoring
• Data & Business analysis
Custom Monitoring
Can drive business decisions

38. 3rd Ingredient:
New technology stack
- The Dust Project
- The Alerts and
Monitoring
- CD at HC

39. 1 Develop, commit and push
Continuous Delivery @HolidayCheck
Our Vision
Staging
Live
Team
Server
1
2
3
4
Build
Basic tests (e.g. Unit, Reg.)
Deploy on staging env
2
3
Sophisticated Tests (integr.,
accuracy)
4
Deploy in production
Smoke tests
* Continuous Monitoring

40. MSs/Containers
100
Teams
12
Deployment Rate
Daily/Weekly
Main languages:
Scala/Go/
ReactJS
Main Backend Techs
ElasticSearch/
MongoDB/Kafka
HC Infrastructure in Numbers
Currently we are at the end of a migration phase in Google Cloud Engine.

41. Conclusions
The adoption of microservices
enforced team autonomy, enabled
the Continuous Delivery and
increased the deployment rate.
The entire infrastructure is more
stable, reliable, maintainable
and scalable
A smooth migration in Google
has been helped only by the
previous micro-services
adoption.

42. Conclusions
The adoption of microservices
enforced team autonomy, enabled
the Continuous Delivery and
increased the deployment rate.
The entire infrastructure is more
stable, reliable, maintainable
and scalable
A smooth migration in Google
has been helped only by the
previous micro-services
adoption.

43. www.holidaycheck.de
HolidayCheck Solutions
Neumarkter Str. 61 | D- 81673 München
Luca Mazzaferro
DevOps Engineer
T: Tel +41 (0) 71 686 9000
luca.mazzaferro@holidaycheck.com
Contact.

44. THANK YOU
FOR YOUR ATTENTION!
Questions?