We sell travel services to more than 10 million customers worldwide in 15 languages across 35 countries, through hundreds of micro-services. What happens if you challenge the way you deliver your products with a new infrastructure based on Kubernetes? You’ll have to face classical developing, deploying and monitoring paradigms and bring everything to an entirely different level. It will also have to kept in mind that the business expects continuous delivery of new features without impact on the final customers experience. In this presentation we will explore our one-year-long journey to move a full business flow from a well-known legacy platform to Kubernetes through a seamless migration. We’ll dive into the full tech stack, from the services we give to our development teams to the way we provision servers, without forgetting the secret sauce we applied to make it happen. We will not hide the failures, the problems and the wrong assumptions we made along the way, but we will celebrate lessons learnt and the goal we achieved; allowing us to boost time-to-market and reliability of our systems.

1. Tales from Lastminute.com machine room:
our journey towards a full on-premise
kubernetes architecture in production
michele.orsi@lastminute.com
manuel.ranieri@lastminute.com
KubeCon - Berlin, 29-30 March 2017

2. An inspiring travel company ..

3. A tech company to the core
Tech department: 300+ people
Applications: ~100
Database: 4 TB of data
Servers: 1400 VMs, 300 physical machines
Locations: Chiasso, Milan, Madrid, London, Bengaluru

4. Business: "technology is slow"
https://www.pexels.com/photo/turtle-walking-on-sand-132936/

5. Technology: "the monolith is the problem"
https://www.flickr.com/photos/southtopia/5702790189

6. https://www.pexels.com/photo/gray-pebbles-with-green-grass-51168/
"... let’s break into microservices!"

7. A lot of issues
● LONG provisioning time
● LACK OF alignment across environments
● LACK OF alignment across applications
● LACK OF awareness about ops

8. A year-long endeavour
● build a new, modern infrastructure
● migrate the search (flight/hotel) product there
... without:
● impacting the business
● throwing away our whole datacenter

9. https://www.pexels.com/photo/colorful-toothed-wheels-171198/
Our infrastructure and our architecture

10. Virtualization platform
TONS
OF
VIRTUAL MACHINES

11. Virtualization platform
TONS
OF
VIRTUAL MACHINES

12. ● CoreOS, the all-in-one choice
○ Cloudconfig configuration
○ Automatable in a shot
○ Really simple patch management
Engage

13. Our Kubernetes on CoreOS architecture is born
● The stack
○ ETCD
○ FLANNELD
○ DOCKER
● KUBERNETES (Google!)
K8S
DOCKER
FLANNELD
ETCD
CoreOS
Pod
Pod
Pod
Server

14. NODE 2
NODE 1
NODE 2
NODE 1
How to talk with pods
NGINX
NGINX
Pod
Pod
Pod
Pod
Pod
np
np
np
Pod
Pod
Pod
Proxy
np
np
Pod
Pod
Pod
np
Proxy
Proxy
Proxy
F5 F5
tcp http
NodePort Ingress

15. In the name of service
- host: awesomeservice.prd.mykubecluster.intra
http:
paths:
- path: /
backend:
serviceName: awesomeservice
servicePort: 8081
awesomeservice-ingress.yaml

16. In the name of service
*.[prd|qa|dev].mykubecluster.intra. IN CNAME kubef5ingress

17. The return of NodePort
np
np
Pod
Pod
Pod
np Proxy
NODE n
F5 TLS TLS
TLS
tcp
TLS

18. The registry brought another question...
?

19. Seriously?

20. Rear window on kubernetes
Server
graphite
OS
collectd
image
Nagios first
Grafana 4 now
icons from http://www.flaticon.com
Kube API

21. We were happy!

22. Not happy anymore

23. Seriously?

24. The change… It’s a kind of magic
KEEP
CALM
and
TRUST KUBERNETES

25. What we learned
Lots of things!

26. The final architecture (so far…)
K8S
DOCKER
FLANNELD
ETCD
Ubuntu
Pod
Pod
Pod
F5
OUTSIDE
KUBERNETES
INSIDE KUBERNETES:
3 different environments
7 MASTERS
2 REGISTRYs
+ 70 PHYSICAL NODES
+ 47 ETCDs
+ 7 DNS
+ 140 Namespaces
+ 1300 PODs
ingress

27. Our infrastructure and our architecture
https://www.pexels.com/photo/colorful-toothed-wheels-171198/

28. Our core axioms
● same architecture across environments
● a common framework to align software
● centralized monitoring/logging, with alerts
● zero downtime deployment
● automation everywhere

29. APP3-PRODUCTION
Kubernetes: our architecture
APP2-PRODUCTION
APP1-PRODUCTION
APP3-PRODUCTION
APP2-PRODUCTION
APP1-PREVIEW
APP3-PRODUCTION
APP2-PRODUCTION
APP1-DEVELOPMENT
APP3-PRODUCTION
APP2-PRODUCTION
APP1-QA
nonproductionproduction

30. Kubernetes: our architecture and choices
APP1-PRODUCTION
deployment
replica-set
app1-production.prd.mykubecluster.intra
secret configmap
POD-3POD-2POD-1
production

31. "To ingress or not to ingress? .."
NODE 1
NODE 2
NODE 3
● easier DNS management
● customizable proxy server
● 3rd party tool
● requires external sync
● all requests go through it
● reload risks
F5
NGINX
NGINX

32. APP1-PRODUCTION
Kubernetes: our architecture and choices
POD
production
applicationfluentd
collectd
carbon

33. APP1-PRODUCTION
POD
Monitoring and alerting: grafana/graphite
cluster
graphite
application
collectd
Grafana 4
icons from http://www.flaticon.com
carbon

34. Zero downtime (1): graceful shutdown
lifecycle:
preStop:
exec:
command: ["/stop_helper.sh"]
deployment.yaml
#!/bin/bash
wget http://localhost:8002/stop
stop_helper.sh

35. Zero downtime (2): graceful startup
private CompletableFuture run(Stream<CompletableFuture> startupJobs)
{
return allOf(startupJobs.toArray(CompletableFuture[]::new))
.thenAccept(this::raiseReadinessUp)
.exceptionally(this::shutdown);
}
JobsExecutor.java

36. Automate everything: pipeline DSL
microservice = factory.newDeployRequest()
.withArtifact("com.lastminute.application1",2)
.fromGitRepo("git.lastminute.com/team/application")
lmn_deployCanaryStrategy(microservice,"qa")
lmn_deployCanaryStrategy(microservice,"preview")
lmn_deployCanaryStrategy(microservice,"production")
pipeline

37. Automate everything: pipeline
pull
jar
build
docker
(gate)
QA
canary
(gate)
QA
stable
(gate)
PREV
canary
(gate)
PREV
stable
(gate)
PROD
canary
(gate)
PROD
stable
● git push
○ continuous integration
○ continuous delivery

38. https://www.flickr.com/photos/ghost_of_kuji/2763674926
.. failure ..

39. nginx ingress controller problem
10.0.0.1
10.0.0.2
10.0.0.3
10.0.0.4
NGINX
NGINX
NGINX
10.0.0.5
10.0.0.6
F5
NGINX
NGINX
NGINX
NGINX
NGINX

40. https://www.pexels.com/photo/grayscale-photography-of-person-at-the-end-of-tunnel-211816/
There’s light .. at the end

41. ● 20K req/sec in the new cluster
● 2M metrics/minute flows
● 10 minutes to create a new environment
● whole pipeline runs in 16 minutes
○ 4 minutes to release 100 instances of a new version
Give me the numbers .. again!

42. Yes, we’re hiring!
THANKS
careers.lastminutegroup.com