SlideShare a Scribd company logo
Take a BlaBlaCar to the stars with rkt !
Simon Lallemand
System Engineer
@slallema
New infrastructure
ecosystem
Agenda
Key principles
100% rkt powered carpooling
Directions for driving change
BlaBlaCar Facts, Figures & History
Founded
in 2006
21 million mobile
app downloaded
(iPhone + Android)
Facts and figures
12 million travelers
per quarter
1 million
tonnes less CO2 in the
past year
40 million
members
Currently in
22 countries
France, Spain, UK, Italy, Poland,
Hungary, Croatia, Serbia, Romania,
Germany, Belgium, India, Mexico,
The Netherlands, Luxembourg,
Portugal, Ukraine, Czech Republic,
Slovakia, Russia, Brazil and Turkey.
CO2
Our server growth
The evolution of the infrastructure from the begining to now:
2006 2008 2010 2012 2013 2014 2015 2016
Web
hosting
Dedicated
servers
Baremetal
1 rack
Baremetal
3 racks
Baremetal
5 racks
Baremetal
8 racks
Baremetal
14 racks
150 servers
2 DC
Baremetal
17 racks
300 servers
3 DC
~300 bare-metal servers
~400 container images
4000+ running containers
Tech evolution
The major tech changes in our infrastructure :
Virtu
2012
Chef
2013
Foreman
2014
Hardware
uniformization
2015
Containers
2015
2016
Key principles
Leading the industrialization at BlaBlaCar
Metal is invisible
Decouple hardware management and
what runs on it
Route everything
Keep the network simple and scalable
Remove snowflakes
Be service oriented
New Infrastructure Ecosystem
100% rkt powered carpooling
Containers
For everything !
CoreOS Container Linux
On 100% of our new servers
rkt
as container runtime
New Infrastructure Ecosystem
The tools
dgr
Container build
and runtime tool
Harmonize the way we build
Quick build
One way of doing things
Easy to understand for newcomers
As little code replication as possible
Templating at container start
A good integration with rkt
github.com/blablacar/dgr
dgr : build directory of an ACI
├── aci-manifest.yml
├── attributes
│ └── redis.yml
├── runlevels
│ └── build
│ └── install.sh
└── templates
└── etc
└── redis
└── redis.conf.tmpl
A standardized structure
Ease maintenance and teamwork
Inspired by config management
Separation of templates, attributes and scripts.
dgr : ACI manifest
name: redis:0.1
aci:
app:
exec:
- /usr/bin/redis-server
- /etc/redis/redis.conf
Simpler manifest format
Fill only what is important
YAML <3
One process per ACI
Composition is done only with POD
dgr : ACI manifest
name: redis:0.1
aci:
app:
exec:
- /usr/bin/redis-server
- /etc/redis/redis.conf
dependencies:
- debian:8.6
Use of dependencies
Composition
Lighter images
dgr : runlevel build
├── aci-manifest.yml
├── attributes
│ └── redis.yml
├── runlevels
│ └── build
│ └── install.sh
├── templates
│ └── etc
│ └── redis
│ └── redis.conf.tmpl
└── tests
Scripts executed in the container
Build from inside the container with all the
dependencies
dgr : runlevel build
#!/bin/bash
apt-get install -y redis-server
Install packages and stuff
For debian based containers you can install
packages using apt-get for instance.
dgr : runlevel builder
name: example.com/aci-redis-dicator:1
builder:
dependencies:
- example.org/aci-go
- example.org/aci-git
aci:
app:
exec:
- /bin/dictator
Build from outside of the container
Dedicated builder image with go & git
dgr : runlevel builder
#!/dgr/bin/busybox sh
git clone 
https://github.com/blablacar/redis-dictator
cd redis-dictator
go build
cp dist/dictator ${ROOTFS}/bin
Builder script
Clone, build and copy binary to
target ACI
Only /bin/dictator in final ACI
dgr : runlevel builder
name: example.org/gentoo-redis:1
builder:
dependencies:
- example.org/gentoo-stage4
aci:
dependencies:
- example.org/base
app:
exec: [ /usr/bin/redis-server ]
Other example
Builder with gentoo’s stage4
dgr : runlevel builder
#!/dgr/bin/busybox sh
emerge -v redis-server
Install packages from outside
Use emerge to install your app and its
dependencies.
Redis with dependencies only in the final
ACI
dgr : templates & attributes
├── aci-manifest.yml
├── attributes
│ └── redis.yml
├── runlevels
│ └── build
│ └── install.sh
└── templates
└── etc
└── redis
└── redis.conf.tmpl
Render configuration files
Templates stored in the aci
Default attributes stored in the aci
Overridable when used as dependencies
Overridable by environment var
dgr : templates & attributes
# templates/etc/redis.conf.tmpl
daemonize no
port {{ .redis.port }}
…
# attributes/redis.yml
default:
redis:
port: 6379
dgr : runlevel prestart
├── aci-manifest.yml
├── attributes
│ └── redis.yml
├── runlevels
│ └── prestart-late
│ └── 10-init-db.sh
└── templates
└── etc
└── redis
└── redis.conf.tmpl
Initialize container
prestart-early and prestart-late scripts
Before and after templating
Initialize environment before exec
dgr : testing
├── aci-manifest.yml
├── attributes
├── runlevels
├── templates
└── tests
└── wait.sh
└── my_cool_tests.bats
Testing
Bats as default tester
wait.sh
Wait for service to be ready
dgr : testing
#!/dgr/bin/bats -x
@test "Redis should be running" {
run bash -c "ps -aux | grep redis-server"
[ "$status" -eq 0 ]
echo ${lines[0]}
[[ "${lines[0]}" =~ "/usr/bin/redis-server" ]]
}
@test "Redis should listen on port: 6379" {
run bash -c "netstat -peanut | grep redis-server"
[ "$status" -eq 0 ]
[[ "${lines[0]}" =~ ":6379" ]]
[[ "${lines[0]}" =~ "redis-server" ]]
}
dgr : subcommands
~ # dgr init
~ # dgr try
~ # dgr build
~ # dgr test
~ # dgr install
~ # dgr push
github.com/blablacar/dgr
ggn
Manage services
in fleet clusters
Pods Services and Environments
Different services that use the same pods
Environments (1 per DC for prod + dev…)
Abstraction of fleet commands
github.com/blablacar/ggn
INSERT
LOGO
HERE
ggn
├── prod-dc1
│ └── attributes
│ └── services
│ └── redis-quota
│ │ └── attributes
│ │ └── service-manifest.yml
│ │ └── unit.tmpl
│ └── redis-rating
├── preprod
│
...
Single directory tree
Describes the services
The environments
Versioned in git
ggn : service manifest
containers:
- blablacar.com/aci-redis:3
- blablacar.com/aci-redis-exporter:1
nodes:
- hostname: redis-quota1
- hostname: redis-quota2
- hostname: redis-quota3
Define the service
ACI images and versions combined
to make a pod
Nodes composing the service
ggn : attributes
# attributes/redis.yml
override:
redis:
maxmemory: 4GB
TEMPLATER_OVERRIDE={“redis”:{“max
memory”: “4GB”}…}
Override attributes
Inject run context attributes as environment var
ggn : unit.tmpl
[Unit]
Description=Redis POD {{.
[Service]
KillMode=mixed
Restart=always
ExecStart=/usr/bin/rkt run 
--set-env=TEMPLATER_OVERRIDE='{{.jsonAttrs}}'
{{.acis}}
ggn : run a POD
$ ggn prod-dc1 redis-quota update
$ ggn prod-dc1 redis-quota1 journal -f
github.com/blablacar/ggn
Service
discovery
with nerve & synapse
AirBnB’s smartstack
Dynamic topology of services
Scalability requires service discovery
We started with smartstack of AirBnB
Since then we improved it and rewrote
it in go
github.com/blablacar/go-nerve
github.com/blablacar/go-synapse
/database/node1
go-nerve does health checks and
reports to zookeeper in service
keys
HAProxy
node1
Applications hit their local
haproxy to access backends
Service Discovery
/database
go-synapse watches zookeeper
service keys and reloads haproxy
if changes are detected
go-nerve
Zookeeper
go-synapse
bare-metal servers
1 type of hardware
3 disk profiles
fleet machines
CoreOS
fleet etcd“Distributed init system”
Hardware
Container Registry
ggn
dgr
Service Codebase
rkt PODs
build
run
store
host
create
Our infrastructure ecosystem
nerve
mysqld
monitoring
mysql-main_1
nerve
php
monitoring
nginx
synapse
front_1
synapse
nerve
zookeeper
Service Discovery
Work In Progress
( We’re hiring )
@slallema
@BlaBlaCarTech
Thanks!
@BlaBlaCarTech
BlaBlaTech.com
Paris container day   june17

More Related Content

What's hot

開放運算&GPU技術研究班
開放運算&GPU技術研究班開放運算&GPU技術研究班
開放運算&GPU技術研究班
Paul Chao
 
Kubernetes上で動作する機械学習モジュールの配信&管理基盤Rekcurd について
Kubernetes上で動作する機械学習モジュールの配信&管理基盤Rekcurd についてKubernetes上で動作する機械学習モジュールの配信&管理基盤Rekcurd について
Kubernetes上で動作する機械学習モジュールの配信&管理基盤Rekcurd について
LINE Corporation
 
From Docker to Production - ZendCon 2016
From Docker to Production - ZendCon 2016From Docker to Production - ZendCon 2016
From Docker to Production - ZendCon 2016
Chris Tankersley
 
Kubernetes Basis: Pods, Deployments, and Services
Kubernetes Basis: Pods, Deployments, and ServicesKubernetes Basis: Pods, Deployments, and Services
Kubernetes Basis: Pods, Deployments, and Services
Jian-Kai Wang
 
Kubernetes for Java developers
Kubernetes for Java developersKubernetes for Java developers
Kubernetes for Java developers
Robert Barr
 
From Docker to Production - SunshinePHP 2017
From Docker to Production - SunshinePHP 2017From Docker to Production - SunshinePHP 2017
From Docker to Production - SunshinePHP 2017
Chris Tankersley
 
Redis Meetup TLV - K8s Session 28/10/2018
Redis Meetup TLV - K8s Session 28/10/2018Redis Meetup TLV - K8s Session 28/10/2018
Redis Meetup TLV - K8s Session 28/10/2018
Danni Moiseyev
 
Dockerを利用したローカル環境から本番環境までの構築設計
Dockerを利用したローカル環境から本番環境までの構築設計Dockerを利用したローカル環境から本番環境までの構築設計
Dockerを利用したローカル環境から本番環境までの構築設計
Koichi Nagaoka
 
KubeCon EU 2016: Kubernetes and the Potential for Higher Level Interfaces
KubeCon EU 2016: Kubernetes and the Potential for Higher Level InterfacesKubeCon EU 2016: Kubernetes and the Potential for Higher Level Interfaces
KubeCon EU 2016: Kubernetes and the Potential for Higher Level Interfaces
KubeAcademy
 
KubeCon EU 2016: A Practical Guide to Container Scheduling
KubeCon EU 2016: A Practical Guide to Container SchedulingKubeCon EU 2016: A Practical Guide to Container Scheduling
KubeCon EU 2016: A Practical Guide to Container Scheduling
KubeAcademy
 
[DockerCon 2019] Hardening Docker daemon with Rootless mode
[DockerCon 2019] Hardening Docker daemon with Rootless mode[DockerCon 2019] Hardening Docker daemon with Rootless mode
[DockerCon 2019] Hardening Docker daemon with Rootless mode
Akihiro Suda
 
Kernel load-balancing for Docker containers using IPVS
Kernel load-balancing for Docker containers using IPVSKernel load-balancing for Docker containers using IPVS
Kernel load-balancing for Docker containers using IPVS
Docker, Inc.
 
Ansible docker
Ansible dockerAnsible docker
Ansible docker
QNIB Solutions
 
Monitoring, Logging and Tracing on Kubernetes
Monitoring, Logging and Tracing on KubernetesMonitoring, Logging and Tracing on Kubernetes
Monitoring, Logging and Tracing on Kubernetes
Martin Etmajer
 
Why Docker? Dayton PHP, April 2017
Why Docker? Dayton PHP, April 2017Why Docker? Dayton PHP, April 2017
Why Docker? Dayton PHP, April 2017
Chris Tankersley
 
Using Nix and Docker as automated deployment solutions
Using Nix and Docker as automated deployment solutionsUsing Nix and Docker as automated deployment solutions
Using Nix and Docker as automated deployment solutions
Sander van der Burg
 
桃園市教育局Docker技術入門與實作
桃園市教育局Docker技術入門與實作桃園市教育局Docker技術入門與實作
桃園市教育局Docker技術入門與實作
Philip Zheng
 
Devfest 2021' - Artifact Registry Introduction (Taipei)
Devfest 2021' - Artifact Registry Introduction (Taipei)Devfest 2021' - Artifact Registry Introduction (Taipei)
Devfest 2021' - Artifact Registry Introduction (Taipei)
KAI CHU CHUNG
 
Terraforming your Infrastructure on GCP
Terraforming your Infrastructure on GCPTerraforming your Infrastructure on GCP
Terraforming your Infrastructure on GCP
Samuel Chow
 
Docker meetup - PaaS interoperability
Docker meetup - PaaS interoperabilityDocker meetup - PaaS interoperability
Docker meetup - PaaS interoperability
Ludovic Piot
 

What's hot (20)

開放運算&GPU技術研究班
開放運算&GPU技術研究班開放運算&GPU技術研究班
開放運算&GPU技術研究班
 
Kubernetes上で動作する機械学習モジュールの配信&管理基盤Rekcurd について
Kubernetes上で動作する機械学習モジュールの配信&管理基盤Rekcurd についてKubernetes上で動作する機械学習モジュールの配信&管理基盤Rekcurd について
Kubernetes上で動作する機械学習モジュールの配信&管理基盤Rekcurd について
 
From Docker to Production - ZendCon 2016
From Docker to Production - ZendCon 2016From Docker to Production - ZendCon 2016
From Docker to Production - ZendCon 2016
 
Kubernetes Basis: Pods, Deployments, and Services
Kubernetes Basis: Pods, Deployments, and ServicesKubernetes Basis: Pods, Deployments, and Services
Kubernetes Basis: Pods, Deployments, and Services
 
Kubernetes for Java developers
Kubernetes for Java developersKubernetes for Java developers
Kubernetes for Java developers
 
From Docker to Production - SunshinePHP 2017
From Docker to Production - SunshinePHP 2017From Docker to Production - SunshinePHP 2017
From Docker to Production - SunshinePHP 2017
 
Redis Meetup TLV - K8s Session 28/10/2018
Redis Meetup TLV - K8s Session 28/10/2018Redis Meetup TLV - K8s Session 28/10/2018
Redis Meetup TLV - K8s Session 28/10/2018
 
Dockerを利用したローカル環境から本番環境までの構築設計
Dockerを利用したローカル環境から本番環境までの構築設計Dockerを利用したローカル環境から本番環境までの構築設計
Dockerを利用したローカル環境から本番環境までの構築設計
 
KubeCon EU 2016: Kubernetes and the Potential for Higher Level Interfaces
KubeCon EU 2016: Kubernetes and the Potential for Higher Level InterfacesKubeCon EU 2016: Kubernetes and the Potential for Higher Level Interfaces
KubeCon EU 2016: Kubernetes and the Potential for Higher Level Interfaces
 
KubeCon EU 2016: A Practical Guide to Container Scheduling
KubeCon EU 2016: A Practical Guide to Container SchedulingKubeCon EU 2016: A Practical Guide to Container Scheduling
KubeCon EU 2016: A Practical Guide to Container Scheduling
 
[DockerCon 2019] Hardening Docker daemon with Rootless mode
[DockerCon 2019] Hardening Docker daemon with Rootless mode[DockerCon 2019] Hardening Docker daemon with Rootless mode
[DockerCon 2019] Hardening Docker daemon with Rootless mode
 
Kernel load-balancing for Docker containers using IPVS
Kernel load-balancing for Docker containers using IPVSKernel load-balancing for Docker containers using IPVS
Kernel load-balancing for Docker containers using IPVS
 
Ansible docker
Ansible dockerAnsible docker
Ansible docker
 
Monitoring, Logging and Tracing on Kubernetes
Monitoring, Logging and Tracing on KubernetesMonitoring, Logging and Tracing on Kubernetes
Monitoring, Logging and Tracing on Kubernetes
 
Why Docker? Dayton PHP, April 2017
Why Docker? Dayton PHP, April 2017Why Docker? Dayton PHP, April 2017
Why Docker? Dayton PHP, April 2017
 
Using Nix and Docker as automated deployment solutions
Using Nix and Docker as automated deployment solutionsUsing Nix and Docker as automated deployment solutions
Using Nix and Docker as automated deployment solutions
 
桃園市教育局Docker技術入門與實作
桃園市教育局Docker技術入門與實作桃園市教育局Docker技術入門與實作
桃園市教育局Docker技術入門與實作
 
Devfest 2021' - Artifact Registry Introduction (Taipei)
Devfest 2021' - Artifact Registry Introduction (Taipei)Devfest 2021' - Artifact Registry Introduction (Taipei)
Devfest 2021' - Artifact Registry Introduction (Taipei)
 
Terraforming your Infrastructure on GCP
Terraforming your Infrastructure on GCPTerraforming your Infrastructure on GCP
Terraforming your Infrastructure on GCP
 
Docker meetup - PaaS interoperability
Docker meetup - PaaS interoperabilityDocker meetup - PaaS interoperability
Docker meetup - PaaS interoperability
 

Similar to Paris container day june17

TIAD 2016 : Migrating 100% of your production services to containers
TIAD 2016 : Migrating 100% of your production services to containersTIAD 2016 : Migrating 100% of your production services to containers
TIAD 2016 : Migrating 100% of your production services to containers
The Incredible Automation Day
 
DCEU 18: Developing with Docker Containers
DCEU 18: Developing with Docker ContainersDCEU 18: Developing with Docker Containers
DCEU 18: Developing with Docker Containers
Docker, Inc.
 
Real World Experience of Running Docker in Development and Production
Real World Experience of Running Docker in Development and ProductionReal World Experience of Running Docker in Development and Production
Real World Experience of Running Docker in Development and Production
Ben Hall
 
Continuous Delivery com Docker, OpenShift e Jenkins
Continuous Delivery com Docker, OpenShift e JenkinsContinuous Delivery com Docker, OpenShift e Jenkins
Continuous Delivery com Docker, OpenShift e Jenkins
Bruno Padilha
 
[EXTENDED] Ceph, Docker, Heroku Slugs, CoreOS and Deis Overview
[EXTENDED] Ceph, Docker, Heroku Slugs, CoreOS and Deis Overview[EXTENDED] Ceph, Docker, Heroku Slugs, CoreOS and Deis Overview
[EXTENDED] Ceph, Docker, Heroku Slugs, CoreOS and Deis Overview
Leo Lorieri
 
Docker for developers on mac and windows
Docker for developers on mac and windowsDocker for developers on mac and windows
Docker for developers on mac and windows
Docker, Inc.
 
Before & After Docker Init
Before & After Docker InitBefore & After Docker Init
Before & After Docker Init
Angel Borroy López
 
Toolbox of a Ruby Team
Toolbox of a Ruby TeamToolbox of a Ruby Team
Toolbox of a Ruby Team
Arto Artnik
 
PuppetConf 2016: The Challenges with Container Configuration – David Lutterko...
PuppetConf 2016: The Challenges with Container Configuration – David Lutterko...PuppetConf 2016: The Challenges with Container Configuration – David Lutterko...
PuppetConf 2016: The Challenges with Container Configuration – David Lutterko...
Puppet
 
Challenges of container configuration
Challenges of container configurationChallenges of container configuration
Challenges of container configuration
lutter
 
Using Docker For Development
Using Docker For DevelopmentUsing Docker For Development
Using Docker For Development
Laura Frank Tacho
 
Docker HK Meetup - 201707
Docker HK Meetup - 201707Docker HK Meetup - 201707
Docker HK Meetup - 201707
Clarence Ho
 
Docking with Docker
Docking with DockerDocking with Docker
Docker as development environment
Docker as development environmentDocker as development environment
Docker as development environment
Bruno de Lima e Silva
 
Unleashing Docker with Pipelines in Bitbucket Cloud
Unleashing Docker with Pipelines in Bitbucket CloudUnleashing Docker with Pipelines in Bitbucket Cloud
Unleashing Docker with Pipelines in Bitbucket Cloud
Atlassian
 
[Codelab 2017] Docker 기초 및 활용 방안
[Codelab 2017] Docker 기초 및 활용 방안[Codelab 2017] Docker 기초 및 활용 방안
[Codelab 2017] Docker 기초 및 활용 방안
양재동 코드랩
 
CI-CD WITH GITLAB WORKFLOW
CI-CD WITH GITLAB WORKFLOWCI-CD WITH GITLAB WORKFLOW
CI-CD WITH GITLAB WORKFLOW
AddWeb Solution Pvt. Ltd.
 
Scaling Docker Containers using Kubernetes and Azure Container Service
Scaling Docker Containers using Kubernetes and Azure Container ServiceScaling Docker Containers using Kubernetes and Azure Container Service
Scaling Docker Containers using Kubernetes and Azure Container Service
Ben Hall
 
Tech Talk: DevOps at LeanIX @ Startup Camp Berlin
Tech Talk: DevOps at LeanIX @ Startup Camp BerlinTech Talk: DevOps at LeanIX @ Startup Camp Berlin
Tech Talk: DevOps at LeanIX @ Startup Camp Berlin
LeanIX GmbH
 
Docker in everyday development
Docker in everyday developmentDocker in everyday development
Docker in everyday development
Justyna Ilczuk
 

Similar to Paris container day june17 (20)

TIAD 2016 : Migrating 100% of your production services to containers
TIAD 2016 : Migrating 100% of your production services to containersTIAD 2016 : Migrating 100% of your production services to containers
TIAD 2016 : Migrating 100% of your production services to containers
 
DCEU 18: Developing with Docker Containers
DCEU 18: Developing with Docker ContainersDCEU 18: Developing with Docker Containers
DCEU 18: Developing with Docker Containers
 
Real World Experience of Running Docker in Development and Production
Real World Experience of Running Docker in Development and ProductionReal World Experience of Running Docker in Development and Production
Real World Experience of Running Docker in Development and Production
 
Continuous Delivery com Docker, OpenShift e Jenkins
Continuous Delivery com Docker, OpenShift e JenkinsContinuous Delivery com Docker, OpenShift e Jenkins
Continuous Delivery com Docker, OpenShift e Jenkins
 
[EXTENDED] Ceph, Docker, Heroku Slugs, CoreOS and Deis Overview
[EXTENDED] Ceph, Docker, Heroku Slugs, CoreOS and Deis Overview[EXTENDED] Ceph, Docker, Heroku Slugs, CoreOS and Deis Overview
[EXTENDED] Ceph, Docker, Heroku Slugs, CoreOS and Deis Overview
 
Docker for developers on mac and windows
Docker for developers on mac and windowsDocker for developers on mac and windows
Docker for developers on mac and windows
 
Before & After Docker Init
Before & After Docker InitBefore & After Docker Init
Before & After Docker Init
 
Toolbox of a Ruby Team
Toolbox of a Ruby TeamToolbox of a Ruby Team
Toolbox of a Ruby Team
 
PuppetConf 2016: The Challenges with Container Configuration – David Lutterko...
PuppetConf 2016: The Challenges with Container Configuration – David Lutterko...PuppetConf 2016: The Challenges with Container Configuration – David Lutterko...
PuppetConf 2016: The Challenges with Container Configuration – David Lutterko...
 
Challenges of container configuration
Challenges of container configurationChallenges of container configuration
Challenges of container configuration
 
Using Docker For Development
Using Docker For DevelopmentUsing Docker For Development
Using Docker For Development
 
Docker HK Meetup - 201707
Docker HK Meetup - 201707Docker HK Meetup - 201707
Docker HK Meetup - 201707
 
Docking with Docker
Docking with DockerDocking with Docker
Docking with Docker
 
Docker as development environment
Docker as development environmentDocker as development environment
Docker as development environment
 
Unleashing Docker with Pipelines in Bitbucket Cloud
Unleashing Docker with Pipelines in Bitbucket CloudUnleashing Docker with Pipelines in Bitbucket Cloud
Unleashing Docker with Pipelines in Bitbucket Cloud
 
[Codelab 2017] Docker 기초 및 활용 방안
[Codelab 2017] Docker 기초 및 활용 방안[Codelab 2017] Docker 기초 및 활용 방안
[Codelab 2017] Docker 기초 및 활용 방안
 
CI-CD WITH GITLAB WORKFLOW
CI-CD WITH GITLAB WORKFLOWCI-CD WITH GITLAB WORKFLOW
CI-CD WITH GITLAB WORKFLOW
 
Scaling Docker Containers using Kubernetes and Azure Container Service
Scaling Docker Containers using Kubernetes and Azure Container ServiceScaling Docker Containers using Kubernetes and Azure Container Service
Scaling Docker Containers using Kubernetes and Azure Container Service
 
Tech Talk: DevOps at LeanIX @ Startup Camp Berlin
Tech Talk: DevOps at LeanIX @ Startup Camp BerlinTech Talk: DevOps at LeanIX @ Startup Camp Berlin
Tech Talk: DevOps at LeanIX @ Startup Camp Berlin
 
Docker in everyday development
Docker in everyday developmentDocker in everyday development
Docker in everyday development
 

Recently uploaded

Demystifying Knowledge Management through Storytelling
Demystifying Knowledge Management through StorytellingDemystifying Knowledge Management through Storytelling
Demystifying Knowledge Management through Storytelling
Enterprise Knowledge
 
"Choosing proper type of scaling", Olena Syrota
"Choosing proper type of scaling", Olena Syrota"Choosing proper type of scaling", Olena Syrota
"Choosing proper type of scaling", Olena Syrota
Fwdays
 
"$10 thousand per minute of downtime: architecture, queues, streaming and fin...
"$10 thousand per minute of downtime: architecture, queues, streaming and fin..."$10 thousand per minute of downtime: architecture, queues, streaming and fin...
"$10 thousand per minute of downtime: architecture, queues, streaming and fin...
Fwdays
 
ScyllaDB Tablets: Rethinking Replication
ScyllaDB Tablets: Rethinking ReplicationScyllaDB Tablets: Rethinking Replication
ScyllaDB Tablets: Rethinking Replication
ScyllaDB
 
Nordic Marketo Engage User Group_June 13_ 2024.pptx
Nordic Marketo Engage User Group_June 13_ 2024.pptxNordic Marketo Engage User Group_June 13_ 2024.pptx
Nordic Marketo Engage User Group_June 13_ 2024.pptx
MichaelKnudsen27
 
"Frontline Battles with DDoS: Best practices and Lessons Learned", Igor Ivaniuk
"Frontline Battles with DDoS: Best practices and Lessons Learned",  Igor Ivaniuk"Frontline Battles with DDoS: Best practices and Lessons Learned",  Igor Ivaniuk
"Frontline Battles with DDoS: Best practices and Lessons Learned", Igor Ivaniuk
Fwdays
 
The Microsoft 365 Migration Tutorial For Beginner.pptx
The Microsoft 365 Migration Tutorial For Beginner.pptxThe Microsoft 365 Migration Tutorial For Beginner.pptx
The Microsoft 365 Migration Tutorial For Beginner.pptx
operationspcvita
 
Poznań ACE event - 19.06.2024 Team 24 Wrapup slidedeck
Poznań ACE event - 19.06.2024 Team 24 Wrapup slidedeckPoznań ACE event - 19.06.2024 Team 24 Wrapup slidedeck
Poznań ACE event - 19.06.2024 Team 24 Wrapup slidedeck
FilipTomaszewski5
 
Leveraging the Graph for Clinical Trials and Standards
Leveraging the Graph for Clinical Trials and StandardsLeveraging the Graph for Clinical Trials and Standards
Leveraging the Graph for Clinical Trials and Standards
Neo4j
 
Principle of conventional tomography-Bibash Shahi ppt..pptx
Principle of conventional tomography-Bibash Shahi ppt..pptxPrinciple of conventional tomography-Bibash Shahi ppt..pptx
Principle of conventional tomography-Bibash Shahi ppt..pptx
BibashShahi
 
Christine's Supplier Sourcing Presentaion.pptx
Christine's Supplier Sourcing Presentaion.pptxChristine's Supplier Sourcing Presentaion.pptx
Christine's Supplier Sourcing Presentaion.pptx
christinelarrosa
 
Main news related to the CCS TSI 2023 (2023/1695)
Main news related to the CCS TSI 2023 (2023/1695)Main news related to the CCS TSI 2023 (2023/1695)
Main news related to the CCS TSI 2023 (2023/1695)
Jakub Marek
 
Essentials of Automations: Exploring Attributes & Automation Parameters
Essentials of Automations: Exploring Attributes & Automation ParametersEssentials of Automations: Exploring Attributes & Automation Parameters
Essentials of Automations: Exploring Attributes & Automation Parameters
Safe Software
 
High performance Serverless Java on AWS- GoTo Amsterdam 2024
High performance Serverless Java on AWS- GoTo Amsterdam 2024High performance Serverless Java on AWS- GoTo Amsterdam 2024
High performance Serverless Java on AWS- GoTo Amsterdam 2024
Vadym Kazulkin
 
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdfHow to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
Chart Kalyan
 
Introduction of Cybersecurity with OSS at Code Europe 2024
Introduction of Cybersecurity with OSS  at Code Europe 2024Introduction of Cybersecurity with OSS  at Code Europe 2024
Introduction of Cybersecurity with OSS at Code Europe 2024
Hiroshi SHIBATA
 
A Deep Dive into ScyllaDB's Architecture
A Deep Dive into ScyllaDB's ArchitectureA Deep Dive into ScyllaDB's Architecture
A Deep Dive into ScyllaDB's Architecture
ScyllaDB
 
Day 2 - Intro to UiPath Studio Fundamentals
Day 2 - Intro to UiPath Studio FundamentalsDay 2 - Intro to UiPath Studio Fundamentals
Day 2 - Intro to UiPath Studio Fundamentals
UiPathCommunity
 
Mutation Testing for Task-Oriented Chatbots
Mutation Testing for Task-Oriented ChatbotsMutation Testing for Task-Oriented Chatbots
Mutation Testing for Task-Oriented Chatbots
Pablo Gómez Abajo
 
Biomedical Knowledge Graphs for Data Scientists and Bioinformaticians
Biomedical Knowledge Graphs for Data Scientists and BioinformaticiansBiomedical Knowledge Graphs for Data Scientists and Bioinformaticians
Biomedical Knowledge Graphs for Data Scientists and Bioinformaticians
Neo4j
 

Recently uploaded (20)

Demystifying Knowledge Management through Storytelling
Demystifying Knowledge Management through StorytellingDemystifying Knowledge Management through Storytelling
Demystifying Knowledge Management through Storytelling
 
"Choosing proper type of scaling", Olena Syrota
"Choosing proper type of scaling", Olena Syrota"Choosing proper type of scaling", Olena Syrota
"Choosing proper type of scaling", Olena Syrota
 
"$10 thousand per minute of downtime: architecture, queues, streaming and fin...
"$10 thousand per minute of downtime: architecture, queues, streaming and fin..."$10 thousand per minute of downtime: architecture, queues, streaming and fin...
"$10 thousand per minute of downtime: architecture, queues, streaming and fin...
 
ScyllaDB Tablets: Rethinking Replication
ScyllaDB Tablets: Rethinking ReplicationScyllaDB Tablets: Rethinking Replication
ScyllaDB Tablets: Rethinking Replication
 
Nordic Marketo Engage User Group_June 13_ 2024.pptx
Nordic Marketo Engage User Group_June 13_ 2024.pptxNordic Marketo Engage User Group_June 13_ 2024.pptx
Nordic Marketo Engage User Group_June 13_ 2024.pptx
 
"Frontline Battles with DDoS: Best practices and Lessons Learned", Igor Ivaniuk
"Frontline Battles with DDoS: Best practices and Lessons Learned",  Igor Ivaniuk"Frontline Battles with DDoS: Best practices and Lessons Learned",  Igor Ivaniuk
"Frontline Battles with DDoS: Best practices and Lessons Learned", Igor Ivaniuk
 
The Microsoft 365 Migration Tutorial For Beginner.pptx
The Microsoft 365 Migration Tutorial For Beginner.pptxThe Microsoft 365 Migration Tutorial For Beginner.pptx
The Microsoft 365 Migration Tutorial For Beginner.pptx
 
Poznań ACE event - 19.06.2024 Team 24 Wrapup slidedeck
Poznań ACE event - 19.06.2024 Team 24 Wrapup slidedeckPoznań ACE event - 19.06.2024 Team 24 Wrapup slidedeck
Poznań ACE event - 19.06.2024 Team 24 Wrapup slidedeck
 
Leveraging the Graph for Clinical Trials and Standards
Leveraging the Graph for Clinical Trials and StandardsLeveraging the Graph for Clinical Trials and Standards
Leveraging the Graph for Clinical Trials and Standards
 
Principle of conventional tomography-Bibash Shahi ppt..pptx
Principle of conventional tomography-Bibash Shahi ppt..pptxPrinciple of conventional tomography-Bibash Shahi ppt..pptx
Principle of conventional tomography-Bibash Shahi ppt..pptx
 
Christine's Supplier Sourcing Presentaion.pptx
Christine's Supplier Sourcing Presentaion.pptxChristine's Supplier Sourcing Presentaion.pptx
Christine's Supplier Sourcing Presentaion.pptx
 
Main news related to the CCS TSI 2023 (2023/1695)
Main news related to the CCS TSI 2023 (2023/1695)Main news related to the CCS TSI 2023 (2023/1695)
Main news related to the CCS TSI 2023 (2023/1695)
 
Essentials of Automations: Exploring Attributes & Automation Parameters
Essentials of Automations: Exploring Attributes & Automation ParametersEssentials of Automations: Exploring Attributes & Automation Parameters
Essentials of Automations: Exploring Attributes & Automation Parameters
 
High performance Serverless Java on AWS- GoTo Amsterdam 2024
High performance Serverless Java on AWS- GoTo Amsterdam 2024High performance Serverless Java on AWS- GoTo Amsterdam 2024
High performance Serverless Java on AWS- GoTo Amsterdam 2024
 
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdfHow to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
 
Introduction of Cybersecurity with OSS at Code Europe 2024
Introduction of Cybersecurity with OSS  at Code Europe 2024Introduction of Cybersecurity with OSS  at Code Europe 2024
Introduction of Cybersecurity with OSS at Code Europe 2024
 
A Deep Dive into ScyllaDB's Architecture
A Deep Dive into ScyllaDB's ArchitectureA Deep Dive into ScyllaDB's Architecture
A Deep Dive into ScyllaDB's Architecture
 
Day 2 - Intro to UiPath Studio Fundamentals
Day 2 - Intro to UiPath Studio FundamentalsDay 2 - Intro to UiPath Studio Fundamentals
Day 2 - Intro to UiPath Studio Fundamentals
 
Mutation Testing for Task-Oriented Chatbots
Mutation Testing for Task-Oriented ChatbotsMutation Testing for Task-Oriented Chatbots
Mutation Testing for Task-Oriented Chatbots
 
Biomedical Knowledge Graphs for Data Scientists and Bioinformaticians
Biomedical Knowledge Graphs for Data Scientists and BioinformaticiansBiomedical Knowledge Graphs for Data Scientists and Bioinformaticians
Biomedical Knowledge Graphs for Data Scientists and Bioinformaticians
 

Paris container day june17

  • 1. Take a BlaBlaCar to the stars with rkt ! Simon Lallemand System Engineer @slallema
  • 2. New infrastructure ecosystem Agenda Key principles 100% rkt powered carpooling Directions for driving change BlaBlaCar Facts, Figures & History
  • 3. Founded in 2006 21 million mobile app downloaded (iPhone + Android) Facts and figures 12 million travelers per quarter 1 million tonnes less CO2 in the past year 40 million members Currently in 22 countries France, Spain, UK, Italy, Poland, Hungary, Croatia, Serbia, Romania, Germany, Belgium, India, Mexico, The Netherlands, Luxembourg, Portugal, Ukraine, Czech Republic, Slovakia, Russia, Brazil and Turkey. CO2
  • 4. Our server growth The evolution of the infrastructure from the begining to now: 2006 2008 2010 2012 2013 2014 2015 2016 Web hosting Dedicated servers Baremetal 1 rack Baremetal 3 racks Baremetal 5 racks Baremetal 8 racks Baremetal 14 racks 150 servers 2 DC Baremetal 17 racks 300 servers 3 DC
  • 5. ~300 bare-metal servers ~400 container images 4000+ running containers
  • 6. Tech evolution The major tech changes in our infrastructure : Virtu 2012 Chef 2013 Foreman 2014 Hardware uniformization 2015 Containers 2015 2016
  • 7. Key principles Leading the industrialization at BlaBlaCar
  • 8. Metal is invisible Decouple hardware management and what runs on it
  • 9. Route everything Keep the network simple and scalable
  • 11. New Infrastructure Ecosystem 100% rkt powered carpooling
  • 13. CoreOS Container Linux On 100% of our new servers
  • 16. dgr Container build and runtime tool Harmonize the way we build Quick build One way of doing things Easy to understand for newcomers As little code replication as possible Templating at container start A good integration with rkt github.com/blablacar/dgr
  • 17. dgr : build directory of an ACI ├── aci-manifest.yml ├── attributes │ └── redis.yml ├── runlevels │ └── build │ └── install.sh └── templates └── etc └── redis └── redis.conf.tmpl A standardized structure Ease maintenance and teamwork Inspired by config management Separation of templates, attributes and scripts.
  • 18. dgr : ACI manifest name: redis:0.1 aci: app: exec: - /usr/bin/redis-server - /etc/redis/redis.conf Simpler manifest format Fill only what is important YAML <3 One process per ACI Composition is done only with POD
  • 19. dgr : ACI manifest name: redis:0.1 aci: app: exec: - /usr/bin/redis-server - /etc/redis/redis.conf dependencies: - debian:8.6 Use of dependencies Composition Lighter images
  • 20. dgr : runlevel build ├── aci-manifest.yml ├── attributes │ └── redis.yml ├── runlevels │ └── build │ └── install.sh ├── templates │ └── etc │ └── redis │ └── redis.conf.tmpl └── tests Scripts executed in the container Build from inside the container with all the dependencies
  • 21. dgr : runlevel build #!/bin/bash apt-get install -y redis-server Install packages and stuff For debian based containers you can install packages using apt-get for instance.
  • 22. dgr : runlevel builder name: example.com/aci-redis-dicator:1 builder: dependencies: - example.org/aci-go - example.org/aci-git aci: app: exec: - /bin/dictator Build from outside of the container Dedicated builder image with go & git
  • 23. dgr : runlevel builder #!/dgr/bin/busybox sh git clone https://github.com/blablacar/redis-dictator cd redis-dictator go build cp dist/dictator ${ROOTFS}/bin Builder script Clone, build and copy binary to target ACI Only /bin/dictator in final ACI
  • 24. dgr : runlevel builder name: example.org/gentoo-redis:1 builder: dependencies: - example.org/gentoo-stage4 aci: dependencies: - example.org/base app: exec: [ /usr/bin/redis-server ] Other example Builder with gentoo’s stage4
  • 25. dgr : runlevel builder #!/dgr/bin/busybox sh emerge -v redis-server Install packages from outside Use emerge to install your app and its dependencies. Redis with dependencies only in the final ACI
  • 26. dgr : templates & attributes ├── aci-manifest.yml ├── attributes │ └── redis.yml ├── runlevels │ └── build │ └── install.sh └── templates └── etc └── redis └── redis.conf.tmpl Render configuration files Templates stored in the aci Default attributes stored in the aci Overridable when used as dependencies Overridable by environment var
  • 27. dgr : templates & attributes # templates/etc/redis.conf.tmpl daemonize no port {{ .redis.port }} … # attributes/redis.yml default: redis: port: 6379
  • 28. dgr : runlevel prestart ├── aci-manifest.yml ├── attributes │ └── redis.yml ├── runlevels │ └── prestart-late │ └── 10-init-db.sh └── templates └── etc └── redis └── redis.conf.tmpl Initialize container prestart-early and prestart-late scripts Before and after templating Initialize environment before exec
  • 29. dgr : testing ├── aci-manifest.yml ├── attributes ├── runlevels ├── templates └── tests └── wait.sh └── my_cool_tests.bats Testing Bats as default tester wait.sh Wait for service to be ready
  • 30. dgr : testing #!/dgr/bin/bats -x @test "Redis should be running" { run bash -c "ps -aux | grep redis-server" [ "$status" -eq 0 ] echo ${lines[0]} [[ "${lines[0]}" =~ "/usr/bin/redis-server" ]] } @test "Redis should listen on port: 6379" { run bash -c "netstat -peanut | grep redis-server" [ "$status" -eq 0 ] [[ "${lines[0]}" =~ ":6379" ]] [[ "${lines[0]}" =~ "redis-server" ]] }
  • 31. dgr : subcommands ~ # dgr init ~ # dgr try ~ # dgr build ~ # dgr test ~ # dgr install ~ # dgr push github.com/blablacar/dgr
  • 32. ggn Manage services in fleet clusters Pods Services and Environments Different services that use the same pods Environments (1 per DC for prod + dev…) Abstraction of fleet commands github.com/blablacar/ggn INSERT LOGO HERE
  • 33. ggn ├── prod-dc1 │ └── attributes │ └── services │ └── redis-quota │ │ └── attributes │ │ └── service-manifest.yml │ │ └── unit.tmpl │ └── redis-rating ├── preprod │ ... Single directory tree Describes the services The environments Versioned in git
  • 34. ggn : service manifest containers: - blablacar.com/aci-redis:3 - blablacar.com/aci-redis-exporter:1 nodes: - hostname: redis-quota1 - hostname: redis-quota2 - hostname: redis-quota3 Define the service ACI images and versions combined to make a pod Nodes composing the service
  • 35. ggn : attributes # attributes/redis.yml override: redis: maxmemory: 4GB TEMPLATER_OVERRIDE={“redis”:{“max memory”: “4GB”}…} Override attributes Inject run context attributes as environment var
  • 36. ggn : unit.tmpl [Unit] Description=Redis POD {{. [Service] KillMode=mixed Restart=always ExecStart=/usr/bin/rkt run --set-env=TEMPLATER_OVERRIDE='{{.jsonAttrs}}' {{.acis}}
  • 37. ggn : run a POD $ ggn prod-dc1 redis-quota update $ ggn prod-dc1 redis-quota1 journal -f github.com/blablacar/ggn
  • 38. Service discovery with nerve & synapse AirBnB’s smartstack Dynamic topology of services Scalability requires service discovery We started with smartstack of AirBnB Since then we improved it and rewrote it in go github.com/blablacar/go-nerve github.com/blablacar/go-synapse
  • 39. /database/node1 go-nerve does health checks and reports to zookeeper in service keys HAProxy node1 Applications hit their local haproxy to access backends Service Discovery /database go-synapse watches zookeeper service keys and reloads haproxy if changes are detected go-nerve Zookeeper go-synapse
  • 40. bare-metal servers 1 type of hardware 3 disk profiles fleet machines CoreOS fleet etcd“Distributed init system” Hardware Container Registry ggn dgr Service Codebase rkt PODs build run store host create Our infrastructure ecosystem nerve mysqld monitoring mysql-main_1 nerve php monitoring nginx synapse front_1 synapse nerve zookeeper Service Discovery
  • 41. Work In Progress ( We’re hiring )