Take a BlaBlaCar to the stars with rkt ! Simon Lallemand

1. Take a BlaBlaCar to the stars with rkt !
Simon Lallemand
System Engineer
@slallema

2. New infrastructure
ecosystem
Agenda
Key principles
100% rkt powered carpooling
Directions for driving change
BlaBlaCar Facts, Figures & History

3. Founded
in 2006
21 million mobile
app downloaded
(iPhone + Android)
Facts and figures
12 million travelers
per quarter
1 million
tonnes less CO2 in the
past year
40 million
members
Currently in
22 countries
France, Spain, UK, Italy, Poland,
Hungary, Croatia, Serbia, Romania,
Germany, Belgium, India, Mexico,
The Netherlands, Luxembourg,
Portugal, Ukraine, Czech Republic,
Slovakia, Russia, Brazil and Turkey.
CO2

4. Our server growth
The evolution of the infrastructure from the begining to now:
2006 2008 2010 2012 2013 2014 2015 2016
Web
hosting
Dedicated
servers
Baremetal
1 rack
Baremetal
3 racks
Baremetal
5 racks
Baremetal
8 racks
Baremetal
14 racks
150 servers
2 DC
Baremetal
17 racks
300 servers
3 DC

5. ~300 bare-metal servers
~400 container images
4000+ running containers

6. Tech evolution
The major tech changes in our infrastructure :
Virtu
2012
Chef
2013
Foreman
2014
Hardware
uniformization
2015
Containers
2015
2016

7. Key principles
Leading the industrialization at BlaBlaCar

8. Metal is invisible
Decouple hardware management and
what runs on it

9. Route everything
Keep the network simple and scalable

10. Remove snowflakes
Be service oriented

11. New Infrastructure Ecosystem
100% rkt powered carpooling

12. Containers
For everything !

13. CoreOS Container Linux
On 100% of our new servers

14. rkt
as container runtime

15. New Infrastructure Ecosystem
The tools

16. dgr
Container build
and runtime tool
Harmonize the way we build
Quick build
One way of doing things
Easy to understand for newcomers
As little code replication as possible
Templating at container start
A good integration with rkt
github.com/blablacar/dgr

17. dgr : build directory of an ACI
├── aci-manifest.yml
├── attributes
│ └── redis.yml
├── runlevels
│ └── build
│ └── install.sh
└── templates
└── etc
└── redis
└── redis.conf.tmpl
A standardized structure
Ease maintenance and teamwork
Inspired by config management
Separation of templates, attributes and scripts.

18. dgr : ACI manifest
name: redis:0.1
aci:
app:
exec:
- /usr/bin/redis-server
- /etc/redis/redis.conf
Simpler manifest format
Fill only what is important
YAML <3
One process per ACI
Composition is done only with POD

19. dgr : ACI manifest
name: redis:0.1
aci:
app:
exec:
- /usr/bin/redis-server
- /etc/redis/redis.conf
dependencies:
- debian:8.6
Use of dependencies
Composition
Lighter images

20. dgr : runlevel build
├── aci-manifest.yml
├── attributes
│ └── redis.yml
├── runlevels
│ └── build
│ └── install.sh
├── templates
│ └── etc
│ └── redis
│ └── redis.conf.tmpl
└── tests
Scripts executed in the container
Build from inside the container with all the
dependencies

21. dgr : runlevel build
#!/bin/bash
apt-get install -y redis-server
Install packages and stuff
For debian based containers you can install
packages using apt-get for instance.

22. dgr : runlevel builder
name: example.com/aci-redis-dicator:1
builder:
dependencies:
- example.org/aci-go
- example.org/aci-git
aci:
app:
exec:
- /bin/dictator
Build from outside of the container
Dedicated builder image with go & git

23. dgr : runlevel builder
#!/dgr/bin/busybox sh
git clone
https://github.com/blablacar/redis-dictator
cd redis-dictator
go build
cp dist/dictator ${ROOTFS}/bin
Builder script
Clone, build and copy binary to
target ACI
Only /bin/dictator in final ACI

24. dgr : runlevel builder
name: example.org/gentoo-redis:1
builder:
dependencies:
- example.org/gentoo-stage4
aci:
dependencies:
- example.org/base
app:
exec: [ /usr/bin/redis-server ]
Other example
Builder with gentoo’s stage4

25. dgr : runlevel builder
#!/dgr/bin/busybox sh
emerge -v redis-server
Install packages from outside
Use emerge to install your app and its
dependencies.
Redis with dependencies only in the final
ACI

26. dgr : templates & attributes
├── aci-manifest.yml
├── attributes
│ └── redis.yml
├── runlevels
│ └── build
│ └── install.sh
└── templates
└── etc
└── redis
└── redis.conf.tmpl
Render configuration files
Templates stored in the aci
Default attributes stored in the aci
Overridable when used as dependencies
Overridable by environment var

27. dgr : templates & attributes
# templates/etc/redis.conf.tmpl
daemonize no
port {{ .redis.port }}
…
# attributes/redis.yml
default:
redis:
port: 6379

28. dgr : runlevel prestart
├── aci-manifest.yml
├── attributes
│ └── redis.yml
├── runlevels
│ └── prestart-late
│ └── 10-init-db.sh
└── templates
└── etc
└── redis
└── redis.conf.tmpl
Initialize container
prestart-early and prestart-late scripts
Before and after templating
Initialize environment before exec

29. dgr : testing
├── aci-manifest.yml
├── attributes
├── runlevels
├── templates
└── tests
└── wait.sh
└── my_cool_tests.bats
Testing
Bats as default tester
wait.sh
Wait for service to be ready

30. dgr : testing
#!/dgr/bin/bats -x
@test "Redis should be running" {
run bash -c "ps -aux | grep redis-server"
[ "$status" -eq 0 ]
echo ${lines[0]}
[[ "${lines[0]}" =~ "/usr/bin/redis-server" ]]
}
@test "Redis should listen on port: 6379" {
run bash -c "netstat -peanut | grep redis-server"
[ "$status" -eq 0 ]
[[ "${lines[0]}" =~ ":6379" ]]
[[ "${lines[0]}" =~ "redis-server" ]]
}

31. dgr : subcommands
~ # dgr init
~ # dgr try
~ # dgr build
~ # dgr test
~ # dgr install
~ # dgr push
github.com/blablacar/dgr

32. ggn
Manage services
in fleet clusters
Pods Services and Environments
Different services that use the same pods
Environments (1 per DC for prod + dev…)
Abstraction of fleet commands
github.com/blablacar/ggn
INSERT
LOGO
HERE

33. ggn
├── prod-dc1
│ └── attributes
│ └── services
│ └── redis-quota
│ │ └── attributes
│ │ └── service-manifest.yml
│ │ └── unit.tmpl
│ └── redis-rating
├── preprod
│
...
Single directory tree
Describes the services
The environments
Versioned in git

34. ggn : service manifest
containers:
- blablacar.com/aci-redis:3
- blablacar.com/aci-redis-exporter:1
nodes:
- hostname: redis-quota1
- hostname: redis-quota2
- hostname: redis-quota3
Define the service
ACI images and versions combined
to make a pod
Nodes composing the service

35. ggn : attributes
# attributes/redis.yml
override:
redis:
maxmemory: 4GB
TEMPLATER_OVERRIDE={“redis”:{“max
memory”: “4GB”}…}
Override attributes
Inject run context attributes as environment var

36. ggn : unit.tmpl
[Unit]
Description=Redis POD {{.
[Service]
KillMode=mixed
Restart=always
ExecStart=/usr/bin/rkt run
--set-env=TEMPLATER_OVERRIDE='{{.jsonAttrs}}'
{{.acis}}

37. ggn : run a POD
$ ggn prod-dc1 redis-quota update
$ ggn prod-dc1 redis-quota1 journal -f
github.com/blablacar/ggn

38. Service
discovery
with nerve & synapse
AirBnB’s smartstack
Dynamic topology of services
Scalability requires service discovery
We started with smartstack of AirBnB
Since then we improved it and rewrote
it in go
github.com/blablacar/go-nerve
github.com/blablacar/go-synapse

39. /database/node1
go-nerve does health checks and
reports to zookeeper in service
keys
HAProxy
node1
Applications hit their local
haproxy to access backends
Service Discovery
/database
go-synapse watches zookeeper
service keys and reloads haproxy
if changes are detected
go-nerve
Zookeeper
go-synapse

40. bare-metal servers
1 type of hardware
3 disk profiles
fleet machines
CoreOS
fleet etcd“Distributed init system”
Hardware
Container Registry
ggn
dgr
Service Codebase
rkt PODs
build
run
store
host
create
Our infrastructure ecosystem
nerve
mysqld
monitoring
mysql-main_1
nerve
php
monitoring
nginx
synapse
front_1
synapse
nerve
zookeeper
Service Discovery

41. Work In Progress
( We’re hiring )

42. @slallema
@BlaBlaCarTech

43. Thanks!
@BlaBlaCarTech
BlaBlaTech.com