INNOVATION DANK DEVOPS
Jesper Richter-Reichhelm (@jrirei)
BackendClient
BackendClient
BackendClient
BackendClient
DevOps
Some say DevOps is about
• Agile admins
• Faster releases
• Virtualization
• Automation tools
I say it’s about
• Big Picture Mindset
• Reality Feedback
• Learning together
Classic way
Dev
Ops
Wooga way
Dev
Ops
Dev
Ops
DevOps is about
Monster World
SQL
app
lb
SQL
Browser
sql sql
appapp
Things we did
Many small optimisations
Bigger databases…
… many, many more of them
app
slave
master master
slave
1. create new DBs
2. setup replication
3. start using masters
4. cut replication
5. truncate data
BA
AB
AB
AB
AB
app
slave
master master
slave slave
master master
slave
AC
AC
BD
BD
AC
AC
BD
BD
Stateless approach
1+ million users / day
10+ billion requests / month
~10 DB writes / request
> 100,000 DB writes / second
http://www.flickr.com/photos/biblicone/3425903181/sizes/l/in/photostream/
Things we did
Many small optimisations
Bigger databases…
… many, many more of them
… and finally the right thing!
SQL SQL
sql sql
SQL SQL
sql sql
SQL SQL
sql
Rd Rd
rd rd
Rd Rd
rd rd
Rd Rd
rd rd
Rd Rd
rd rd
app appapp app app app app app app app app appapp app app
app appapp app app app app app app app app appapp app app
app appapp app app app app app app app app appapp app app
app appapp app app app app app app app app appapp app app
app appapp app app app app app app app app appapp app app
app appapp app app app app app app app app appapp app app
app appapp app app app app app app app app appapp app app
SQL
EBS EBS EBS EBS
lb lblblb
EBSSQLBrowser
lb
lb
lb
lb
app
app
app
app
app
app
app app
EBSSQLBrowser
lb
lb
lb
lb
app
app
app
app
app
app
app app
EBSSQLBrowser
lb
lb
lb
lb
app
app
app
app
app
app
app app
SQL SQL
sql sql
SQL SQL
sql sql
SQL SQL
sql
Rd Rd
rd rd
Rd Rd
rd rd
Rd Rd
rd rd
Rd Rd
rd rd
app appapp app app app app app app app app appapp app app
app appapp app app app app app app app app appapp app app
app appapp app app app app app app app app appapp app app
app appapp app app app app app app app app appapp app app
app appapp app app app app app app app app appapp app app
app appapp app app app app app app app app appapp app app
app appapp app app app app app app app app appapp app app
SQL
EBS EBS EBS EBS
lb lblblb
38 single points of failure
flickr.com/photos/qubodup/9244593352
Innovation
Wooga way
FE Dev
BE Dev
Art
Product
“You build it, you run it.
- Werner Vogels
Freedom
Monster World
Monster World
Magic LandMonster World
Magic Landter World
Magic Land
Server Database
Server Database
One Game Session
Stateful approach
1+ million users / day
4 sessions / user / day
1 DB write / session
~50 DB writes / second
Server
sessionsessionsessionsession
Server
sessionsessionsessionsession
Server
sessionsessionsessionsession
S3
http://www.flickr.com/photos/aigle_dore/
Knowledge
Exchange
DevOpsDevOps ???
Wooga way
FE Dev
BE Dev
Art
Product
Exchanging knowledge
flickr.com/photos/wili/233621595
Being transparent
flickr.com/photos/marcomagrini/698692268
flickr.com/groups/bestfavoriterelayrace/
Rotating jobs
2009
Ruby
LAMP
stateless
BB BI
MW
Redis HH
LAMP
stateless
BB BI
Ruby MW
MW
2
Ruby MW
MW
2
2010
LAMP
stateless
BB BI
Redis HH
Ruby MW
MW
2
2010
Erlang
stateful
ML
2011
Redis
Ruby
LAMP
stateless
BB BI
MW
HH
MW
2
Erlang
stateful
ML
offline
DD
Erlang
stateful
ML
2011
Redis
Ruby
Riak
Javascript
ev. Ruby
1 doc
LAMP
stateless
BB BI
MW
HH
MW
2
PI
PP
MWM
2012
Erlang
Redis
Ruby
stateful
Riak
Javascript
offline
ev. Ruby
1 doc
JRuby
shared
LAMP
multiplay
stateless
BB BI
MW
HH
MW
2
ML
DD
PI
PP
MWM PV
GX
FT
KB
BI 2
2013
Erlang
Redis
Ruby
stateful
Riak
Javascript
realtime
offline
ev. Ruby
1 doc
JRuby Scala
shared
LAMP
generic
multiplay
stateless
services
BB BI
MW
HH
MW
2
ML
DD
PI
PP
MWM PV
SBS
DYN
GX
FT
KB
BI 2
DIV
…
2014
Erlang
Redis
Ruby
stateful
Riak
Javascript
realtime
offline
ev. Ruby
1 doc
JRuby Scala
shared
LAMP
generic
multiplay
stateless
services
BB BI
MW
HH
MW
2
ML
DD
PI
PP
MWM PV
SBS
DYN
GX
FT
KB
BI 2
DIV
… …
Erlang
Redis
Ruby
stateful
Riak
Javascript
realtime
offline
ev. Ruby
1 doc
JRuby Scala
shared
Go
Elixir
hybrid
LAMP
generic
multiplay
stateless
services
BB BI
MW
HH
MW
2
ML
DD
PI
PP
MWM PV
SBS
DYN
GX
FT
KB
BI 2
DIV
… …
…
…
…
2015
Erlang
Redis
Ruby
stateful
Riak
Javascript
realtime
offline
ev. Ruby
1 doc
JRuby Scala
shared
Go
Elixir
hybrid
LAMP
generic
multiplay
stateless
services2015
MySQLPython
Python MySQL
Erlang
RedisRuby stateful
Riak
Javascript
realtime
offline
ev. Ruby
1 doc
JRuby
Scala
shared
Go
Elixir
hybrid
generic
multiplay
stateless
services
Languages DBs State mngt. Misc.
Python MySQL
Erlang
RedisRuby stateful
Riak
Javascript
realtime
offline
ev. Ruby
1 doc
JRuby
Scala
shared
Go
Elixir
hybrid
generic
multiplay
stateless
services
Languages DBs State mngt. Misc.
Wooga’s
toolbox
Having trust
“Build projects around
motivated individuals.
Give them the
environment and
support they need,
and trust them to get
the job done.”
- Agile Manifesto
Innovation
Thank you
Questions?
Jesper Richter-Reichhelm (@jrirei)
Questions?
Jesper Richter-Reichhelm (@jrirei)

Innovation dank DevOps (DevOpsCon Berlin 2015)