"Walking on water and developing software from a specification are easy if both are frozen." - Edward V. Berard Decisions are context dependent. A right decision at the beginning may seem like a wrong decision 4 years later. This talk will compare the effect of architecture decisions of a small versus a mature project, how to make the "right" decision, and how to deal with problems when the right decision isn't right anymore.

1. The world is not black and white
03.09.2016 Eric Reiche

2. 1
Eric Reiche
26, married, has a daughter
Head of Software Engineering @ FlixBus
Working with 50 people in 7 teams
People from 17 countries
Offices in Berlin, Munich, Kharkiv
Who is this guy?

3. 2
• Technology company
• AirBnB has no flats
• Uber has no cars
• FlixBus has no buses!
• One big brand
• One endpoint for customers
FlixBus

4. 3
Source: MeinFernbus.de, 2013
March 2012

5. 4
Source: MeinFernbus.de
July 2013

6. 5
Source: MeinFernbus.de
December 2014

7. 6
Source: FlixBus.de, 15.08.2016
August 2016
• About 1.000 buses
• 250 bus companies
• 13.000 agencies
• 100.000 tickets daily
• Europe’s largest
long-distance
bus operator

8. 7
2012
• We started with Symfony2, Doctrine, Capistrano
• SOA: Frontend, API, Backoffice
• Webshop for tickets, including payments
• 3 Developers
• First commit to go-live in 3 months
Let‘s go back again

9. 8
Was it right?
Yes!
2012

10. 9
Is it still right?
Of course not.
2016

11. 10
• Without hacking a payment system together in a
few weeks, we would have never made it this far.
• Without sacrificing code quality, we wouldn‘t
have gone live in 3 months.
What‘s the point?

12. 11
BUT!
Over time, these practices need evolution and
revolution.
Be careful with assumptions.
2016

13. 12
"Walking on water and developing software from a
specification are easy if both are frozen.”
- Edward V. Berard

14. 13
• Handle changing requirements
• Only way to grow as fast as we did
• React to the market
• The fast eat the slow
• Don’t use it blindly
• (Agile development makes no sense for NASA)
This is why we are agile

15. 14
• Payment provider changed 6 times
• Payment module rewritten 4 times
• Jenkins for deploys
• Peer code-review
• SOA was a very good idea
• (Our execution wasn‘t)
2016

16. 15
• Team skills
• Symfony2 just came out of beta
• And seemed to do lots of things right
• PHPs strengths are in rapid prototyping
• Runs anywhere (cheap hosting)
• Started bare-metal
• Slowly moving to Kubernetes
Image source: http://imgur.com/gallery/0S5spVm
Why PHP?

17. 16
Back to the future

18. 17
How to handle > 1 million hits per day?
• Caching on load balancer
• NginX can handle a LOT of static traffic
• Caching expensive calculations in memcache/redis
• Queuing transactions
• PDF generation
• Emails

19. 18
• You can start your company with prototype code
• As the project grows, so do the requirements
• Enterprise applications are possible, but need the
same architecture planning as on other platforms
• There are no bad languages, just bad code
(except maybe Visual Basic ;) )
Architecture

20. 19
• Files - only work as long as you have a single server
• MySQL – super heavy
• MySQL Memcache interface – breaks by replication
• Memcache – What if you need to reboot?
• Redis – cluster, persistence
Session management

21. 20
Source: https://twitter.com/codinghorror/status/347070841059692545?
Best practices at FlixBus

22. 21
• Use UTF-8.
• Use mb_*
• Again: don‘t make assumptions.
• UTF-8 might be needed in unexpected places
• E-Mail addresses (International domain names)
Encoding

23. 22
• People have exactly one canonical full name.
• People have exactly one full name which they go by.
• People have, at this point in time, exactly one canonical full name.
• People have, at this point in time, one full name which they go by.
• People have exactly N names, for any value of N.
• People’s names fit within a certain defined amount of space.
• People’s names do not change.
• People’s names change, but only at a certain enumerated set of events.
• People’s names are written in ASCII.
• People’s names are written in any single character set.
• People’s names are all mapped in Unicode code points.
• People’s names are case sensitive.
• [...]
• People whose names break my system are weird outliers. They should have had solid, acceptable names,
like 田中太郎.
• People have names.
http://bit.ly/2aW1AQf / https://www.kalzumeus.com/2010/06/17/falsehoods-programmers-believe-about-names/
Falsehoods Programmers Believe About Names

24. 23
• Monoliths will blow up your database
• Master-Master hot-standby replication (for now)
1. stop replication
2. apply migration
3. re-sync
4. repeat on other master
• Takes hours or days
Database scaling

25. 24
How to prevent „Works on my machine“?
• Make it EASY to get a standardized, local
environment running (Vagrant, Docker)
• Have enough staging environments for everyone
• Make deploys easy
What about staging?

26. 25
• We slice our database dumps for staging and dev
• Anonymize names, emails, phone numbers...
• Remove rides older than one month
• Remove everything attached to these rides
• Orders, Reservations, Customer data
• Runs for days
• Compressed 4GB
What about staging databases?

27. 26
• Nice for consistency
• Prevents decoupling & Microservices
• At some point it will explode
• Now you need to move it to the application layer
• And spend a year refactoring
Foreign key constraints

28. 27
• We used them quite a lot
• Changing them becomes a real PITA
• Migrations are expensive
• JOINs are also expensive
• Depends on the situation
• Recently migrated an ENUM for currencies to
integer in 10 biggest tables
• It took us almost a week – really painful
ENUMs

29. 28
• Small applications can scale SO MUCH easier
• APIs can be reused for other things
• You can start by using bundles which don‘t share
anything (but be pedantic!)
• Use event listeners (but still think about what
you‘re doing/coupling)
SOA / Decoupling

30. 29
• Database transactions can become too big easily
• Try to keep them as small as possible
• Do load testing!
Transactions

31. 30
PM: How long does this story take?
You: 13 points.
PM: And without tests?
You: 21 points.
PM: Why?
You: Because then we have to test everything by
hand and will notice some bugs only later.
Testing

32. 31
• Shared Percona cluster for Business Intelligence
• Very fast kick-off
• Stuck with dependencies between departments
• API would have been nice
Shared database

33. 32
• 2013 we assumed people know what they‘re doing
• Permission system was based on trust
• (which means non-existent)
• Not possible anymore for 1000+ people
• Legal regulations
Security & Permissions

34. 33
• Don‘t use technology „just because“
• Understand what advantages and disadvantages a technology has
• Try to keep your stack small at the beginning
• Utilize the talent in your team once you‘re big enough
By Jeremykemp at English Wikipedia, CC BY-SA 3.0, https://commons.wikimedia.org/w/index.php?curid=10547051
Gartner Hype Cycle

35. 34
• ALWAYS USE UTC!
• It doesn‘t really cost you anything extra
• All projects will grow (or die)
• There are libraries which make your life easier
• https://github.com/celest-time/celest
• It‘s by one of our guys
Time zones

36. 35
• Started with Excel/Word files
• Tried Pootle
• Needs to be idiot proof
• For translation agencies
• Developed our own
• Github / Gitlab support
• Merge Requests
• SPA will be open source
Announcement will be made on http://flix.tech
Translations

37. 36
• Bahn-strike effect: 10x traffic + transactions (!)
• TV mentioning
• If your application is built with scalability in mind,
just fire up some EC2s.
• What happens if the
peak stays?
• Not everything is
easily scalable
Peaks

38. 37
• Don‘t be scared to make a decision
• Always have a good reason
• Plan to re-plan
• If plan A doesn‘t work, the alphabet has 25 more
letters
• Trial & Error is comparably cheap in IT
Summary

39. 38
Want to learn from a big team of awesome people?
Join us in Berlin, Munich or Kharkiv!
https://github.com/flix-tech/jobs
We‘re hiring!

40. 39
Questions?
@ericreiche
eric.reiche@flixbus.com
Feedback welcome: http://bit.do/flix-eric
Thank you!