The document details the infrastructure scaling challenges faced by Cabify from 2011 to 2017, focusing on their transition from VPS to bare metal servers and various database solutions. It highlights performance issues, solutions implemented like changing database technology and load balancing strategies, and the importance of monitoring and system administration. The narrative showcases the evolution and lessons learned in handling increased loads and optimizing their service architecture over the years.

1. Scaling when you are not Google
Abel Muiño

2. Abel Muino
‣ Lead Software Engineer
‣ Tweets as @amuino
‣ In another life, co-owned
1uptalent.com, played with
Docker and used AWS for
everything.

3. Disclaimer
‣ Cabify is 5 years old
‣ I joined Cabify about 1.5 years ago to work on product
‣ What you will hear today might be
‣ 70% folklore / 30% experience
‣ Only about production
‣ Not applicable to other areas (data analytics)

5. Cabify

6. 2011 2012 2013 2014 2015 2016
Completed Journeys
(Axis has no legend because NDA and stuff)

7. Backend committers
0
5
9
14
18
2011 2012 2013 2014 2015 2016

8. We are hiring!
(As if it wasn’t obvious from the charts)

9. Circadian rhythm

10. Prelude
???? - 2014

11. Cabify foundations
‣ Mostly Ruby, some Go
‣ Running on VPS
‣ No sysadmins (devops?)
‣ CouchDB
‣ Redis
‣ Home-grown metrics &
monitoring (limited)

12. Servers
‣ 3 ⨉ Host servers
‣ Horizontally scalable
‣ Most services included (sidecars)
‣ Front + Back + Queue workers
‣ 1 ⨉ Realtime server
‣ Single Point of Failure
‣ Ansible for setting them up
VPS
Provider
LB
web1 web2 web3
worker1
LBLB
redis1 redis2elastic
realtime osrm
websock

13. CouchDB
‣ Used to be run in-house → Unreliable
‣ Moved to Cloudant
‣ Managed
‣ Bare metal servers
‣ Requisite for everything else: to run on the same datacenter
‣ …because the network matters
Database of choice for Cabify

14. Pros
‣ Cheap servers
‣ Profesional DB management
‣ Still cheaper than in-house staff
‣ Scales up by either
‣ Emailing Cloudant
‣ Deploying new VPSs
‣ Datacenter lock-in
‣ Scarce visibility on load
‣ Low VPS utilization (for some
services)
Cons

15. Tl;dr: everything was fine
Until it wasn’t

16. 2015 Road to bare metal

17. In 2014 we handled
7 times the load of 2013

18. Installed NewRelic
‣ Monitors our ruby stack
‣ Built custom adapters for API toolkit and CouchDB
‣ Golang not supported 😭
‣ Low hanging fruit for increasing performance
‣ Hint: Always contact a Sales Rep
‣ Bye bye home-grown monitoring! 👋

19. VPS provider
DDoSed
‣ Several times a week
‣ Cabify was unreachable
‣ VPSs where unreachable
on the internal network
‣ Slow & bad support
‣ Reputation
‣ Solution: Level up!

20. Nobody ever got fired for choosing IBM
Moved to Bare Metal @ Softlayer
Same guys hosting our Cloudant cluster 👍

21. Mindset
Control the core, minimise work for everything else

22. Everything must go
VPS
Provider
web1 web2 web3
worker1realtime
LBLBLB
redis1 redis2elastic
osrm
subscriber

23. Load Balancer
‣ Multiple PoP (starting operations in several countries)
‣ CDN
‣ Supporting websockets
‣ … and Load Balancing
‣ Low TCO
‣ https://www.incapsula.com

24. Redis, ElasticSearch
‣ Same datacenter
‣ Completely managed
‣ Clustered / reliable
‣ RedisLabs
‣ Bonus: Memcached
‣ Qbox

25. OSRM
‣ Same datacenter
‣ Completely managed
‣ Enhanced dataset
‣ Google Maps & Places (with enterprise license)
‣ 2 / 3, good enough

26. Can do better?
Can we manage less infra?
Softlayer
web1 web2 web3
worker1realtime
Google
subscriber
Incapsula
RedislabsRedislabsRedislabs
qboxqboxQbox
RedislabsRedislabsCloudant

27. Subscriber
‣ Felt like reinventing the wheel
‣ Looked for battle-tested bus / queue / broker
‣ In the same datacenter
‣ Had previous experience with RabbitMQ
‣ CloudAMQP
Homebrew message bus / queue

28. Sidecars
‣ Every server could run Cabify
‣ All services installed
‣ Except Realtime (SPOF)
‣ Horizontal scaling
‣ Good server utilisation (bare metal servers are larger)
Make each host self-sufficient

29. Cut own servers in 50%
Served 5 times more requests
Softlayer
host01 host02 host03
realtime
Google
Incapsula
RedislabsRedislabsRedislabs
qboxqboxqbox
RedislabsRedislabsCloudant
CloudAMQPCloudAMQP

30. Pros
‣ Same-datacenter latencies
‣ Only care about our product
‣ Still cheaper than in-house staff
‣ Scales up by either
‣ Emailing a provider
‣ Deploying new Servers
‣ Good visibility on perf
‣ Datacenter lock-in
‣ Still no visibility on Golang perf
‣ Competing services on each
server with different needs
‣ Fast & light http requests
‣ Slow & heavy queue workers
‣ Debugability
Cons

31. Tl;dr: everything was fine
Until it wasn’t

32. 2016, pushing to
the limit

33. In 2015 we handled
5 times the load of 2014

34. In 2016 we would invade LatAm
(new countries, cities, marketing…)

35. Bumps on the road
‣ Start seeing intermittent latency spikes on Cloudant
‣ Disable some services, get back on track
‣ Tied to peak hours
‣ We lived through these, but was stressful

36. Be easy on the database
‣ Removed frequent N+1 queries patterns
‣ Moved some queries to ElasticSearch
‣ Started caching more on Memcache
‣ Grew the cluster
‣ From 200ms to 100ms (average) 👏
(trying to sleep better)

37. RabbitMQ can’t cope
‣ We saturated the cluster CPU with moderate load
‣ Tied to us using tag-based routing
‣ Messages were delivered much later than expected
‣ Made changes to use simpler routing
‣ Is there anything simpler than RabbitMQ for simple routing? 🤔

38. Interlude
DynDNS goes down, Cloudant uses them
We lose access to our databases cluster load balancer
Patched /etc/hosts with the actual ips in 30 minutes

39. The right tool for the job
‣ Clouchdb / Cloudant, not the best database for frequent updates
‣ Looking for alternatives to store fast-changing models
‣ RethinkDB
‣ Fast, easy to use, hosted options in same datacenter
‣ Streaming query updates
Expecting growth in line with previous years

40. Broke RethinkDB load balancer
Database stats were OK, but the LB couldn’t handle our rate
Slow support, no “enterprise” option
Decided to phase out RethinkDB

41. Wrote our first «database»
Simple in-memory store, backed by Couchdb
Update indexes on writes. All queries are indexed
Implemented in Golang, consumed from Ruby
Replaces RethinkDB, which replaced CouchDB

42. Cloudant latency spikes fixed!
Grow the cluster for the second time in the year
Load balancers hardware upgraded, problems gone
Also reduced the number of connections from ruby

43. Relax the Sidecars
‣ Load on background workers interfering with serving http
‣ Split the servers:
‣ Front (ruby/golang http interface)
‣ Workers (ruby job queues, ruby background)

44. Remove RabbitMQ
Replace with NSQ
Nice mix of sidecar and discovery

45. Softlayer
Multiplied own servers by 3
Served 4 times more requests
Google
Incapsula
RedislabsRedislabsRedislabs
qboxqboxqbox
RedislabsRedislabsCloudant
CloudAMQPCloudAMQP
host01-09host01-09host01-09host01-09
rt01-02rt01-02
work01-03work01-03work01-03

46. Pros
‣ Despite the problems, we had
top-notch support from
Cloudant
‣ Easy to scale out
‣ In-process database opened
doors to new features
‣ Datacenter lock-in
‣ Still no visibility on Golang perf
Cons

47. Cabify @ 2017

48. In 2016 we handled
4 times the load of 2015

49. Hired our first
full-time sysadmin!

50. Taking ownership
Improve our infra

51. Own load balancers
‣ Still use Incapsula for its PoP
‣ Achieved much better load balancing
‣ 3 new dedicated servers
Better control & traceability

52. Plans for the future

53. Own redis cluster
‣ Migrating from Redislabs hosted to Redislabs Enterprise
‣ hosted used virtual servers
‣ we rely heavily on redis (and memcached)
‣ 3 new dedicated servers
‣ WIP
Better control & traceability

54. Ruby → Elixir
‣ Fun to code with
‣ Higher performance
‣ Less memory
‣ Investment, about to release first service to production

55. Extract from Product
Dedicated teams and resources for specific components
Make the core of Cabify leaner

56. Thanks!
And sorry for the 60 slides

57. Questions?

58. Abel Muiño
@amuino