Learn about Fiverr's Atlas journey

1. MongoDB @ Fiverr:
The Road to Atlas
14 May 2019

2. © 2019 Fiverr Int. Lmt. All Rights Reserved. Proprietary & Confidential.
Sergei Makarevich Moshe Belostotsky
Hi! We are,

3. © 2019 Fiverr Int. Lmt. All Rights Reserved. Proprietary & Confidential.
What is Fiverr?

4. © 2019 Fiverr Int. Lmt. All Rights Reserved. Proprietary & Confidential.
Fiverr's mission is to change how the world works
together.
Fiver is a global marketplace connecting freelancers
and businesses for their digital services needs.
Meet Fiverr.

5. © 2019 Fiverr Int. Lmt. All Rights Reserved. Proprietary & Confidential.

6. © 2019 Fiverr Int. Lmt. All Rights Reserved. Proprietary & Confidential.
Background: Fiverr development DNA
Some facts...
● Task Forces - Business Domain oriented development teams
● Very fast MVP development cycle
● Many AB tests at once (80~ in parallel, 25~ new per week)
● Large amount of rollouts (52~ at once)
● High deploy rate (100~ per day)
● Ruby as the main language, Golang and Nodejs
● Micro-services architecture with CQRS (command query responsibility segregation) and event sourcing

7. © 2019 Fiverr Int. Lmt. All Rights Reserved. Proprietary & Confidential.
Once upon a time
There was a tiny startup
That had to get to market- FAST

8. © 2019 Fiverr Int. Lmt. All Rights Reserved. Proprietary & Confidential.
Architecture V1 – The quick way
Main incentive: get to market - FAST
SQL
Monolith:
Ruby on Rails
● Fastest to iterate
● Single SQL Database
● Memcached as caching layer
Memcached
READ/WRITE READ/WRITE

9. © 2019 Fiverr Int. Lmt. All Rights Reserved. Proprietary & Confidential.
Advantages of Relational Schema backed Database
Why do we want to use it?
● Fast and simple integration with RoR ORM
● “Protection” from data inconsistency
● Common knowledge and practice
● Reliability and resilience

10. © 2019 Fiverr Int. Lmt. All Rights Reserved. Proprietary & Confidential.
But then…
The small startup became bigger...

11. © 2019 Fiverr Int. Lmt. All Rights Reserved. Proprietary & Confidential.
Relational Database Bottleneck
The collapse of the naive approach
● Table locks on multiple insert/updates
● Severe impact on large tables with high write rate due to the locks above
● Schema changes are a “pain”

12. © 2019 Fiverr Int. Lmt. All Rights Reserved. Proprietary & Confidential.
Architecture V2 - Async writes
Main incentive: Fiverr at scale
SQL
Ruby on Rails
Monolith
● Async writes to alleviate stress from web
● RabbitMQ for passing command events
RabbitMQ
WorkerREAD/WRITE
WRITE
Memcached
READ

13. © 2019 Fiverr Int. Lmt. All Rights Reserved. Proprietary & Confidential.
And bigger...

14. © 2019 Fiverr Int. Lmt. All Rights Reserved. Proprietary & Confidential.
Problem: Relational Database slows us down
We need to find another solution...
● Databases became too big with too many indexes to alter schema without downtime
● Database schema changes require “maintenance windows”
● Transactions are not “MUST” on most of the features
● The need to change schema slows the development
● Code reverts should drive schema updates as well

15. © 2019 Fiverr Int. Lmt. All Rights Reserved. Proprietary & Confidential.
Solution: Document store Database
Why do we want to use it?
● Faster than Relational database on large sets
● No schema
○ No need of infra changes with new features
○ Also when reverting a feature…
● Have good enough ORM for RoR (Mongoid)
● Can create databases/collections without DevOps as bottle-neck

16. © 2019 Fiverr Int. Lmt. All Rights Reserved. Proprietary & Confidential.
Architecture V3 - Microservices
Main incentive: Scale for productivity
Ruby on Rails Monolith
● Microservices
● Use new tech outside the monolith
● Teams get more ownership and
autonomy
● Schemaless and fast storage -
Introduced MongoDB RabbitMQ
Worker
AMQP
Consumer
MongoDB
Service
HTTP API
Worker
AMQP
Consumer
MongoDB
Service
HTTP API
READ/WRITE
Gigs serviceOrders service
Memcached
READ/WRITE
READ/WRITE

17. © 2019 Fiverr Int. Lmt. All Rights Reserved. Proprietary & Confidential.
MongoDB replica-sets in Fiverr
mongo-rs
fiverr_prod
RoR
app1
app2app3
app4
app5
mongo-rs2
fiverr_prod
app4
app5
app6
mongo-rs3
fiverr_prod
app7
app8
app9
mongo-rs4
fiverr_prod
app10
app11
app12

18. © 2019 Fiverr Int. Lmt. All Rights Reserved. Proprietary & Confidential.
And even bigger...

19. © 2019 Fiverr Int. Lmt. All Rights Reserved. Proprietary & Confidential.
Architecture V4 - LOSA
Data flow
Worker
AMQP&Kafka
Consumer
Dataview
Service
HTTP API
Gig authoring service
RoR (API Gateway)
Worker
Kafka
Consumer
Dataview
Service
HTTP API
Controller
READ WRITE
Buyer dashboard
Kafka
PRODUCE/CONSUME
View (Server side rendered)
CONSUME
● Micro-frontends
● Kubernetes
● Backend, view oriented, microservices
serving data over REST
● Read optimized private DBs
● Kafka as event broker

20. © 2019 Fiverr Int. Lmt. All Rights Reserved. Proprietary & Confidential.
MongoDB replica-sets in fiverr
mongo-rs
RoR
app1
app2app3
app4
app5
mongo-rs2
fiverr_prod
app4
app5
app6
mongo-rs3
fiverr_prod
app7
app8
app9
mongo-rs4
app10
app11
app12
mongo-rs5
app...
app...
app...
mongo-rs6
app...
app...
app...
mongo-rs7
app...
app...
app...
fiverr_prod
mongo-rs8
app...
app...
app...

21. © 2019 Fiverr Int. Lmt. All Rights Reserved. Proprietary & Confidential.
Problem: Cross-domain self-hosted databases
● Database sizing by the highest consumer and buffer
● Failure causes cross domain interruptions - Up to full service denial
● No ownership on the data and performance: DevOps as bottleneck
● Takes a lot of time and resources to maintain/backup/support/optimize the databases

22. © 2019 Fiverr Int. Lmt. All Rights Reserved. Proprietary & Confidential.
Solution: Domain Dedicated databases
● More accurate size: smaller databases
● Failures cause small extent of damage (mostly one domain)
● Co-ownership between Domain (Development Team) ownership and DevOps
● Higher Development team autonomy

23. © 2019 Fiverr Int. Lmt. All Rights Reserved. Proprietary & Confidential.
MongoDB as a service
Do you really want to manage so many
clusters on your own??

24. © 2019 Fiverr Int. Lmt. All Rights Reserved. Proprietary & Confidential.
MongoDB Atlas
And so, our road to Atlas begins...

25. © 2019 Fiverr Int. Lmt. All Rights Reserved. Proprietary & Confidential.
MongoDB Atlas
First, we need to make some lists...

26. © 2019 Fiverr Int. Lmt. All Rights Reserved. Proprietary & Confidential.
Knowing Yourself
Making checklists and preparing the existing environment
● Our source databases (versions, storage engines)
● Client specs (their types and versions)
● Application classification (migration readiness)
● Source infra specifications (VPC)
● New database scale and sizing
● Activating new features after the cutoff

27. © 2019 Fiverr Int. Lmt. All Rights Reserved. Proprietary & Confidential.
Knowing Yourself
More preparations and coordinations
● How and when to migrate?
● Who should check the process?
● A new staging environment?
● Breaking changes that might pop-up?

28. © 2019 Fiverr Int. Lmt. All Rights Reserved. Proprietary & Confidential.
MongoDB Atlas
Once we were all set up, we took on the road

29. © 2019 Fiverr Int. Lmt. All Rights Reserved. Proprietary & Confidential.
The Migration Process
Automating the process
● Running the migration process itself
● Pitfalls and how to overcome them
● Failures along the road
● System impact during the migration

30. © 2019 Fiverr Int. Lmt. All Rights Reserved. Proprietary & Confidential.
The Migration Process
Switching over to Atlas
● Finishing the migration process
● Defining successful migration
● Post migration actions
● Monitoring application and cluster behavior

31. © 2019 Fiverr Int. Lmt. All Rights Reserved. Proprietary & Confidential.
MongoDB Atlas
Taking first steps, and it is a whole new world

32. © 2019 Fiverr Int. Lmt. All Rights Reserved. Proprietary & Confidential.
New Environment and Features
First technical impressions and perks
● The alerts and activity systems
● Real time and metrics monitoring
● The performance advisor tab
● Users and teams access control
● On-demand cluster configuration update

33. © 2019 Fiverr Int. Lmt. All Rights Reserved. Proprietary & Confidential.
Knowing the Road
The ongoing work and migrations
● Monitoring systems consolidation
● Building new and scaling existing clusters
● Working with the Atlas API
● Setting up proper alerts and signals
● Refreshing the staging environment

34. © 2019 Fiverr Int. Lmt. All Rights Reserved. Proprietary & Confidential.
So what’s next?
Things we are planning
● Fully automate cluster creation and scaling
● Continue data migrations to Atlas
● Align all clients with Atlas prerequisites
● Test new Atlas infra features (NVM)

35. © 2019 Fiverr Int. Lmt. All Rights Reserved. Proprietary & Confidential.
moshe.b@fiverr.com
sergei.m@fiverr.com
Thank You
We’re hiring :)