Fiverr https://www.fiverr.com/ is the largest marketplace for digital services in the world. Being a highly transactional platform, with a hyper growing organization, Fiverr's engineering team tackles some of the most interesting problems in scaling - both technical but also organizational. In this session, we will explore the evolution of Fiverr's architecture from the traditional Ruby on rails monolith to microservices, CQRS, event driven data integration using Apache Kafka and even further - the journey to micro apps. Alongside the technical discussion, we will also discuss team autonomy and how to create highly effective, decoupled, product engineering teams. We will deep dive into the different architectural decisions we made along the way, and share lessons learned - what worked and what didn't.

1. From ruby on rails to event
driven micro-apps
lessons learned from building microservices @ Fiverr

2. #indoerswetrust
Hi there!
● Alex Litvak
● 2 years ago - TLV --> NYC
● Fresh daddy
● 12 years of software
engineering
● Group manager @ Fiverr
● Past - fintech, IDF

3. #indoerswetrust
● Largest marketplace for
digital services (gigs) in
the world
● Service As a Product
● 100+ categories
● A new transaction every 5
seconds
● A new service (gig) is
created every 4 minutes
Who are we?

4. #indoerswetrust
Programming languages

5. #indoerswetrust
Persistence

6. #indoerswetrust
Messaging & Data integration

7. #indoerswetrust
Ops

8. #indoerswetrust
Autonomous
cross functional task forces
● ~15 Task Forces
● 2 engineering sites
● Loosely coupled, highly aligned

9. #indoerswetrust
Our Story of Building Microservices
From an architectural point of view

10. #indoerswetrust
Chapter 1: The Monolith

11. #indoerswetrust
● 2010
● RoR + MySQL +
memcached
● Fast bootstrapping
● Easy to test and
deploy
The monolith

12. #indoerswetrust
● Models lose cohesiveness
● Coupled, slowed down teams
● No Ownership
● Fear of deploy
● Long running tests
● No way to scale parts of the system
● More data -> slower queries ->
longer response times
But, the monolith “matures”...

13. #indoerswetrust
We needed a solution that can scale

14. #indoerswetrust

15. #indoerswetrust
Chapter 2: The Chimera

16. #indoerswetrust
Chimera: a fire-breathing she-monster in Greek mythology
having a lion's head, a goat's body, and a serpent's tail
A Fiverr microservice template consisting of
(1) a REST API endpoint for synchronous reads
(2) a RabbitMQ consumer for asynchronous writes
Residing in the same Repo, sharing the same business
logic code

17. #indoerswetrust
- Very popular general
purpose message
broker
- point to point,
request/reply and
pub-sub
communication styles
patterns
- smart broker / dumb
consumer model
RabbitMQ

18. #indoerswetrust
Reading Data

19. #indoerswetrust
Writing Data

20. #indoerswetrust
Other out of the box chimera “features”
- Various DB connectors
- Logging & Monitoring
- Write side fail safety

21. #indoerswetrust
- Read side failure - we can just refresh the
page….
- Write side failure - can have business
implications
Write Side Fail Safety

22. #indoerswetrust

23. #indoerswetrust
To requeue events:
Idempotency + one transaction

24. #indoerswetrust
First Services -
“Service for
every entity”
concept

25. #indoerswetrust
Services per entity
becomes a
bloated API
for every feature
and lose their
cohesiveness

26. #indoerswetrust
Services per entity are
services with blurry
boundaries and loose
team ownership

27. #indoerswetrust
So, How should have we split the monolith?

28. #indoerswetrust
Let’s talk about bounded contexts
- DDD deals with large
models by dividing
them into different
Bounded Contexts
- Boundaries are usually
defined by
- Context &
meaning
- Business
processes /
activities source: https://martinfowler.com/bliki/BoundedContext.html

29. #indoerswetrust
Moving to services based on business processes / activities.
Not based on Entities.

30. #indoerswetrust
Sometimes, easier said
than done...

31. #indoerswetrust
Well defined service boundaries require access to shared
data.

32. #indoerswetrust
And sometimes to data from a specific
domain...

33. #indoerswetrust
How would our services share data?

34. #indoerswetrust
- Coupling at the lowest level
- Altering same data
- Service not optimizing storage
for unique usage
- Data computation cannot scale
Share a database

35. #indoerswetrust
- ServiceA API changes due to
serviceB needs
- REST sucks as a query
language - no way to slice and
dice data
- “Do this!” instead of “something
happened”
- Data corruption over time
Pull data,
Push Commands

36. #indoerswetrust
- Producer and consumer are
decoupled
- Services become reactive
- Data can be sent in the payload of the
event (OrderCreated)
BUT
- If data is persisted in each service -
lacking single point of truth
- No history of data transported - no
way of reconstructing the data
Event Driven

37. #indoerswetrust
Commands -> Domain Events
StartSellerVacation -> SellerVacationStarted

38. #indoerswetrust

39. #indoerswetrust

40. #indoerswetrust
CQRS - Command Query Responsibility Segregation

41. #indoerswetrust
Reads > Writes
90% 10%

42. #indoerswetrust

43. #indoerswetrust

44. #indoerswetrust

45. #indoerswetrust
Solving transporting / sharing of data between services

46. #indoerswetrust
Building on top of our current event
driven architecture, we started looking
into centralized persisted stream of
events to be able to have
- One source of truth
- The ability to have a history of all
events and replay them if needed

47. #indoerswetrust
- designed for high volume
publish-subscribe messages
and streams
- provides a durable message
store
- stores streams of records in
categories called topics
- Dumb broker, smart
consumers
- Developed originally by
LinkedIn
Apache Kafka

48. #indoerswetrust
Not the best idea to consume data from
domain events...

49. #indoerswetrust
We decided to define a new type of event
A State Change Event

50. #indoerswetrust
Chapter 3: The Raven

51. #indoerswetrust
Ravens: Can fly great distances at speed, and are used by
the maesters of Westeros to pass messages between the
castles and cities of the Seven Kingdoms
A Fiverr State Change Event describing a change in the
attributes of an entity followed by a command to a chimera.

52. #indoerswetrust
● Fine Grained - describe the exact change that occurred, and only that
change.
○ GigUpdated - BAD
○ GigDescriptionUpdated - GOOD
○ UserOnVacationStatusUpdated - GOOD
○ OrderDeliveryTimeUpdated - GOOD
○ OrderUpdated - BAD
● Named in a past participle verb - it’s something that already happened
● Immutable
● Contains just the information about that specific change and nothing more
Ravens - Some Guidelines

53. #indoerswetrust
But what we currently have is a weakly typed,
decentralized messaging contract

54. #indoerswetrust
Raven -
A Strongly Typed
State Change
Event
implemented using Protocol Buffers
“Protocol buffers are Google's
language-neutral, platform-neutral,
extensible mechanism for
serializing structured data.

55. #indoerswetrust
Raven -
A Strongly Typed,
Centralized
State Change
Event
implemented using Protocol Buffers

56. #indoerswetrust
Aggregate - a cluster of domain objects that can be treated
as a unit. It consists of a root entity and possibly one or more
other associated entities and value objects.
(DDD / Eric Evans)

57. #indoerswetrust
Mapping our domain into Aggregates & Entities.

58. #indoerswetrust
message Metadata {
string message_type = 1;
string event_uid = 2;
string process_uid = 3;
int32 version = 4;
string aggregate_root_id = 5;
raven.aggregate.Root aggregate_root = 6;
raven.Entity related_entity_type = 7;
string related_entity_id = 8;
int32 created_at = 9;
raven.Operation operation = 10;
}
Describing Changes in our domain in a generic way using
aggregates & entities

59. #indoerswetrust
-
From sending domain events

60. #indoerswetrust
To event sourcing on the consuming chimera
{ id: 5 }
{id: 5, title: “cool gig”}
{id: 5, title: “cool gig”,
description: “bla bla…” }

61. #indoerswetrust
To event sourcing everywhere - events as source of truth
{ id: 5 }
{id: 5, title: “cool gig”}
{id: 5, title: “cool gig”,
description: “bla bla…” }
{ id: 5 }
{id: 5, title: “cool gig”}
{id: 5, title: “cool gig”,
description: “bla bla…” }

62. #indoerswetrust
- Storage + Messaging
- Highly scalable - data is immutable,
only append operations - O(1)
- The smarts are isolated inside
each bounded context.
- The Data is shared.
The Operation on the data is
Decentralized - inside the
bounded
context of the service.
- The distributed log becomes
Single point of truth.
Sharing data between Chimeras is solved!

63. #indoerswetrust
But What about Domain Events?

64. #indoerswetrust
Domain Events must be send via Kafka to maintain
ordering

65. #indoerswetrust
Problems solved - Transporting and Sharing Data,
Team autonomy on the service level
Remaining Problems - Rails is a big bowl of Read side mud,
some services are still serving as both read and write side, No
team ownership

66. #indoerswetrust
We had a new mission - Kill Ruby on Rails.

67. #indoerswetrust

68. #indoerswetrust
Chapter 4: The Perseus

69. #indoerswetrust
Perseus: the greatest Greek hero and slayer of monsters
before the days of Heracles
The Fiverr Slayer of Ruby on Rails - a NodeJS based
micro-frontend structured as a ReactJS SPA for a specific
vertical (gig page, order page, checkout page)

70. #indoerswetrust
LOSA - Thanks,
Lots Of Small Apps

71. #indoerswetrust
LOSA = Lots Of Small Apps
- Each microfrontend is a web application
encapsulating a UX vertical.
- Can be implemented and owned
independently.
- Reusable FE components implemented as
NPM packages.

72. #indoerswetrust
Are we still
constructing
the data for
the
presentation
layer as a
composition of
service calls?

73. #indoerswetrust
Service data composition -> complex caching

74. #indoerswetrust
Chapter 4: The Phoenix

75. #indoerswetrust
Phoenix: a unique bird that lived for five or six centuries in
the Arabian desert, after this time burning itself on a funeral
pyre and rising from the ashes with renewed youth to live
through another cycle.
A Fiverr presentation data aggregator - consuming Ravens
and building a read optimized projection of the data
consumed by Perseus to generate views.
Can renew its data view by replaying the Ravens.

76. #indoerswetrust
Event Sourcing works great with
CQRS!

77. #indoerswetrust
Read
Optimized
Views

78. #indoerswetrust

79. #indoerswetrust
V4 Backend Architecture Explained
The Full Picture

80. #indoerswetrust
● Only embark on this journey if you need to really SCALE
● Give teams full autonomy but have a dedicated Architect / committee that has
the full picture in mind
● Break the monolith by bounded contexts, not data
● Prefer async communication between services
● Less technical coupling <-> Less organizational coupling
● Domain events are great to propagate notifications, not data
Lessons Learned - Summary

81. #indoerswetrust
● Design for failure
● Most of the time, your system will be in the “in between” phase
● Think about your reading and writing patterns - design your system
accordingly
● Prefer strongly typed, well documented messaging contracts
● If you struggle with cache invalidations -> there is another way
● Event sourcing works great with CQRS and microservices integration, but
don’t event source everything.
Lessons Learned - Summary

82. #indoerswetrust
Stay in touch!
alex.l@fiverr.com
https://www.linkedin.com/in/alexlitvak/
http://engineering.fiverr.com - our engineering blog
We are hiring!
https://www.fiverr.com/jobs/offices/nyc/rnd

83. Thank You.