Mark Dunphy discusses the evolution of the Activity Feed system at Behance from MongoDB to Cassandra. The initial MongoDB implementation worked well for the smaller user base in 2011 but struggled with node failures, disk fragmentation, and downtime as the system scaled. Cassandra was chosen to replace it due to its scalability, performance, and ease of maintenance. Dunphy details the process of learning Cassandra, modeling the data, and developing a strategy for writes and reads that allowed the system to handle thousands of writes and reads per second while serving the Activity Feed to users.

1. Building an Activity
Feed with Cassandra
Mark Dunphy, Software Engineer
Behance/Adobe
@dunphtastic

2. Disclaimer
Not an operations person.
Will pretend to be one for the purpose of this talk.

3. Quick Overview
What is the Behance Activity Feed?

5. • Actions
• Comments, Appreciations, Etc
• Entities
• Projects, Works in Progress
• Actors
• Users

6. Project Entity
Actions taken
by actors

7. Activity Fan Out

8. User A publishes
a new project
Write to Follower A’s feed
Write to Follower B’s feed
Write to Follower C’s feed
Write to Follower D’s feed

9. Now that that’s over…

10. MongoDB
2011

11. • Smaller user base (~340,000).
• Built very quickly. Worked well at the time.
• Not well researched.

12. Fast forward to 2014

13. • Frequent node failures
• Heavy disk fragmentation caused by deletes
• Slow reads from disk. Started storing in RAM.
• Primary -> Secondary caused downtime for
some.
• Scaled out vertically and horizontally.

14. Why Cassandra?

15. • Riak
• Very close. Community seemed lacking.
• Redis
• No native cluster. Too much maintenance.
• Memcached/MySQL
• Too much complex app logic.

16. Cassandra Wins.

17. • Fantastic community. #cassandra on Twitter
• Easy to read documentation
• Linearly scalable. Easy to grow cluster.
• Low maintenance overhead for ops team.
• Handles time series data very well.

18. Learning

19. • Cassandra Summit 2014
• Other team in Adobe
• Long nights reading documentation

20. Our Data

21. • Ephemeral
• “Source of truth” lives in a MySQL database
• Okay with *some* data loss

22. Our Rules

23. • User’s feed is comprised of entities with one set
of actions
• User’s feed only contains one of any given entity
• An entity’s set of actions contains up to seven of
the most recent actions taken by that user’s
network

24. Planning

25. Language Support
• Most services on Behance are PHP
• No official Datastax PHP driver

26. –Mark Dunphy, 2014
“Looks like I’m learning python.”

27. Go to Production
No, nothing is working yet. I didn’t skip a slide.

28. • App/cluster in production before anything works
• Test real life load
• Fail spectacularly without anybody noticing
• Deploy risky changes without fear
• Run alongside MongoDB

29. January 19th, 2015

30. Query Patterns
• “Create your data models based on the queries
you want to run” - Basically Everybody
• Wanted to…
• Read a user’s feed entities by type and time of
most recent action…separately.
• Write/Update a user’s feed entities with new
actions while knowing only user id and entity id

31. Data Models

32. –Mark Dunphy, January 2015
“An UPDATE in Cassandra works like an
UPSERT! Let’s store the user’s entire feed in a
single row in a table! It’s so simple!”
First Data Model

33. CREATE TYPE activity.action (
created_on timestamp,
secondary_entity_id int,
actor_id int,
verb_id int
);
CREATE TYPE activity.entity (
entity_type_id int,
entity_id int
);

34. CREATE TABLE activity.project_actions (
modified_on timestamp,
entity_id int,
user_id int,
actions list<frozen<action>>,
PRIMARY KEY(user_id, entity_id)
)

35. CREATE TABLE activity.feeds (
modified_entities list<frozen<entity>>,
modified_on timestamp,
project_ids list<int>,
user_id int,
wip_revision_ids list<int>,
PRIMARY KEY(user_id)
)

36. First Data Model

37. First Data Model
Moments Before Everything Exploded

38. –Mark Dunphy, January 2015
“Okay let’s keep nearly the same model, but
use INSERT and DELETE instead of always
UPDATE. Just use batch statements.”
Second Data Model

39. Second Data Model
This was also a very very bad idea.

40. • Lose the benefit of Cassandra being distributed
• All queries go through the same coordinator
which puts a lot of stress and responsibility on
one node.
• Use concurrency and prepared statements
instead. Datastax drivers make this easy.
Second Data Model

41. Second Data Model
Oops

42. Okay…

43. Now we’ve got it.

44. Winning Data Model

45. CREATE TYPE activity.action (
created_on timestamp,
secondary_entity_id int,
actor_id int,
verb_id int
);

46. CREATE TABLE activity.projects (
created_on timestamp,
user_id int,
entity_id int,
actions list<frozen<action>>,
PRIMARY KEY(user_id, created_on, entity_id)
)

47. CREATE TABLE activity.project_actions (
modified_on timestamp,
entity_id int,
user_id int,
actions list<frozen<action>>,
PRIMARY KEY(user_id, entity_id)
)

48. Much Nicer

49. Write Strategy
• “User A comments on Project A. User B follows
User A.”
• Request out to add the comment action to User
B’s feed
• Read existing actions for that entity (Project A) in
B’s feed. Push new action on top.
• Write new actions list into new “row” in projects
table

50. Read Strategy
• SELECT * FROM projects WHERE user_id
= 123 AND created_on > 123214373
• Optimized for quick/easy reads. More important
that a user’s feed loads quickly than it updating
quickly.
• Use timestamp to “page” through data.

51. Lessons Learned
• Duplicate your data to achieve desired queries.
Storage is cheap. Writes are cheap.
• Think outside the box. Cassandra is not
relational.
• Never ever ever ignore inserts/deletes in favor of
an update only workflow. Never. It is literally
insane.

52. Final Specs
• 16 node cluster on AWS EC2 c3.8xlarge
• Mix of SizeTieredCompactionStrategy and
DateTieredCompactionStrategy
• NetworkTopologyStrategy
• Replication factor 3
• ConsistencyLevel = ONE for most requests

53. Final Specs
• Bursty write volume. Consistent read volume.
• 5k to 80k writes per second
• 2k to 4k reads per second

54. Questions?
I might have answers.

55. Thank you!
Mark Dunphy, Software Engineer
Behance/Adobe
@dunphtastic