Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Scaling Slack - The Good, the Unexpected, and the Road Ahead

126 views

Published on

Video and slides synchronized, mp3 and slide download available at URL https://bit.ly/2BJ8u7F.

Mike Demmer talks about the major changes that Slack has made to the service architecture to meet the needs for larger and larger enterprise customers. Demmer presents 3 of these changes: decomposition of the real-time message service, client-side lazy loading via edge caching, and scaling the primary data storage tier with fine-grained horizontal sharding using Vitess. Filmed at qconsf.com.

Mike Demmer is a member of Slack's Infrastructure Engineering team, where he works on hard problems of scalability and reliability and leads the development of Slack's next generation database architecture. Previously, he was co-founder and CTO of Jut - a startup applying a new dataflow language to observability for developers and operations engineers.

Published in: Technology
  • Be the first to comment

Scaling Slack - The Good, the Unexpected, and the Road Ahead

  1. 1. Michael Demmer November 6, 2018 Scaling Slack The Good, The Unexpected, and The Road Ahead mdemmer@slack-corp.com | @mjdemmer
  2. 2. InfoQ.com: News & Community Site • Over 1,000,000 software developers, architects and CTOs read the site world- wide every month • 250,000 senior developers subscribe to our weekly newsletter • Published in 4 languages (English, Chinese, Japanese and Brazilian Portuguese) • Post content from our QCon conferences • 2 dedicated podcast channels: The InfoQ Podcast, with a focus on Architecture and The Engineering Culture Podcast, with a focus on building • 96 deep dives on innovative topics packed as downloadable emags and minibooks • Over 40 new content items per week Watch the video with slide synchronization on InfoQ.com! https://www.infoq.com/presentations/ slack-scalability-2018
  3. 3. Purpose of QCon - to empower software development by facilitating the spread of knowledge and innovation Strategy - practitioner-driven conference designed for YOU: influencers of change and innovation in your teams - speakers and topics driving the evolution and innovation - connecting and catalyzing the influencers and innovators Highlights - attended by more than 12,000 delegates since 2007 - held in 9 cities worldwide Presented at QCon San Francisco www.qconsf.com
  4. 4. Me
  5. 5. (Not) This Talk 1. 2016: Monolith 2. 2016-2018: Microservices 3. 2016-2018: Best Practices 4. 2018: Lessons Learned
  6. 6. This Talk 1. 2016: How Slack Worked 2. 2016-2018: Things Got More Interesting 3. 2016-2018: What We Did About It 4. 2018+: Themes and Road Ahead
  7. 7. Slack in 2016
  8. 8. Slack
  9. 9. Workspaces, Channels, Users, and more Duff Beer Oceanic Airlines Delos A workspace logically contains all channels and messages, as well as users, emoji, bots, and more. All interactions occur within the workspace boundary. us_east_1 Acme Corp #brainstorming #proj-roadrunner #marketing … @alice @bob @carol ...
  10. 10. User Base 4M Daily Active Users Largest Organizations >10,000 Active Users Connectivity 2.5M peak simultaneous connected Avg 10 hrs/day Engineering Style Conservative, Pragmatic, Minimal Most systems > 10 year old technology Slack Facts (2016)
  11. 11. us_east_1 How Slack Works (2016) RTM ServiceRTM ServiceMessage Server (Java) WebappWebappWebapp (PHP) RTM Service RTM Service Message Proxy us_west_1 Client Websocket HTTP API Calls Job Queue MySQL MySQL
  12. 12. Client / Server Flow Initial login: ● Download full workspace model with all channels, users, emoji, etc. ● Establish real time websocket Webapp (PHP) Message Proxy 1: rtm.start 2: prefs: {...}, users: {...}, channels: {...}, emoji: {...}, ms: “ms1.slack-msgs.com” 3: websocket connect
  13. 13. Client / Server Flow Initial login: ● Download full workspace model with all channels, users, emoji, etc. ● Establish real time websocket While connected: ● Push updates via websocket ● API calls for channel history, message edits, create channels, etc. Webapp (PHP) Message Proxy reactions.add {message: ...}
  14. 14. Sharding And Routing Workspace Sharding ● Assign a workspace to a DB and MS shard at creation ● Metadata table lookup for each API request to route Mains RTM Service RTM Service Message Servers MySQL Shards Webapp (PHP) select * from teams where id=1234 {id:1234, domain:demmer, db_shard:35, ms_shard:11, ...}
  15. 15. Sharding And Routing Workspace Sharding ● Assign a workspace to a DB and MS shard at creation ● Metadata table lookup for each API request to route “Herd of Pets” ● DBs run in active/active pairs with application failover ● Service hosts are addressed in config and manually replaced Mains RTM Service RTM Service Message Servers MySQL Shards Webapp (PHP)
  16. 16. Server Experience Implementation model is straightforward, easy to reason about and debug. ● All operations are workspace scoped ● Horizontally scale by adding servers ● Few components or dependencies Why This Worked Client Experience Data model lends itself to a seamless, rich real-time client experience. ● Full data model available in memory ● Updates appear instantly ● Everything feels real time
  17. 17. Things Get More Interesting...
  18. 18. Things Get More Interesting Product Model Size and Scale
  19. 19. Slack Growth
  20. 20. User Base >8M Daily Active Users Largest Organizations >125,000 Active Users Connectivity >7M peak simultaneous connected Avg 10 hrs/day Engineering Style Still pragmatic, but embrace complexity where needed to solve hardest problems Slack Facts (2018)
  21. 21. User Base >8M Daily Active Users Largest Organizations >125,000 Active Users Connectivity >7M peak simultaneous connected Avg 10 hrs/day Engineering Style Still pragmatic, but embrace complexity where needed to solve hardest problems Slack Facts (2018) 2x 10x ! 3x
  22. 22. Change the Model Duff Beer Oceanic Airlines Delos A workspace logically contains all channels and messages, as well as users, emoji, bots, and more. All interactions occur within the workspace boundary. us_east_1 Acme Corp #brainstorming #proj-roadrunner #marketing … @alice @bob @carol ...
  23. 23. Change the Model Acme Corp Duff Beer Oceanic Airlines Delos Wayne Enterprises Wayne Shipping Wayne Finance Wayne Security EnterpriseWorkspaces
  24. 24. Change the Model Acme Corp Duff Beer Oceanic Airlines Agents of SHIELD Stark Industries Delos Wayne Enterprises Wayne Shipping Wayne Finance Wayne Security Shared ChannelsWorkspaces Enterprise
  25. 25. Challenges Recurring Issues ● Large organizations: Boot metadata download is slow and expensive ● Thundering Herd: Load to connect >> Load in steady state ● Hot spots: Overwhelm database hosts (mains and shards) and other systems ● Herd of Pets: Manual operation to replace specific servers ● Cross Workspace Channels: Need to change assumptions about partitioning
  26. 26. So What Did We Do?
  27. 27. What Did We Do Message Services Service Decomposition Vitess Fine-Grained DB Sharding Thin Client Model Flannel Cache
  28. 28. What Did We Do Thin Client Model Flannel Cache
  29. 29. Challenge: Boot Model Explosion boot_payload_size ~= (num_users * user_profile_bytes) + (num_channels * (channel_info_size + (num_users_in_channel * user_id bytes))) Users Profiles Channels Total 12 6 KB 1 KB 7 KB 530 140 KB 28 KB 168 KB 4,008 5 MB 2 MB 7 MB
  30. 30. Challenge: Boot Model Explosion boot_payload_size ~= (num_users * user_profile_bytes) + (num_channels * (channel_info_size + (num_users_in_channel * user_id bytes))) Users Profiles Channels Total 12 6 KB 1 KB 7 KB 530 140 KB 28 KB 168 KB 4,008 5 MB 2 MB 7 MB 44,030 36 MB 25 MB 59 MB 148,170 78 MB 40 MB 118 MB
  31. 31. us_east_1 Thin Client Model RTM ServiceRTM ServiceMessage Server WebappWebapp Webapp RTM Service RTM Service Message Proxy Client Websocket HTTP API Calls Job Queue MySQL MySQL us_west_1
  32. 32. RTM Service RTM Service Flannel Cache us_west_1 us_east_1 Thin Client Model RTM ServiceRTM ServiceMessage Server WebappWebapp Webapp RTM Service RTM Service Message Proxy us_west_1 Client Websocket HTTP API Calls Job Queue MySQL MySQL Consul
  33. 33. Thin Client Model RTM Service RTM ServiceFlannel Flannel Service Globally distributed edge cache Minimize Workspace Model Much smaller boot payload Routing Workspace affinity for cache locality Query API Fetch unknown objects from cache Cache Updates Proxy subscription messages to clients Websocket
  34. 34. Thin Client Model Unblock Large Organizations Adapting clients to a lazy load model was a critical change to enable Slack for large organizations. ● Huge reduction in payload times on initial connect ● Flannel efficiently responds to > 1+ million queries per second ● Adds challenges of cache coherency and reconciling business logic
  35. 35. What Did We Do Vitess Fine-Grained DB Sharding
  36. 36. Challenge: Hot Spots & Manual Repair
  37. 37. RTM Service RTM Service Flannel Cache us_west_1 us_east_1 Vitess RTM ServiceRTM ServiceMessage Server WebappWebapp Webapp RTM Service RTM Service Message Proxy us_west_1 Client Websocket HTTP API Calls Job Queue MySQL MySQL Consul
  38. 38. RTM Service RTM Service us_west_1 us_east_1 Vitess RTM ServiceRTM ServiceMessage Server WebappWebapp Webapp RTM Service RTM Service Message Proxy us_west_1 Client Websocket HTTP API Calls Job Queue MySQL MySQL VtTablet MySQL VtGateVtGateVtGate Flannel Cache Consul
  39. 39. Vitess VtTablet MySQL VtGateVtGateVtGate Flexible Sharding Vitess manages per-table sharding policy Topology Management Database servers self-register Single Master Using GTID and semi-sync replication Failover Orchestrator promotes a replica on failover Resharding Workflows Automatically expand the cluster WebappWebappWebapp
  40. 40. Vitess Fine-Grained Sharding Migrating to a channel-sharded / user-sharded data model helps mitigate hot spots for large teams and thundering herds. ● Retains MySQL at the core for developer / operations continuity ● More mature topology management and cluster expansion systems ● Data migrations that change the sharding model take a long time
  41. 41. What Did We Do Message Services Service Decomposition
  42. 42. Challenge: Shared Channels? Agents of SHIELD Stark Industries Message Server Message Server
  43. 43. Challenge: Shared Channels? Agents of SHIELD Stark Industries Message Server Message Server
  44. 44. RTM Service RTM Service Flannel Cache us_west_1 us_east_1 Message Server to Services RTM ServiceRTM ServiceMessage Server WebappWebapp Webapp RTM Service RTM Service Message Proxy us_west_1 Client Websocket HTTP API Calls Job Queue MySQL MySQL VtTablet MySQL VtGateVtGateVtGate Consul
  45. 45. us_east_1 Message Server to Services Client WebappWebapp Webapp Job Queue VtTablet MySQL VtGateVtGateVtGate RTM Service RTM Service Channel Server RTM Service RTM Service Gateway Server RTM Service RTM Service Presence Server RTM Service RTM Service Message Server VtGateVtGateAdmin Server RTM Service RTM Service us_west_1 Consul MySQL MySQL Flannel Cache Websocket HTTP API Calls
  46. 46. Gateway Server Websocket termination and subscriptions Admin Server Cluster management and routing Presence Server Store and distribute presence state Channel Server Pub/Sub fanout with 5 minute buffering Message Server to Services (Legacy) Message Server Used for reminders, Google Calendar integration Channel Server Gateway Server Presence Server Message Server Admin Server
  47. 47. Message Server to Services Generic Messaging Services Everything is a pub/sub “channel”, including message channels as well as workspace / user metadata channels. ● Clients / Flannel subscribes to updates for all relevant objects ● Each Message Service has dedicated clear roles and responsibilities ● Self-healing cluster orchestration to maintain availability ● Each user session now depends on many more servers being available
  48. 48. What Did We Do Message Services Service Decomposition Vitess Fine-Grained DB Sharding Lazy Client Flannel Cache
  49. 49. Some Themes...
  50. 50. Topology Management For each of these projects (and more), architecture evolved from hand-configured server hostnames to a discovery mesh. ● Enables self-registration and automatic cluster repair ● Adds reliance on service discovery infrastructure (consul) ● Led to changes in service ownership and on-call rotation Herd of Pets to Service Mesh
  51. 51. Scatter May Be Harmful Fine-Grained Sharding Migrating from a workspace-scope to channel or user scoped spreads out the load but adds a requirement to sometimes scatter/gather. ● Removes artificial couplings on back end systems ● Teams are less isolated, so need extra protections from noisy neighbors ● When scattering, clients should tolerate partial results and retry ● Tail latencies can dominate performance when fetching from many
  52. 52. Deprecation Challenges As hard as it is to add new services into production under load, it’s proven as hard if not harder to remove old ones. ● With few exceptions, all 2016 services still in production ● Need to support legacy clients and integrations ● Data migrations need application changes takes time Deploying Is Only The Beginning
  53. 53. Performance Short Game Architectural rework is necessary, but less glamorous performance optimizations pay huge dividends ● Simple approaches to caching or refactoring ● Client-side jitter to spread out load ● Eliminate unnecessary methods / queries Grinding It Out
  54. 54. us_east_1 How Slack Works (2016) RTM ServiceRTM ServiceMessage Server (Java) WebappWebappWebapp (PHP) RTM Service RTM Service Message Proxy us_west_1 Client Websocket HTTP API Calls Job Queue MySQL MySQL
  55. 55. us_east_1 How Slack Works (2018) Client WebappWebapp Webapp Job Queue VtTablet MySQL VtGateVtGateVtGate RTM Service RTM Service Channel Server RTM Service RTM Service Gateway Server RTM Service RTM Service Presence Server RTM Service RTM Service Message Server VtGateVtGateAdmin Server RTM Service RTM Service us_west_1 Consul MySQL MySQL Flannel Cache Websocket HTTP API Calls
  56. 56. We’re Not Done Yet Storage POPs Geographically distributed back end Services Services Services Decompose the monolith and improve service mesh. Job Queue Revamp the asynchronous task queue Resiliency Degraded functionality when subsystems are unavailable Eventual Consistency Change API expectations Network Scale Stay ahead of the growth curve
  57. 57. Thank You! 55
  58. 58. BACKUP
  59. 59. us_east_1 How Slack Works (c 2018) Client Websocket HTTP API Calls WebappWebapp Webapp Job Queue VtTablet MySQL VtGateVtGateVtGate RTM Service RTM Service Channel Server RTM Service RTM Service Gateway Server RTM Service RTM Service Presence Server RTM Service RTM Service Message Server VtGateVtGateAdmin Server RTM Service RTM Service us_west_1 Consul MySQL MySQL Flannel Cache
  60. 60. Client Connections Websocket termination, user / connection state and subscriptions Webapp Actions Communication/routing from Webapp → Message Server for channel messages Presence Indications User presence state, updates & presence subscriptions - that little green indicator Subscriptions and Fanout Last 5 minutes of history, as well as initial subscription and fanout of messages Message Server Scheduled Messages Used for reminders, Google Calendar integration RTM Service RTM ServiceMessage Server (Java)
  61. 61. Team Sharded MySQL Team Sharding Application-defined sharding policy routes all queries to the team shard Manual Topology Management Operator-managed host configuration is injected into application code Active Master / Master Both sides are writable masters, biases for availability with best-effort consistency Application Retry Failover If preferred side is unavailable, connect to the backup side and try again Split Shards Manually orchestrated switchover to divide some teams to new host. MySQL MySQL WebappWebappWebapp
  62. 62. QCon 2016 QCon 2017
  63. 63. Watch the video with slide synchronization on InfoQ.com! https://www.infoq.com/presentations/ slack-scalability-2018

×