Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Concourse - near real time notifications platform at linkedin

131 views

Published on

Slides from our talk at the Streams processing meetup (https://www.meetup.com/Stream-Processing-Meetup-LinkedIn/) about Concourse. Concourse is a scalable near real time content generation and fanout platform which is used to power multiple notifications at LinkedIn. Concourse supports fanout based on a chosen graph and/or other rule based targeting logic and each notification candidate is scored and filtered to generate the most relevant notifications.

Published in: Engineering
  • Be the first to comment

  • Be the first to like this

Concourse - near real time notifications platform at linkedin

  1. 1. Concourse : Near real time notifications platform at Linkedin ​Jeff Weiner ​Chief Executive Officer ​Ajith Muralidharan ​Staff Engineer, Data/Relevance ​LinkedIn ​Vivek Nelamangala ​Staff Engineer, Data/ Relevance-infra ​ LinkedIn 1
  2. 2. Outline Background Nearline non-transactional notifications Concourse Nuts and Bolts Performance optimization Results 2
  3. 3. Outline Background Nearline non-transactional notifications Concourse Nuts and Bolts Performance optimization Results 3
  4. 4. Notifications at Linkedin • Communicate timely and relevant information to our members • Bring users to LinkedIn to engage with content/events that they can’t miss • Our mission : Deliver right content to the right people at right time via the right channel (email, push and in-app) for all members. Jeff Weiner, is in the news: “The top CEOs of 2017” Ajith Muralidharan has sent a connection invite 4
  5. 5. App based notifications at Linkedin • UI push • Badge update In both cases, the notifications can be found in the relevant tab on the app 5
  6. 6. M2M (Invites, Messages, etc.) Air Traffic Controller (ATC) Concourse Near real-time generation Beehive Offline generation Misc. In-app Tab badge App badge UI push Email Notifications Service Flock LinkedIn’s Notification Ecosystem
  7. 7. Outline Background Nearline non-transactional notifications Concourse Nuts and Bolts Performance optimization Results 7
  8. 8. Nearline non-transactional notifications Proactively reach right audience with highly relevant content in a timely manner, so that LinkedIn members stay well-informed and easily engaged with professional communities to boost their professional goals. 8 Examples : Activity based notifications, job change, daily news
  9. 9. Non-transactional notifications • keep members informed about their network • surface LinkedIn’s value to the members • make content contributors feel heard • move recipients through the engagement funnel Benefits Challenges • irrelevant notifications will annoy members • disablement / unsubscription / complaint • bad reputation for LinkedIn 9
  10. 10. Nearline non-transactional notifications Vivek creates a LinkedIn post Vivek’s connections receive a notification Example Use-Case Activity based Notification 10
  11. 11. Benefits of nearline processing Legacy offline workflow ○ 4 hour time delay to notify a user. ○ Limited targeting capability - can only scale to a subset of the targeting. Nearline processing is required to generate a snappy member experience. 11
  12. 12. Outline Background Nearline non-transactional notifications Concourse Nuts and Bolts Performance optimization Results 12
  13. 13. Nearline non-transactional notifications 1. Vivek creates a LinkedIn post 2. Concourse fans out content to eligible recipients: connections, followers 4. Concourse prunes low-quality candidates based on score 3. Concourse scores each candidate 5. Remaining candidates trigger notifications to ATC, where they are scored again (second pass scoring) and optimized (aggregation, capping, etc.) Concourse 1ATC 13 1 1
  14. 14. 14 Samza Workflow Kafka Stream Rest API
  15. 15. Workflow manager (Repartitioner) 15 Samza Workflow Kafka Stream
  16. 16. Fanout 16 Fanout *** Recipient Stream 1 To Scorer Recipient Stream 2 Recipient Stream N Update data Fanout data Targeting attributes Actor partitioned * Recipient partitioned** Connections Graph Follows Graph Other Graphs
  17. 17. Scorer 17
  18. 18. Features processor Features- Processor Aggregated Feature Stream (**) Feature Stream N Feature Stream 2 Feature Stream 1 Data Transfor- mation HDFS Dataset 1 Data Transfor- mation HDFS Dataset 2 Data Transfor- mation Input Kafka Stream Scorer 18 Samza Workflow Kafka Stream Recipient Partitioned Hadoop Workflow **
  19. 19. Relevance support ● Access item/actor features from existing rest endpoints ● Member/edge features ingested into rocksDB ● Data collection for model training ● Scoring and filtering ○ Support for single & multi utility models. 19
  20. 20. Concourse as a platform Onboarding process ● Integrate new content type in the workflow manager ● Fetch fanout/targeting from existing/new external REST endpoints ● Onboard new item features ● Reuse existing member features and/or onboard new ones ● Specify and onboard relevance model, collect training data and iterate on models. 20
  21. 21. Outline Background Nearline non-transactional notifications Concourse Nuts and Bolts Performance optimization Results 21
  22. 22. Scale ● 500K notifications scored per second ● ~2000 notifications sent per second ● 34 Billion feature records (2TB) ingested into Samza local stores ● 1024 samza tasks ● 60 seconds P90 delay 22
  23. 23. Scale Optimization Feature optimization ● Strongly typed schema 4x size reduction ● Delta feature push with a smoothed full feature push ● Rate Limiting to reduce kafka load 23
  24. 24. Scale Optimization Samza native optimizations ● Host-affinity in samza for faster deploys ● Dedicated samza cluster ● Samza Local State to make high QPS scoring possible ● Distributed traffic across data centers ● Side inputs feature to minimize state footprint 24
  25. 25. Outline Background Nearline non-transactional notifications Concourse Nuts and Bolts Performance optimization Results 25
  26. 26. Offline to nearine migrations Activity-based notifications ● Increased engagement/sessions and a p90 delay of a few seconds (from 4 hours) ● unlocked much larger additional gains from expanded fanout and enhanced targeting Job change ● Increase in engagement and response rates and a p90 delay of a few seconds (from 10 hours) 26
  27. 27. Concourse as a platform Hosts growing set of use cases ○ Activity based notifications ○ Job change notifications ○ Notifications about conversations on your posts ○ Group activity notifications ○ ... 27
  28. 28. Thank you 28 Blog post : https://engineering.linkedin.com/blog/2018/05/concourse--generating-personalized-content-notifications-in-near

×