Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Real Time Search at Yammer


Published on

Presentation on the architecture, scalability concerns, performance bottlenecks, operational characteristics and lessons learned while designing and implementing Yammer distributed real-time search system.

  • Be the first to comment

Real Time Search at Yammer

  1. 1. Realtime revolution at work REAL-TIME SEARCH AT YAMMER May 25, 2011 By Boris Aleksandrovsky Yammer, Inc.
  2. 2. <ul><li>Communication is hard, search is harder </li></ul><ul><ul><li>What me grammar? </li></ul></ul><ul><ul><li>Private language </li></ul></ul><ul><ul><li>Conversational language </li></ul></ul><ul><ul><li>Time compressed </li></ul></ul><ul><ul><li>Transient </li></ul></ul><ul><ul><li>Poorly organized </li></ul></ul><ul><ul><li>Authority is suspect </li></ul></ul><ul><ul><li>Social pressures </li></ul></ul>
  3. 3.
  4. 4. Challenges - From information to knowledge Information Facts Knowledge Attention Engagement Retention Messages Metadata Personalized Search
  5. 5. Agenda <ul><li>Background </li></ul><ul><li>Why search? </li></ul><ul><li>Indexing </li></ul><ul><li>Search </li></ul><ul><li>Tools and methodologies </li></ul><ul><li>Lessons learned </li></ul><ul><li>Future </li></ul><ul><li>Q&A </li></ul>
  6. 6. : Putting Social Media to Work <ul><li>Yammer makes work </li></ul><ul><li>Real-time, Social, Mobile </li></ul><ul><li>Collaborative, Contextual </li></ul><ul><li>More Human! </li></ul><ul><li>Similar to: </li></ul><ul><li>Facebook </li></ul><ul><li>Twitter </li></ul><ul><li>Wikis </li></ul><ul><li>Groups </li></ul>Knowledge Management: Document-oriented Enterprise Collaboration: Outcome-focused Social Media: People-centric
  7. 7. Yammer: The Enterprise Social Network <ul><li>Messaging and Feeds </li></ul><ul><li>Direct Messaging </li></ul><ul><li>User Profiles </li></ul><ul><li>Company Directory </li></ul><ul><li>Groups (Internal) </li></ul><ul><li>Communities (External) </li></ul><ul><li>File Sharing </li></ul><ul><li>Applications </li></ul><ul><li>Integrations </li></ul><ul><li>Web, Desktop, Mobile, Tablet </li></ul><ul><li>Translations </li></ul><ul><li>Network Consultation and Support </li></ul>Easy. Shared. Searchable. Real-time. Where your company’s knowledge lives.
  8. 8. 100,000+ companies, including 85% of the Fortune 500 – and growing.
  9. 9. What do you discuss at work, and with whom? <ul><li>Who do you need to communicate with, across the company? </li></ul><ul><li>How often are the same questions asked? </li></ul><ul><li>Who has the answers? Who has new ideas? Who can help? </li></ul>What do our employees think of our 401K program? Is everybody saving? What’s the latest with the XYZ account? What are our recommendations for financial and regulatory reform given the latest news about…? What will be discussed at our Quarterly Sales Kickoff? Where can I find out more about customer events here at the ABC conference? Who’s free to meet up? How can my team better prepare for our next product release? Who has any fresh ideas for… Who will I be working with on this new project?
  10. 10.
  11. 11. Search use case - Transient Awareness <ul><li>Reverse-chronological </li></ul><ul><li>Simple queries </li></ul><ul><li>Facet </li></ul><ul><ul><li>Date </li></ul></ul><ul><ul><li>Sender </li></ul></ul><ul><ul><li>Group </li></ul></ul>
  12. 12. Search use case - Knowledge Exploration <ul><li>Complicated relevance story </li></ul><ul><ul><li>tf/idf </li></ul></ul><ul><ul><li>popularity </li></ul></ul><ul><ul><li>engagement </li></ul></ul><ul><ul><li>social distance </li></ul></ul><ul><li>Complicated queries </li></ul><ul><li>Facet </li></ul><ul><ul><li>Date </li></ul></ul><ul><ul><li>Sender </li></ul></ul><ul><ul><li>Group </li></ul></ul><ul><ul><li>Object type </li></ul></ul>
  13. 13. Challenges for Yammer’s search engine <ul><li>More knowledge is generated in realtime </li></ul><ul><ul><li>Availability latency < 1 sec </li></ul></ul><ul><ul><li>Not always well formed </li></ul></ul><ul><li>Complicated relevance story </li></ul><ul><ul><li>experts and their reputation </li></ul></ul><ul><ul><li>popularity </li></ul></ul><ul><ul><li>social graph </li></ul></ul><ul><ul><li>tagging/topics </li></ul></ul><ul><ul><li>engagement signals </li></ul></ul><ul><ul><li>timeliness </li></ul></ul><ul><ul><li>location </li></ul></ul>
  14. 14. Team <ul><li>2 engineers </li></ul><ul><li>8 man months </li></ul><ul><li>Lots of fun </li></ul>
  15. 15. Indexing <ul><li>DB to replica </li></ul>
  16. 16.
  17. 17. Replication <ul><li>Independent near-replicas based on a single distributed source of truth </li></ul><ul><li>Can (will) get out of sync </li></ul><ul><li>Automatic monitoring of replication quality </li></ul><ul><ul><li>Are replicas out of sync with other replicas? </li></ul></ul><ul><ul><ul><li>number of docs </li></ul></ul></ul><ul><ul><ul><li>alert > X </li></ul></ul></ul><ul><ul><li>Are replicas out of sync with the DB? </li></ul></ul><ul><ul><ul><li>statistical sample of docs </li></ul></ul></ul>
  18. 18. Indexing <ul><li>In-replica to index </li></ul>
  19. 19. 30s
  20. 20. Why is it hard? <ul><li>No timeliness guarantee </li></ul><ul><li>Fragmentation </li></ul><ul><li>Out-of-order deliveries </li></ul><ul><li>Index dependencies </li></ul><ul><ul><li>Need to denormalize the information </li></ul></ul><ul><li>Need to build for network partition tolerance and redundancy </li></ul><ul><li>But </li></ul><ul><ul><li>Eventual consistency </li></ul></ul><ul><ul><li>Eventual delivery </li></ul></ul>
  21. 21. How do we cope? <ul><li>Out of order delivery source of (most) evil </li></ul><ul><li>? </li></ul><ul><ul><li>A) Assure in-order delivery </li></ul></ul><ul><ul><ul><li>buffer and wait </li></ul></ul></ul><ul><ul><ul><ul><li>degrades performance, availability and timeliness and is only very eventual consistent </li></ul></ul></ul></ul><ul><ul><li>B) Minimize probability and ignore </li></ul></ul><ul><ul><ul><li>timestamp precision </li></ul></ul></ul><ul><ul><ul><li>clock skew </li></ul></ul></ul><ul><ul><li>C) Arbitrate </li></ul></ul><ul><ul><ul><li>timestamp / vector clocks </li></ul></ul></ul><ul><ul><ul><li>semantics </li></ul></ul></ul><ul><ul><ul><li>need to index lifecycle events </li></ul></ul></ul><ul><li>Need to build for network partition tolerance and redundancy </li></ul><ul><li>But </li></ul><ul><ul><li>Consistency guarantee </li></ul></ul><ul><ul><li>Eventual delivery </li></ul></ul>
  22. 22. Delete-update race <ul><li>[create Message “hello” id=5 ts=12:34:39] </li></ul><ul><li>[delete Message “hello there” id=5 ts=12:45:01] </li></ul><ul><li>[modify Message “hello there” id=5 ts=12:45:01] </li></ul>id timestamp tombstone 5 12:34:39 no 5 12:45:01 yes
  23. 23. Multiple update race <ul><li>[create Message “hello” id=5 ts=12:34:39] </li></ul><ul><li>[modify Message “hello there now” id=5 ts=12:45:01] </li></ul><ul><li>[modify Message “hello there” id=5 ts=12:45:01] </li></ul>id timestamp text 5 12:34:39 hello 5 12:45:01 hello there now
  24. 24. Dupes <ul><li>[create Message “hello” id=5 ts=12:34:39] </li></ul><ul><li>[like Message id=5 userId=3 ts=12:45:01] </li></ul><ul><li>[like Message id=5 userId=3 ts=12:45:02] </li></ul><ul><li>[unlike Message id=5 userId=3 ts=12:45:04] </li></ul>id timestamp numLikes 5 12:34:39 0 5 12:45:01 1 5 12:45:02 1 5 12:45:04 0
  25. 25. Thread example
  26. 26. Zoie <ul><li>Realtime indexing system </li></ul><ul><li>Open sourced by LinkedIn </li></ul><ul><li>Used by LinkedIn in production for about 3 years </li></ul><ul><li>Deployed at dozen or so locations </li></ul><ul><li>Thanks Xiaoyang Gu, Yasuhiro Matsuda, John Wang and Lei Wang </li></ul>
  27. 27. Zoie <ul><li>Push events into buffer and the transaction log </li></ul><ul><li>Push buffer into Zoie </li></ul><ul><li>When Zoie commits, transaction log is truncated. </li></ul>
  28. 28. Indexing HA <ul><li>Cluster queue systems </li></ul><ul><ul><li>Round-robin of Rabbits introduce further out-of-order problems. </li></ul></ul><ul><li>Transaction log </li></ul><ul><ul><li>Between RabbiMQ dequeue and Zoie disk commit </li></ul></ul>
  29. 29. Dual indexing <ul><li>Primary for serving out </li></ul><ul><li>Secondary for reindexing </li></ul><ul><li>Verify secondary index consistency </li></ul><ul><li>foreach replica do </li></ul><ul><ul><li>shutdown </li></ul></ul><ul><ul><li>mv secondary to primary </li></ul></ul><ul><ul><li>restart </li></ul></ul><ul><li>Availability should not be affected except for slight chance of system failure </li></ul>
  30. 30. Index consistency problems <ul><li>Detect </li></ul><ul><ul><li>integrity check against the :source of truth: </li></ul></ul><ul><li>Reindex </li></ul><ul><ul><li>gaps </li></ul></ul><ul><ul><li>whole </li></ul></ul><ul><ul><li>reindex into secondary, swap with primary </li></ul></ul><ul><li>Repair </li></ul><ul><ul><li>patch in place </li></ul></ul><ul><ul><li>run on restart </li></ul></ul>
  31. 31. Search <ul><li><insert animated architecture slide> </li></ul>
  32. 32. Goal <ul><li>50/50-500/100 per partition </li></ul><ul><li>50M docs </li></ul><ul><li>50 msec P75 - 500 msec P99 </li></ul><ul><li>100 qps </li></ul>
  33. 33. REST-full API over HTTP <ul><li>,05242001 </li></ul>
  34. 34. Payload <ul><li>Payload is usually small json object </li></ul><ul><li>For security reasons only ids and scores are send out </li></ul><ul><li>One page (usually 10 items) x 6 index types. </li></ul>
  35. 35. Payload
  36. 36. Web Server <ul><li>Jersey over Jetty </li></ul><ul><li> </li></ul><ul><ul><li>Custom configuration </li></ul></ul><ul><ul><ul><li>tuned to the required 100 qps </li></ul></ul></ul><ul><ul><ul><li>generally impeccable, occasional lock contention </li></ul></ul></ul><ul><li> </li></ul><ul><ul><li>Annotation driven </li></ul></ul><ul><ul><li>Much easier to test </li></ul></ul>
  37. 37. Search master <ul><li>More like a router </li></ul><ul><li>Knows about partitioning scheme </li></ul><ul><li>Performs load normalization </li></ul><ul><ul><li>Call all, take the first </li></ul></ul><ul><ul><ul><li>Possible to use multicast </li></ul></ul></ul><ul><ul><li>Round Robin </li></ul></ul><ul><ul><ul><li>switch to for scale </li></ul></ul></ul><ul><ul><li>DLB (Least busy) </li></ul></ul><ul><li>Maintains primary SLA metrics </li></ul>
  38. 38. Partitioning <ul><li>Simple Jenkins 64bit hash of networkId </li></ul><ul><li>2 level hash to split large partitions </li></ul><ul><li>Exception list to split large partition </li></ul><ul><li>Limitation: Cannot partition inside a single network </li></ul><ul><li>Repartitioning story is expensive </li></ul><ul><li>Consistent hashing? </li></ul>
  39. 39. Testing <ul><li>Indexing </li></ul><ul><ul><li>Idempotent </li></ul></ul><ul><ul><li>Out-of-order delivery </li></ul></ul><ul><ul><li>Duplicate and incomplete docs tolerance </li></ul></ul><ul><ul><li>10K docs delivered in random order with X% of dupes and Y% incomplete records </li></ul></ul><ul><li>Search </li></ul><ul><ul><li>Small manual index by recording event </li></ul></ul><ul><ul><li>Unit style tests (testng) with Asserts </li></ul></ul>
  40. 40. Production <ul><li>Measure </li></ul><ul><li>Hardware is cheap, people are not </li></ul><ul><ul><li>People require more maintenance </li></ul></ul><ul><li>Have enough redundancy </li></ul>
  41. 41. Metrics <ul><li>JVM, Queue, Logging and Configuration </li></ul>
  42. 42. Metrics <ul><li>Gauges </li></ul>
  43. 43. Metrics <ul><li>Meters </li></ul>
  44. 44. Metrics <ul><li>Timers </li></ul>
  45. 45. Metrics <ul><li> </li></ul>
  46. 46. Lessons <ul><li>Do not underestimate your data model </li></ul><ul><li>Tradeoff between consistency, RT availability and correctness </li></ul><ul><li>Measure </li></ul><ul><li>Flexible partitioning scheme </li></ul><ul><li>Data recovery plan </li></ul>
  47. 47. Future <ul><li>Dynamic routing </li></ul><ul><ul><li>Zookeeper </li></ul></ul><ul><li>Partition rebalancing </li></ul><ul><li>Multiple sub-partitions with different SLAs </li></ul><ul><li>Work on relevancy </li></ul><ul><li>Multiple languages </li></ul><ul><li>Document parsing </li></ul><ul><li>External data </li></ul><ul><li>Scala </li></ul>
  48. 48. Q&A Session: What’s On Your Mind?