SF ElasticSearch Meetup - How HipChat Scaled to 1B Messages
Upcoming SlideShare
Loading in...5
×
 

Like this? Share it with your network

Share

SF ElasticSearch Meetup - How HipChat Scaled to 1B Messages

on

  • 1,392 views

HipChat Engineer, Zuhaib Siddique, presents how HipChat uses Elasticsearch in its infrastructure to provide near real-time search and chat history to the hundreds of thousands of active chatters using ...

HipChat Engineer, Zuhaib Siddique, presents how HipChat uses Elasticsearch in its infrastructure to provide near real-time search and chat history to the hundreds of thousands of active chatters using HipChat. Zuhaib covers cover of the issues the HipChat team has overcome using elasticsearch with AWS and CouchDB.

Statistics

Views

Total Views
1,392
Views on SlideShare
1,392
Embed Views
0

Actions

Likes
5
Downloads
15
Comments
2

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
  • hi
    Are you sure you want to
    Your message goes here
    Processing…
  • Greeting from me, First I will like to apologies for intruding into your privacy, since we don't know each other,my Name is Haret I saw your profile and i become interested to be your friend so Please i will like you to write back to me through my email address (haret.desmond@yahoo.com) i don't use to be online so that i will tell you more about my self
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

SF ElasticSearch Meetup - How HipChat Scaled to 1B Messages Presentation Transcript

  • 1. Wednesday, November 20, 13
  • 2. Heavy Lifting ZUHAIB SIDDIQUE Wednesday, November 20, 13 • @ZUHAIB • ENGINEER
  • 3. Wednesday, November 20, 13
  • 4. 1. Persistent chat rooms Wednesday, November 20, 13
  • 5. 2. File sharing & Chat history Wednesday, November 20, 13
  • 6. 3. @mentions + notifications Wednesday, November 20, 13
  • 7. 4. Integrations and API Wednesday, November 20, 13
  • 8. 5. Runs on everything Wednesday, November 20, 13
  • 9. 5. Runs on everything Wednesday, November 20, 13
  • 10. 5. Runs on everything Wednesday, November 20, 13
  • 11. Group chat and IM for teams. Wednesday, November 20, 13
  • 12. 1,000,000,000 1,200,000,000 messages sent Wednesday, November 20, 13
  • 13. What is a HipChat Message? { } Wednesday, November 20, 13 "id": "b0952da8-faa1-48f7-b1df-a8b2fb24bdfe", "from": { "nick": "Zuhaib Siddique", "group_id": 1, "user_id": 1 }, "room": { "group_id": 1, "name": "testing", "id": 1 }, "stanza_type": "muc_message", "date": "2013-11-14T23:20:13Z 424967", "stanza_data": { "body": "ElasticSearch FTW" }
  • 14. In the beginning there was … Wednesday, November 20, 13
  • 15. Then we added... https://github.com/rnewson/couchdb-lucene Wednesday, November 20, 13
  • 16. CouchDB + CouchDB-Lucene •Started to tip over at 200M Docs •m2.4xlarge Wednesday, November 20, 13
  • 17. CouchDB + CouchDB-Lucene m2.4xlarge m2.4xlarge Worked well for the first 500M Wednesday, November 20, 13
  • 18. m2.4xlarge Wednesday, November 20, 13 hs1.8xlarge
  • 19. But it still wasn’t big enough. Wednesday, November 20, 13
  • 20. Dont F*ck the Customer Wednesday, November 20, 13
  • 21. First Attempt at Scaling Wednesday, November 20, 13
  • 22. Wednesday, November 20, 13
  • 23. Why ElasticSearch for HipChat Wednesday, November 20, 13
  • 24. Why ElasticSearch for HipChat { "query_string" : { "default_field" : "content", "query" : "this AND that OR thus" } } Wednesday, November 20, 13
  • 25. Why ElasticSearch for HipChat Wednesday, November 20, 13
  • 26. Search Error Rate Wednesday, November 20, 13
  • 27. Wednesday, November 20, 13
  • 28. Don’t Forget about CouchDB AWS Max EBS size Wednesday, November 20, 13
  • 29. Stop the CouchDB bleeding Wednesday, November 20, 13
  • 30. CouchDB end of the road Wednesday, November 20, 13
  • 31. Wednesday, November 20, 13
  • 32. (allthethings) ElasticSearch ElasticSearch { "_index": "muc-2013.11", "_type": "document", "_id": "b0952da8-faa1-48f7-b1df-a8b2fb24bdfe", "_version": 1, "exists": true, "_source": { "from": { "nick": "Zuhaib Siddique", "group_id": 1, "user_id": 1 }, "room": { "group_id": 1, "name": "testing", "id": 1 }, "stanza_type": "muc_message", "date": "2013-11-14T23:20:13Z 424967", "_id": "b0952da8-faa1-48f7-b1df-a8b2fb24bdfe", "stanza_data": { "body": "ElasticSearch FTW" } } } Wednesday, November 20, 13 CouchDB { "id": "b0952da8-faa1-48f7-b1df-a8b2fb24bdfe", "from": { "nick": "Zuhaib Siddique", "group_id": 1, "user_id": 1 }, "room": { "group_id": 1, "name": "testing", "id": 1 }, "stanza_type": "muc_message", "date": "2013-11-14T23:20:13Z 424967", "stanza_data": { "body": "ElasticSearch FTW" } }
  • 33. (allthethings) ElasticSearch ElasticSearch { "_index": "muc-2013.11", "_type": "document", "_id": "b0952da8-faa1-48f7-b1df-a8b2fb24bdfe", "_version": 1, "exists": true, "_source": { "from": { "nick": "Zuhaib Siddique", "group_id": 1, "user_id": 1 }, "room": { "group_id": 1, "name": "testing", "id": 1 }, "stanza_type": "muc_message", "date": "2013-11-14T23:20:13Z 424967", "_id": "b0952da8-faa1-48f7-b1df-a8b2fb24bdfe", "stanza_data": { "body": "ElasticSearch FTW" } } } Wednesday, November 20, 13 CouchDB { "id": "b0952da8-faa1-48f7-b1df-a8b2fb24bdfe", "from": { "nick": "Zuhaib Siddique", "group_id": 1, "user_id": 1 }, "room": { "group_id": 1, "name": "testing", "id": 1 }, "stanza_type": "muc_message", "date": "2013-11-14T23:20:13Z 424967", "stanza_data": { "body": "ElasticSearch FTW" } }
  • 34. CouchDB vs ElasticSearch CouchDB Spike to 800ms! Avg 100ms with ES Wednesday, November 20, 13
  • 35. Running ElasticSearch HipChat Customer master Wednesday, November 20, 13 clients Thanks Philip O’Toole - Loggly
  • 36. $ fab create_esindex:hostname=esindex-d4 Wednesday, November 20, 13
  • 37. Behind the Scenes • Spin up EC2 Instance • Install Oracle Java 7 • Create Provision IOPs EBS • Install and configure ElasticSearch • ??? • Profit! Wednesday, November 20, 13
  • 38. # Elasticsearch nodes = partial_search(:node, "role:elasticsearch_server AND chef_environment:#{node.chef_environment}", :keys => { 'hostname' => [ 'hostname' ] }) if nodes.empty? Chef::Log.error "site.json - No Elasticsearch servers found" else es_nodes = nodes.map {|n| { 'host' => n['hostname'], 'port' => 9200 } } data["elasticsearch"] = { "connections" => es_nodes.sort_by {|n| n['host'] } } end Wednesday, November 20, 13
  • 39. site.json "elasticsearch": { "connections": [ { “host": "esindex-c1", "port": 9200 }, ... { "host": "esindex-e3", "port": 9200 } ] } Wednesday, November 20, 13
  • 40. You Wednesday, November 20, 13
  • 41. Running Elasticsearch Wednesday, November 20, 13
  • 42. Running ElasticSearch Single Node OOM! Wednesday, November 20, 13
  • 43. Monitor, Monitor, Monitor! Wednesday, November 20, 13
  • 44. Backup Wednesday, November 20, 13
  • 45. 100,000,000,000? To infinity and beyond! Wednesday, November 20, 13
  • 46. Questions? Wednesday, November 20, 13