Your SlideShare is downloading. ×

SF ElasticSearch Meetup - How HipChat Scaled to 1B Messages

1,039

Published on

HipChat Engineer, Zuhaib Siddique, presents how HipChat uses Elasticsearch in its infrastructure to provide near real-time search and chat history to the hundreds of thousands of active chatters using …

HipChat Engineer, Zuhaib Siddique, presents how HipChat uses Elasticsearch in its infrastructure to provide near real-time search and chat history to the hundreds of thousands of active chatters using HipChat. Zuhaib covers cover of the issues the HipChat team has overcome using elasticsearch with AWS and CouchDB.

Published in: Technology, Business
2 Comments
5 Likes
Statistics
Notes
  • hi
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Greeting from me, First I will like to apologies for intruding into your privacy, since we don't know each other,my Name is Haret I saw your profile and i become interested to be your friend so Please i will like you to write back to me through my email address (haret.desmond@yahoo.com) i don't use to be online so that i will tell you more about my self
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Views
Total Views
1,039
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
21
Comments
2
Likes
5
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Wednesday, November 20, 13
  • 2. Heavy Lifting ZUHAIB SIDDIQUE Wednesday, November 20, 13 • @ZUHAIB • ENGINEER
  • 3. Wednesday, November 20, 13
  • 4. 1. Persistent chat rooms Wednesday, November 20, 13
  • 5. 2. File sharing & Chat history Wednesday, November 20, 13
  • 6. 3. @mentions + notifications Wednesday, November 20, 13
  • 7. 4. Integrations and API Wednesday, November 20, 13
  • 8. 5. Runs on everything Wednesday, November 20, 13
  • 9. 5. Runs on everything Wednesday, November 20, 13
  • 10. 5. Runs on everything Wednesday, November 20, 13
  • 11. Group chat and IM for teams. Wednesday, November 20, 13
  • 12. 1,000,000,000 1,200,000,000 messages sent Wednesday, November 20, 13
  • 13. What is a HipChat Message? { } Wednesday, November 20, 13 "id": "b0952da8-faa1-48f7-b1df-a8b2fb24bdfe", "from": { "nick": "Zuhaib Siddique", "group_id": 1, "user_id": 1 }, "room": { "group_id": 1, "name": "testing", "id": 1 }, "stanza_type": "muc_message", "date": "2013-11-14T23:20:13Z 424967", "stanza_data": { "body": "ElasticSearch FTW" }
  • 14. In the beginning there was … Wednesday, November 20, 13
  • 15. Then we added... https://github.com/rnewson/couchdb-lucene Wednesday, November 20, 13
  • 16. CouchDB + CouchDB-Lucene •Started to tip over at 200M Docs •m2.4xlarge Wednesday, November 20, 13
  • 17. CouchDB + CouchDB-Lucene m2.4xlarge m2.4xlarge Worked well for the first 500M Wednesday, November 20, 13
  • 18. m2.4xlarge Wednesday, November 20, 13 hs1.8xlarge
  • 19. But it still wasn’t big enough. Wednesday, November 20, 13
  • 20. Dont F*ck the Customer Wednesday, November 20, 13
  • 21. First Attempt at Scaling Wednesday, November 20, 13
  • 22. Wednesday, November 20, 13
  • 23. Why ElasticSearch for HipChat Wednesday, November 20, 13
  • 24. Why ElasticSearch for HipChat { "query_string" : { "default_field" : "content", "query" : "this AND that OR thus" } } Wednesday, November 20, 13
  • 25. Why ElasticSearch for HipChat Wednesday, November 20, 13
  • 26. Search Error Rate Wednesday, November 20, 13
  • 27. Wednesday, November 20, 13
  • 28. Don’t Forget about CouchDB AWS Max EBS size Wednesday, November 20, 13
  • 29. Stop the CouchDB bleeding Wednesday, November 20, 13
  • 30. CouchDB end of the road Wednesday, November 20, 13
  • 31. Wednesday, November 20, 13
  • 32. (allthethings) ElasticSearch ElasticSearch { "_index": "muc-2013.11", "_type": "document", "_id": "b0952da8-faa1-48f7-b1df-a8b2fb24bdfe", "_version": 1, "exists": true, "_source": { "from": { "nick": "Zuhaib Siddique", "group_id": 1, "user_id": 1 }, "room": { "group_id": 1, "name": "testing", "id": 1 }, "stanza_type": "muc_message", "date": "2013-11-14T23:20:13Z 424967", "_id": "b0952da8-faa1-48f7-b1df-a8b2fb24bdfe", "stanza_data": { "body": "ElasticSearch FTW" } } } Wednesday, November 20, 13 CouchDB { "id": "b0952da8-faa1-48f7-b1df-a8b2fb24bdfe", "from": { "nick": "Zuhaib Siddique", "group_id": 1, "user_id": 1 }, "room": { "group_id": 1, "name": "testing", "id": 1 }, "stanza_type": "muc_message", "date": "2013-11-14T23:20:13Z 424967", "stanza_data": { "body": "ElasticSearch FTW" } }
  • 33. (allthethings) ElasticSearch ElasticSearch { "_index": "muc-2013.11", "_type": "document", "_id": "b0952da8-faa1-48f7-b1df-a8b2fb24bdfe", "_version": 1, "exists": true, "_source": { "from": { "nick": "Zuhaib Siddique", "group_id": 1, "user_id": 1 }, "room": { "group_id": 1, "name": "testing", "id": 1 }, "stanza_type": "muc_message", "date": "2013-11-14T23:20:13Z 424967", "_id": "b0952da8-faa1-48f7-b1df-a8b2fb24bdfe", "stanza_data": { "body": "ElasticSearch FTW" } } } Wednesday, November 20, 13 CouchDB { "id": "b0952da8-faa1-48f7-b1df-a8b2fb24bdfe", "from": { "nick": "Zuhaib Siddique", "group_id": 1, "user_id": 1 }, "room": { "group_id": 1, "name": "testing", "id": 1 }, "stanza_type": "muc_message", "date": "2013-11-14T23:20:13Z 424967", "stanza_data": { "body": "ElasticSearch FTW" } }
  • 34. CouchDB vs ElasticSearch CouchDB Spike to 800ms! Avg 100ms with ES Wednesday, November 20, 13
  • 35. Running ElasticSearch HipChat Customer master Wednesday, November 20, 13 clients Thanks Philip O’Toole - Loggly
  • 36. $ fab create_esindex:hostname=esindex-d4 Wednesday, November 20, 13
  • 37. Behind the Scenes • Spin up EC2 Instance • Install Oracle Java 7 • Create Provision IOPs EBS • Install and configure ElasticSearch • ??? • Profit! Wednesday, November 20, 13
  • 38. # Elasticsearch nodes = partial_search(:node, "role:elasticsearch_server AND chef_environment:#{node.chef_environment}", :keys => { 'hostname' => [ 'hostname' ] }) if nodes.empty? Chef::Log.error "site.json - No Elasticsearch servers found" else es_nodes = nodes.map {|n| { 'host' => n['hostname'], 'port' => 9200 } } data["elasticsearch"] = { "connections" => es_nodes.sort_by {|n| n['host'] } } end Wednesday, November 20, 13
  • 39. site.json "elasticsearch": { "connections": [ { “host": "esindex-c1", "port": 9200 }, ... { "host": "esindex-e3", "port": 9200 } ] } Wednesday, November 20, 13
  • 40. You Wednesday, November 20, 13
  • 41. Running Elasticsearch Wednesday, November 20, 13
  • 42. Running ElasticSearch Single Node OOM! Wednesday, November 20, 13
  • 43. Monitor, Monitor, Monitor! Wednesday, November 20, 13
  • 44. Backup Wednesday, November 20, 13
  • 45. 100,000,000,000? To infinity and beyond! Wednesday, November 20, 13
  • 46. Questions? Wednesday, November 20, 13

×