0
Wednesday, November 20, 13
Heavy Lifting

ZUHAIB SIDDIQUE

Wednesday, November 20, 13

•

@ZUHAIB

•

ENGINEER
Wednesday, November 20, 13
1. Persistent chat rooms

Wednesday, November 20, 13
2. File sharing & Chat history

Wednesday, November 20, 13
3. @mentions + notifications

Wednesday, November 20, 13
4. Integrations and API

Wednesday, November 20, 13
5. Runs on everything

Wednesday, November 20, 13
5. Runs on everything

Wednesday, November 20, 13
5. Runs on everything

Wednesday, November 20, 13
Group chat and IM
for teams.

Wednesday, November 20, 13
1,000,000,000
1,200,000,000
messages sent
Wednesday, November 20, 13
What is a HipChat Message?
{

}
Wednesday, November 20, 13

"id": "b0952da8-faa1-48f7-b1df-a8b2fb24bdfe",
"from": {
"nick"...
In the beginning there was …

Wednesday, November 20, 13
Then we added...

https://github.com/rnewson/couchdb-lucene
Wednesday, November 20, 13
CouchDB + CouchDB-Lucene
•Started to tip over at
200M Docs

•m2.4xlarge

Wednesday, November 20, 13
CouchDB + CouchDB-Lucene

m2.4xlarge

m2.4xlarge

Worked well for the first 500M
Wednesday, November 20, 13
m2.4xlarge
Wednesday, November 20, 13

hs1.8xlarge
But it still wasn’t big enough.

Wednesday, November 20, 13
Dont F*ck the Customer

Wednesday, November 20, 13
First Attempt at Scaling

Wednesday, November 20, 13
Wednesday, November 20, 13
Why ElasticSearch for HipChat

Wednesday, November 20, 13
Why ElasticSearch for HipChat
{
"query_string" : {
"default_field" : "content",
"query" : "this AND that OR thus"
}
}

Wed...
Why ElasticSearch for HipChat

Wednesday, November 20, 13
Search Error Rate

Wednesday, November 20, 13
Wednesday, November 20, 13
Don’t Forget about CouchDB

AWS Max
EBS size
Wednesday, November 20, 13
Stop the CouchDB bleeding

Wednesday, November 20, 13
CouchDB end of the road

Wednesday, November 20, 13
Wednesday, November 20, 13
(allthethings) ElasticSearch
ElasticSearch
{
"_index": "muc-2013.11",
"_type": "document",
"_id": "b0952da8-faa1-48f7-b1df...
(allthethings) ElasticSearch
ElasticSearch
{
"_index": "muc-2013.11",
"_type": "document",
"_id": "b0952da8-faa1-48f7-b1df...
CouchDB vs ElasticSearch
CouchDB Spike
to 800ms!

Avg 100ms with ES

Wednesday, November 20, 13
Running ElasticSearch

HipChat Customer

master
Wednesday, November 20, 13

clients

Thanks Philip O’Toole - Loggly
$ fab create_esindex:hostname=esindex-d4

Wednesday, November 20, 13
Behind the Scenes
• Spin up EC2 Instance
• Install Oracle Java 7
• Create Provision IOPs EBS
• Install and configure Elast...
# Elasticsearch
nodes = partial_search(:node, "role:elasticsearch_server AND
chef_environment:#{node.chef_environment}",
:...
site.json
"elasticsearch": {
"connections": [
{
“host": "esindex-c1",
"port": 9200
},
...
{
"host": "esindex-e3",
"port": ...
You

Wednesday, November 20, 13
Running Elasticsearch

Wednesday, November 20, 13
Running ElasticSearch
Single Node OOM!

Wednesday, November 20, 13
Monitor, Monitor, Monitor!

Wednesday, November 20, 13
Backup

Wednesday, November 20, 13
100,000,000,000?
To infinity and beyond!

Wednesday, November 20, 13
Questions?

Wednesday, November 20, 13
Upcoming SlideShare
Loading in...5
×

SF ElasticSearch Meetup - How HipChat Scaled to 1B Messages

1,066

Published on

HipChat Engineer, Zuhaib Siddique, presents how HipChat uses Elasticsearch in its infrastructure to provide near real-time search and chat history to the hundreds of thousands of active chatters using HipChat. Zuhaib covers cover of the issues the HipChat team has overcome using elasticsearch with AWS and CouchDB.

Published in: Technology, Business
2 Comments
5 Likes
Statistics
Notes
  • hi
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Greeting from me, First I will like to apologies for intruding into your privacy, since we don't know each other,my Name is Haret I saw your profile and i become interested to be your friend so Please i will like you to write back to me through my email address (haret.desmond@yahoo.com) i don't use to be online so that i will tell you more about my self
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Views
Total Views
1,066
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
21
Comments
2
Likes
5
Embeds 0
No embeds

No notes for slide

Transcript of "SF ElasticSearch Meetup - How HipChat Scaled to 1B Messages"

  1. 1. Wednesday, November 20, 13
  2. 2. Heavy Lifting ZUHAIB SIDDIQUE Wednesday, November 20, 13 • @ZUHAIB • ENGINEER
  3. 3. Wednesday, November 20, 13
  4. 4. 1. Persistent chat rooms Wednesday, November 20, 13
  5. 5. 2. File sharing & Chat history Wednesday, November 20, 13
  6. 6. 3. @mentions + notifications Wednesday, November 20, 13
  7. 7. 4. Integrations and API Wednesday, November 20, 13
  8. 8. 5. Runs on everything Wednesday, November 20, 13
  9. 9. 5. Runs on everything Wednesday, November 20, 13
  10. 10. 5. Runs on everything Wednesday, November 20, 13
  11. 11. Group chat and IM for teams. Wednesday, November 20, 13
  12. 12. 1,000,000,000 1,200,000,000 messages sent Wednesday, November 20, 13
  13. 13. What is a HipChat Message? { } Wednesday, November 20, 13 "id": "b0952da8-faa1-48f7-b1df-a8b2fb24bdfe", "from": { "nick": "Zuhaib Siddique", "group_id": 1, "user_id": 1 }, "room": { "group_id": 1, "name": "testing", "id": 1 }, "stanza_type": "muc_message", "date": "2013-11-14T23:20:13Z 424967", "stanza_data": { "body": "ElasticSearch FTW" }
  14. 14. In the beginning there was … Wednesday, November 20, 13
  15. 15. Then we added... https://github.com/rnewson/couchdb-lucene Wednesday, November 20, 13
  16. 16. CouchDB + CouchDB-Lucene •Started to tip over at 200M Docs •m2.4xlarge Wednesday, November 20, 13
  17. 17. CouchDB + CouchDB-Lucene m2.4xlarge m2.4xlarge Worked well for the first 500M Wednesday, November 20, 13
  18. 18. m2.4xlarge Wednesday, November 20, 13 hs1.8xlarge
  19. 19. But it still wasn’t big enough. Wednesday, November 20, 13
  20. 20. Dont F*ck the Customer Wednesday, November 20, 13
  21. 21. First Attempt at Scaling Wednesday, November 20, 13
  22. 22. Wednesday, November 20, 13
  23. 23. Why ElasticSearch for HipChat Wednesday, November 20, 13
  24. 24. Why ElasticSearch for HipChat { "query_string" : { "default_field" : "content", "query" : "this AND that OR thus" } } Wednesday, November 20, 13
  25. 25. Why ElasticSearch for HipChat Wednesday, November 20, 13
  26. 26. Search Error Rate Wednesday, November 20, 13
  27. 27. Wednesday, November 20, 13
  28. 28. Don’t Forget about CouchDB AWS Max EBS size Wednesday, November 20, 13
  29. 29. Stop the CouchDB bleeding Wednesday, November 20, 13
  30. 30. CouchDB end of the road Wednesday, November 20, 13
  31. 31. Wednesday, November 20, 13
  32. 32. (allthethings) ElasticSearch ElasticSearch { "_index": "muc-2013.11", "_type": "document", "_id": "b0952da8-faa1-48f7-b1df-a8b2fb24bdfe", "_version": 1, "exists": true, "_source": { "from": { "nick": "Zuhaib Siddique", "group_id": 1, "user_id": 1 }, "room": { "group_id": 1, "name": "testing", "id": 1 }, "stanza_type": "muc_message", "date": "2013-11-14T23:20:13Z 424967", "_id": "b0952da8-faa1-48f7-b1df-a8b2fb24bdfe", "stanza_data": { "body": "ElasticSearch FTW" } } } Wednesday, November 20, 13 CouchDB { "id": "b0952da8-faa1-48f7-b1df-a8b2fb24bdfe", "from": { "nick": "Zuhaib Siddique", "group_id": 1, "user_id": 1 }, "room": { "group_id": 1, "name": "testing", "id": 1 }, "stanza_type": "muc_message", "date": "2013-11-14T23:20:13Z 424967", "stanza_data": { "body": "ElasticSearch FTW" } }
  33. 33. (allthethings) ElasticSearch ElasticSearch { "_index": "muc-2013.11", "_type": "document", "_id": "b0952da8-faa1-48f7-b1df-a8b2fb24bdfe", "_version": 1, "exists": true, "_source": { "from": { "nick": "Zuhaib Siddique", "group_id": 1, "user_id": 1 }, "room": { "group_id": 1, "name": "testing", "id": 1 }, "stanza_type": "muc_message", "date": "2013-11-14T23:20:13Z 424967", "_id": "b0952da8-faa1-48f7-b1df-a8b2fb24bdfe", "stanza_data": { "body": "ElasticSearch FTW" } } } Wednesday, November 20, 13 CouchDB { "id": "b0952da8-faa1-48f7-b1df-a8b2fb24bdfe", "from": { "nick": "Zuhaib Siddique", "group_id": 1, "user_id": 1 }, "room": { "group_id": 1, "name": "testing", "id": 1 }, "stanza_type": "muc_message", "date": "2013-11-14T23:20:13Z 424967", "stanza_data": { "body": "ElasticSearch FTW" } }
  34. 34. CouchDB vs ElasticSearch CouchDB Spike to 800ms! Avg 100ms with ES Wednesday, November 20, 13
  35. 35. Running ElasticSearch HipChat Customer master Wednesday, November 20, 13 clients Thanks Philip O’Toole - Loggly
  36. 36. $ fab create_esindex:hostname=esindex-d4 Wednesday, November 20, 13
  37. 37. Behind the Scenes • Spin up EC2 Instance • Install Oracle Java 7 • Create Provision IOPs EBS • Install and configure ElasticSearch • ??? • Profit! Wednesday, November 20, 13
  38. 38. # Elasticsearch nodes = partial_search(:node, "role:elasticsearch_server AND chef_environment:#{node.chef_environment}", :keys => { 'hostname' => [ 'hostname' ] }) if nodes.empty? Chef::Log.error "site.json - No Elasticsearch servers found" else es_nodes = nodes.map {|n| { 'host' => n['hostname'], 'port' => 9200 } } data["elasticsearch"] = { "connections" => es_nodes.sort_by {|n| n['host'] } } end Wednesday, November 20, 13
  39. 39. site.json "elasticsearch": { "connections": [ { “host": "esindex-c1", "port": 9200 }, ... { "host": "esindex-e3", "port": 9200 } ] } Wednesday, November 20, 13
  40. 40. You Wednesday, November 20, 13
  41. 41. Running Elasticsearch Wednesday, November 20, 13
  42. 42. Running ElasticSearch Single Node OOM! Wednesday, November 20, 13
  43. 43. Monitor, Monitor, Monitor! Wednesday, November 20, 13
  44. 44. Backup Wednesday, November 20, 13
  45. 45. 100,000,000,000? To infinity and beyond! Wednesday, November 20, 13
  46. 46. Questions? Wednesday, November 20, 13
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×