Your SlideShare is downloading. ×
Kill bottlenecks with gearman, sphinx, and memcached, Confoo 2011
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×

Saving this for later?

Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime - even offline.

Text the download link to your phone

Standard text messaging rates apply

Kill bottlenecks with gearman, sphinx, and memcached, Confoo 2011

1,451
views

Published on


0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
1,451
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
27
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Killing Bottlenecks paul reinheimer @preinheimer 1Thursday, March 10, 2011 1
  • 2. Goals for Today Horror Stories Introduce Tools Help you recognize problems Hints on implementation 2Thursday, March 10, 2011 2
  • 3. Provides a simple queue service Workers connect and identify work they can complete, clients hand off jobs when available Created by Danga Interactive (livejournal) 3Thursday, March 10, 2011 3
  • 4. Stores data in memory really really fast! Many servers can be used at once to provide a single large datastore Created by Danga Interactive (livejournal) 4Thursday, March 10, 2011 4
  • 5. Excellent search tool Import data from a variety of sources (commonly a database), index it quickly, then run searches 5Thursday, March 10, 2011 5
  • 6. Know Thyself 6Thursday, March 10, 2011 6
  • 7. Workers sign up with work they can do, many workers can each sign up with a different set of jobs. Clients connect to Gearman and pass off jobs, this is done asynchronously, they get to keep working Failed jobs are re-submitted a specified number of times. 7Thursday, March 10, 2011 7
  • 8. Gearman Workers sign up with work they can do, many workers can each sign up with a different set of jobs. Clients connect to Gearman and pass off jobs, this is done asynchronously, they get to keep working Failed jobs are re-submitted a specified number of times. 8Thursday, March 10, 2011 8
  • 9. Clients 9Thursday, March 10, 2011 9
  • 10. Workers 10Thursday, March 10, 2011 10
  • 11. Managing Workers 11Thursday, March 10, 2011 11
  • 12. Managing Workers 12Thursday, March 10, 2011 12
  • 13. Managing Workers 13Thursday, March 10, 2011 13
  • 14. Managing Workers 13Thursday, March 10, 2011 13
  • 15. Managing Workers 13Thursday, March 10, 2011 13
  • 16. Supervisor 14Thursday, March 10, 2011 14
  • 17. Gearman Alternatives RabitMQ Amazon Simple Queue Service ZeroMQ 15Thursday, March 10, 2011 15
  • 18. Stores data in memory really really fast! Many servers can be used at once to provide a single large datastore Created by Danga Interactive (livejournal) 16Thursday, March 10, 2011 16
  • 19. Promises 17Thursday, March 10, 2011 17
  • 20. Promises memcached doesnt lie, it just doesnt make promises 18Thursday, March 10, 2011 18
  • 21. Promises it makes a promise 19Thursday, March 10, 2011 19
  • 22. Promises data will no longer be available after it expires 20Thursday, March 10, 2011 20
  • 23. Promises When memcached doesnt have the information you need, get it elsewhere. Use a read-through caching callback or put it in your caching object. memcached is allowed to lose data! 21Thursday, March 10, 2011 21
  • 24. Consistent Hashing 22Thursday, March 10, 2011 22
  • 25. Hot Items = Hot Servers 23Thursday, March 10, 2011 23
  • 26. Hot Items = Hot Servers 23Thursday, March 10, 2011 23
  • 27. Memcached Alternatives redis Berkeley DB Tokyo Cabinet Tokyo Tyrant Voldemort Riak 24Thursday, March 10, 2011 24
  • 28. Excellent search tool Import data from a variety of sources (commonly a database), index it quickly, then run searches 25Thursday, March 10, 2011 25
  • 29. id title year rating 1 TMNT 1988 9.5 2 Goonies 1989 9.0 3 The Postman 1992 3.5 4 Alien 1994 8.5 26Thursday, March 10, 2011 26
  • 30. id title year rating studio category 1 TMNT 1988 9.5 x action 2 Goonies 1989 9.0 y adventure 3 The Postman 1992 3.5 z boring 4 Alien 1994 8.5 a awesome 27Thursday, March 10, 2011 27
  • 31. id title year rating studio category 1 TMNT 1988 9.5 x action 2 Goonies 1989 9.0 y adventure 3 The Postman 1992 3.5 z boring 4 Alien 1994 8.5 a awesome 28Thursday, March 10, 2011 28
  • 32. id director film_id tag_id 1 blue film_id description 1 2 2 red 1 Four awesome turtles... 2 3 3 green 2 Friends, pirates, goonies. 3 4 4 orange 3 Nap 4 5 4 They tried to turn around id title year rating studio category 1 TMNT 1988 9.5 x action 2 Goonies 1989 9.0 y adventure 3 The Postman 1992 3.5 z boring 4 Alien 1994 8.5 a awesome id tag 1 kids id producer id actor 1 1 2 awesome 1 Mel Gibson 2 2 3 sci-fi 2 Shredder 3 3 3 Sigourney Weaver 4 4 28Thursday, March 10, 2011 28
  • 33. id director film_id tag_id 1 blue film_id description 1 2 2 red 1 Four awesome turtles... 2 3 3 green 2 Friends, pirates, goonies. 3 4 4 orange 3 Nap 4 5 4 They tried to turn around id title year rating studio category 1 TMNT 1988 9.5 x action 2 Goonies 1989 9.0 y adventure 3 The Postman 1992 3.5 z boring 4 Alien 1994 8.5 a awesome id tag 1 kids id producer id actor 1 1 2 awesome 1 Mel Gibson 2 2 3 sci-fi 2 Shredder 3 3 3 Sigourney Weaver 4 4 29Thursday, March 10, 2011 29
  • 34. id director film_id tag_id 1 film_id blue director_id film_id description 1 film_id 2 tag_id 2 1 red 1 1 film_id desc_id Four awesome turtles... 2 1 3 1 3 2 green 2 2 Friends, pirates, goonies. 1 1 3 2 4 2 4 3 orange 3 3 2 Nap 2 4 3 5 3 4 4 4 They tried to turn around 3 3 4 4 4 4 id title year rating studio category 1 TMNT 1988 9.5 x action 2 Goonies 1989 9.0 y adventure 3 The Postman 1992 3.5 z boring 4 Alien 1994 8.5 a awesome id tag 1 id producer film_id kids tag_id id actor 1 film_id1 producer_id 2 1 awesome 1 1 film_id Mel Gibson actor_id 2 1 2 1 3 2 sci-fi 2 2 Shredder 1 1 3 2 3 2 3 3 3 2 Sigourney Weaver 2 4 3 4 3 4 4 3 3 4 4 4 4 29Thursday, March 10, 2011 29
  • 35. SELECT `dvd_scenes`.`scene_id`, `dvds`.`dvd_id`, `dvds`.`name`, `dvds`.`name` AS sort_title, `dvds`.`featured` AS premium, `dvds`.`description`, `dvds`.`date_released`, `dvds`.`channel_id`, `dvds`.`studio_id`, `dvd_scenes_dyn`.`times_viewed` AS sub_order, `dvd_scenes_dyn`.`rating` AS rating, `channels`.`name` AS channel_name, `studios`.`name` AS studio_name, `channels_keywords`.`keywords` AS channel_keywords, GROUP_CONCAT(DISTINCT `categories`.`name` ORDER BY `categories`.`name` DESC SEPARATOR , ) AS categories_list, GROUP_CONCAT(DISTINCT `models`.`name` ORDER BY `models`.`name` DESC SEPARATOR , ) AS models_list FROM `dvd_scenes` INNER JOIN dvds ON dvds.dvd_id = dvd_scenes.dvd_id INNER JOIN dvd_scenes_dyn ON dvd_scenes_dyn.id = dvd_scenes.scene_id LEFT JOIN scenes_categories ON dvd_scenes.scene_id = scenes_categories.scene_id LEFT JOIN categories ON `scenes_categories`.`category_id` = categories.category_id LEFT JOIN studios ON studios.studio_id = dvds.studio_id LEFT JOIN scenes_models ON scenes_models.scene_id = dvd_scenes.scene_id LEFT JOIN `channels` ON channels.channel_id = dvds.channel_id LEFT JOIN `channels_keywords` ON dvds.channel_id = channels_keywords.channel_id LEFT JOIN `models` ON scenes_models.model_id = models.model_id WHERE dvds.is_active = 1 AND dvd_scenes.is_active = 1 AND dvds.date_released < UNIX_TIMESTAMP() AND dvds.date_released > 0 AND dvds.channel_id NOT IN (21,26) 30 GROUP BY scene_idThursday, March 10, 2011 30
  • 36. sphinx http://www.flickr.com/photos/waldenpond/4237758518/ 31Thursday, March 10, 2011 31
  • 37. Sphinx Alternatives Lucene With Elastic Search solr 32Thursday, March 10, 2011 32
  • 38. Assigning Problems 33Thursday, March 10, 2011 33
  • 39. Appendix Paul Reinheimer - @preinheimer - http://blog.preinheimer.com Memcached - http://memcached.org/ supervisor - http://supervisord.org/ Sphinx - http://sphinxsearch.com/ Gearman - http://gearman.org/ 34Thursday, March 10, 2011 34
  • 40. 5 Stars! 35Thursday, March 10, 2011 35
  • 41. Feedback http://joind.in/2867 36Thursday, March 10, 2011 36