0
Killing Bottlenecks         paul reinheimer        @preinheimer   1Thursday, March 10, 2011                           1
Goals for Today                        Horror Stories                           Introduce Tools                           ...
Provides a simple queue service                           Workers connect and identify work they can complete, clients    ...
Stores data in memory                            really really fast!                           Many servers can be used at...
Excellent search tool                           Import data from a variety of sources (commonly a database),              ...
Know Thyself                                          6Thursday, March 10, 2011                      6
Workers sign up with work they can do, many workers can each                           sign up with a different set of job...
Gearman                           Workers sign up with work they can do, many workers can each                           s...
Clients                            9Thursday, March 10, 2011        9
Workers                            10Thursday, March 10, 2011         10
Managing Workers                                     11Thursday, March 10, 2011                  11
Managing Workers                                     12Thursday, March 10, 2011                  12
Managing Workers                                     13Thursday, March 10, 2011                  13
Managing Workers                                     13Thursday, March 10, 2011                  13
Managing Workers                                     13Thursday, March 10, 2011                  13
Supervisor                               14Thursday, March 10, 2011            14
Gearman Alternatives                           RabitMQ                           Amazon Simple Queue Service              ...
Stores data in memory                            really really fast!                           Many servers can be used at...
Promises                             17Thursday, March 10, 2011          17
Promises                           memcached doesnt lie, it just doesnt                                   make promises   ...
Promises                             it makes a promise                                                  19Thursday, March...
Promises                data will no longer be available after it expires                                                 ...
Promises                           When memcached doesnt have the                           information you need, get it e...
Consistent Hashing                                       22Thursday, March 10, 2011                    22
Hot Items = Hot Servers                                            23Thursday, March 10, 2011                         23
Hot Items = Hot Servers                                            23Thursday, March 10, 2011                         23
Memcached Alternatives                           redis                           Berkeley DB                           Tok...
Excellent search tool                           Import data from a variety of sources (commonly a database),              ...
id      title      year   rating                           1      TMNT        1988    9.5                           2     ...
id        title      year   rating   studio   category                     1       TMNT        1988    9.5       x       a...
id        title      year   rating   studio   category                     1       TMNT        1988    9.5       x       a...
id          director         film_id                tag_id              1                 blue                             ...
id          director         film_id                tag_id              1                 blue                             ...
id              director         film_id      tag_id                                   1    film_id          blue           ...
SELECT          `dvd_scenes`.`scene_id`,          `dvds`.`dvd_id`,          `dvds`.`name`,          `dvds`.`name`         ...
sphinx        http://www.flickr.com/photos/waldenpond/4237758518/            31Thursday, March 10, 2011                    ...
Sphinx Alternatives                           Lucene                            With Elastic Search                       ...
Assigning Problems                                       33Thursday, March 10, 2011                    33
Appendix                           Paul Reinheimer - @preinheimer - http://blog.preinheimer.com                           ...
5 Stars!                                      35Thursday, March 10, 2011                   35
Feedback                             http://joind.in/2867                                                    36Thursday, M...
Upcoming SlideShare
Loading in...5
×

Kill bottlenecks with gearman, sphinx, and memcached, Confoo 2011

1,495

Published on

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
1,495
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
28
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Transcript of "Kill bottlenecks with gearman, sphinx, and memcached, Confoo 2011"

  1. 1. Killing Bottlenecks paul reinheimer @preinheimer 1Thursday, March 10, 2011 1
  2. 2. Goals for Today Horror Stories Introduce Tools Help you recognize problems Hints on implementation 2Thursday, March 10, 2011 2
  3. 3. Provides a simple queue service Workers connect and identify work they can complete, clients hand off jobs when available Created by Danga Interactive (livejournal) 3Thursday, March 10, 2011 3
  4. 4. Stores data in memory really really fast! Many servers can be used at once to provide a single large datastore Created by Danga Interactive (livejournal) 4Thursday, March 10, 2011 4
  5. 5. Excellent search tool Import data from a variety of sources (commonly a database), index it quickly, then run searches 5Thursday, March 10, 2011 5
  6. 6. Know Thyself 6Thursday, March 10, 2011 6
  7. 7. Workers sign up with work they can do, many workers can each sign up with a different set of jobs. Clients connect to Gearman and pass off jobs, this is done asynchronously, they get to keep working Failed jobs are re-submitted a specified number of times. 7Thursday, March 10, 2011 7
  8. 8. Gearman Workers sign up with work they can do, many workers can each sign up with a different set of jobs. Clients connect to Gearman and pass off jobs, this is done asynchronously, they get to keep working Failed jobs are re-submitted a specified number of times. 8Thursday, March 10, 2011 8
  9. 9. Clients 9Thursday, March 10, 2011 9
  10. 10. Workers 10Thursday, March 10, 2011 10
  11. 11. Managing Workers 11Thursday, March 10, 2011 11
  12. 12. Managing Workers 12Thursday, March 10, 2011 12
  13. 13. Managing Workers 13Thursday, March 10, 2011 13
  14. 14. Managing Workers 13Thursday, March 10, 2011 13
  15. 15. Managing Workers 13Thursday, March 10, 2011 13
  16. 16. Supervisor 14Thursday, March 10, 2011 14
  17. 17. Gearman Alternatives RabitMQ Amazon Simple Queue Service ZeroMQ 15Thursday, March 10, 2011 15
  18. 18. Stores data in memory really really fast! Many servers can be used at once to provide a single large datastore Created by Danga Interactive (livejournal) 16Thursday, March 10, 2011 16
  19. 19. Promises 17Thursday, March 10, 2011 17
  20. 20. Promises memcached doesnt lie, it just doesnt make promises 18Thursday, March 10, 2011 18
  21. 21. Promises it makes a promise 19Thursday, March 10, 2011 19
  22. 22. Promises data will no longer be available after it expires 20Thursday, March 10, 2011 20
  23. 23. Promises When memcached doesnt have the information you need, get it elsewhere. Use a read-through caching callback or put it in your caching object. memcached is allowed to lose data! 21Thursday, March 10, 2011 21
  24. 24. Consistent Hashing 22Thursday, March 10, 2011 22
  25. 25. Hot Items = Hot Servers 23Thursday, March 10, 2011 23
  26. 26. Hot Items = Hot Servers 23Thursday, March 10, 2011 23
  27. 27. Memcached Alternatives redis Berkeley DB Tokyo Cabinet Tokyo Tyrant Voldemort Riak 24Thursday, March 10, 2011 24
  28. 28. Excellent search tool Import data from a variety of sources (commonly a database), index it quickly, then run searches 25Thursday, March 10, 2011 25
  29. 29. id title year rating 1 TMNT 1988 9.5 2 Goonies 1989 9.0 3 The Postman 1992 3.5 4 Alien 1994 8.5 26Thursday, March 10, 2011 26
  30. 30. id title year rating studio category 1 TMNT 1988 9.5 x action 2 Goonies 1989 9.0 y adventure 3 The Postman 1992 3.5 z boring 4 Alien 1994 8.5 a awesome 27Thursday, March 10, 2011 27
  31. 31. id title year rating studio category 1 TMNT 1988 9.5 x action 2 Goonies 1989 9.0 y adventure 3 The Postman 1992 3.5 z boring 4 Alien 1994 8.5 a awesome 28Thursday, March 10, 2011 28
  32. 32. id director film_id tag_id 1 blue film_id description 1 2 2 red 1 Four awesome turtles... 2 3 3 green 2 Friends, pirates, goonies. 3 4 4 orange 3 Nap 4 5 4 They tried to turn around id title year rating studio category 1 TMNT 1988 9.5 x action 2 Goonies 1989 9.0 y adventure 3 The Postman 1992 3.5 z boring 4 Alien 1994 8.5 a awesome id tag 1 kids id producer id actor 1 1 2 awesome 1 Mel Gibson 2 2 3 sci-fi 2 Shredder 3 3 3 Sigourney Weaver 4 4 28Thursday, March 10, 2011 28
  33. 33. id director film_id tag_id 1 blue film_id description 1 2 2 red 1 Four awesome turtles... 2 3 3 green 2 Friends, pirates, goonies. 3 4 4 orange 3 Nap 4 5 4 They tried to turn around id title year rating studio category 1 TMNT 1988 9.5 x action 2 Goonies 1989 9.0 y adventure 3 The Postman 1992 3.5 z boring 4 Alien 1994 8.5 a awesome id tag 1 kids id producer id actor 1 1 2 awesome 1 Mel Gibson 2 2 3 sci-fi 2 Shredder 3 3 3 Sigourney Weaver 4 4 29Thursday, March 10, 2011 29
  34. 34. id director film_id tag_id 1 film_id blue director_id film_id description 1 film_id 2 tag_id 2 1 red 1 1 film_id desc_id Four awesome turtles... 2 1 3 1 3 2 green 2 2 Friends, pirates, goonies. 1 1 3 2 4 2 4 3 orange 3 3 2 Nap 2 4 3 5 3 4 4 4 They tried to turn around 3 3 4 4 4 4 id title year rating studio category 1 TMNT 1988 9.5 x action 2 Goonies 1989 9.0 y adventure 3 The Postman 1992 3.5 z boring 4 Alien 1994 8.5 a awesome id tag 1 id producer film_id kids tag_id id actor 1 film_id1 producer_id 2 1 awesome 1 1 film_id Mel Gibson actor_id 2 1 2 1 3 2 sci-fi 2 2 Shredder 1 1 3 2 3 2 3 3 3 2 Sigourney Weaver 2 4 3 4 3 4 4 3 3 4 4 4 4 29Thursday, March 10, 2011 29
  35. 35. SELECT `dvd_scenes`.`scene_id`, `dvds`.`dvd_id`, `dvds`.`name`, `dvds`.`name` AS sort_title, `dvds`.`featured` AS premium, `dvds`.`description`, `dvds`.`date_released`, `dvds`.`channel_id`, `dvds`.`studio_id`, `dvd_scenes_dyn`.`times_viewed` AS sub_order, `dvd_scenes_dyn`.`rating` AS rating, `channels`.`name` AS channel_name, `studios`.`name` AS studio_name, `channels_keywords`.`keywords` AS channel_keywords, GROUP_CONCAT(DISTINCT `categories`.`name` ORDER BY `categories`.`name` DESC SEPARATOR , ) AS categories_list, GROUP_CONCAT(DISTINCT `models`.`name` ORDER BY `models`.`name` DESC SEPARATOR , ) AS models_list FROM `dvd_scenes` INNER JOIN dvds ON dvds.dvd_id = dvd_scenes.dvd_id INNER JOIN dvd_scenes_dyn ON dvd_scenes_dyn.id = dvd_scenes.scene_id LEFT JOIN scenes_categories ON dvd_scenes.scene_id = scenes_categories.scene_id LEFT JOIN categories ON `scenes_categories`.`category_id` = categories.category_id LEFT JOIN studios ON studios.studio_id = dvds.studio_id LEFT JOIN scenes_models ON scenes_models.scene_id = dvd_scenes.scene_id LEFT JOIN `channels` ON channels.channel_id = dvds.channel_id LEFT JOIN `channels_keywords` ON dvds.channel_id = channels_keywords.channel_id LEFT JOIN `models` ON scenes_models.model_id = models.model_id WHERE dvds.is_active = 1 AND dvd_scenes.is_active = 1 AND dvds.date_released < UNIX_TIMESTAMP() AND dvds.date_released > 0 AND dvds.channel_id NOT IN (21,26) 30 GROUP BY scene_idThursday, March 10, 2011 30
  36. 36. sphinx http://www.flickr.com/photos/waldenpond/4237758518/ 31Thursday, March 10, 2011 31
  37. 37. Sphinx Alternatives Lucene With Elastic Search solr 32Thursday, March 10, 2011 32
  38. 38. Assigning Problems 33Thursday, March 10, 2011 33
  39. 39. Appendix Paul Reinheimer - @preinheimer - http://blog.preinheimer.com Memcached - http://memcached.org/ supervisor - http://supervisord.org/ Sphinx - http://sphinxsearch.com/ Gearman - http://gearman.org/ 34Thursday, March 10, 2011 34
  40. 40. 5 Stars! 35Thursday, March 10, 2011 35
  41. 41. Feedback http://joind.in/2867 36Thursday, March 10, 2011 36
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×