Kill bottlenecks with gearman, sphinx, and memcached, Confoo 2011

Killing Bottlenecks paul reinheimer @preinheimer 1Thursday, March 10, 2011 1

Goals for Today Horror Stories Introduce Tools Help you recognize problems Hints on implementation 2Thursday, March 10, 2011 2

Provides a simple queue service Workers connect and identify work they can complete, clients hand off jobs when available Created by Danga Interactive (livejournal) 3Thursday, March 10, 2011 3

Stores data in memory really really fast! Many servers can be used at once to provide a single large datastore Created by Danga Interactive (livejournal) 4Thursday, March 10, 2011 4

Excellent search tool Import data from a variety of sources (commonly a database), index it quickly, then run searches 5Thursday, March 10, 2011 5

Know Thyself 6Thursday, March 10, 2011 6

Workers sign up with work they can do, many workers can each sign up with a different set of jobs. Clients connect to Gearman and pass off jobs, this is done asynchronously, they get to keep working Failed jobs are re-submitted a speciﬁed number of times. 7Thursday, March 10, 2011 7

Gearman Workers sign up with work they can do, many workers can each sign up with a different set of jobs. Clients connect to Gearman and pass off jobs, this is done asynchronously, they get to keep working Failed jobs are re-submitted a speciﬁed number of times. 8Thursday, March 10, 2011 8

Clients 9Thursday, March 10, 2011 9

Workers 10Thursday, March 10, 2011 10

Managing Workers 11Thursday, March 10, 2011 11

Supervisor 14Thursday, March 10, 2011 14

Gearman Alternatives RabitMQ Amazon Simple Queue Service ZeroMQ 15Thursday, March 10, 2011 15

Stores data in memory really really fast! Many servers can be used at once to provide a single large datastore Created by Danga Interactive (livejournal) 16Thursday, March 10, 2011 16

Promises 17Thursday, March 10, 2011 17

Promises memcached doesnt lie, it just doesnt make promises 18Thursday, March 10, 2011 18

Promises it makes a promise 19Thursday, March 10, 2011 19

Promises data will no longer be available after it expires 20Thursday, March 10, 2011 20

Promises When memcached doesnt have the information you need, get it elsewhere. Use a read-through caching callback or put it in your caching object. memcached is allowed to lose data! 21Thursday, March 10, 2011 21

Consistent Hashing 22Thursday, March 10, 2011 22

Hot Items = Hot Servers 23Thursday, March 10, 2011 23

Memcached Alternatives redis Berkeley DB Tokyo Cabinet Tokyo Tyrant Voldemort Riak 24Thursday, March 10, 2011 24

Excellent search tool Import data from a variety of sources (commonly a database), index it quickly, then run searches 25Thursday, March 10, 2011 25

id title year rating 1 TMNT 1988 9.5 2 Goonies 1989 9.0 3 The Postman 1992 3.5 4 Alien 1994 8.5 26Thursday, March 10, 2011 26

id title year rating studio category 1 TMNT 1988 9.5 x action 2 Goonies 1989 9.0 y adventure 3 The Postman 1992 3.5 z boring 4 Alien 1994 8.5 a awesome 27Thursday, March 10, 2011 27

id title year rating studio category 1 TMNT 1988 9.5 x action 2 Goonies 1989 9.0 y adventure 3 The Postman 1992 3.5 z boring 4 Alien 1994 8.5 a awesome 28Thursday, March 10, 2011 28

id director film_id tag_id 1 blue film_id description 1 2 2 red 1 Four awesome turtles... 2 3 3 green 2 Friends, pirates, goonies. 3 4 4 orange 3 Nap 4 5 4 They tried to turn around id title year rating studio category 1 TMNT 1988 9.5 x action 2 Goonies 1989 9.0 y adventure 3 The Postman 1992 3.5 z boring 4 Alien 1994 8.5 a awesome id tag 1 kids id producer id actor 1 1 2 awesome 1 Mel Gibson 2 2 3 sci-fi 2 Shredder 3 3 3 Sigourney Weaver 4 4 28Thursday, March 10, 2011 28

id director film_id tag_id 1 blue film_id description 1 2 2 red 1 Four awesome turtles... 2 3 3 green 2 Friends, pirates, goonies. 3 4 4 orange 3 Nap 4 5 4 They tried to turn around id title year rating studio category 1 TMNT 1988 9.5 x action 2 Goonies 1989 9.0 y adventure 3 The Postman 1992 3.5 z boring 4 Alien 1994 8.5 a awesome id tag 1 kids id producer id actor 1 1 2 awesome 1 Mel Gibson 2 2 3 sci-fi 2 Shredder 3 3 3 Sigourney Weaver 4 4 29Thursday, March 10, 2011 29

id director film_id tag_id 1 film_id blue director_id film_id description 1 film_id 2 tag_id 2 1 red 1 1 film_id desc_id Four awesome turtles... 2 1 3 1 3 2 green 2 2 Friends, pirates, goonies. 1 1 3 2 4 2 4 3 orange 3 3 2 Nap 2 4 3 5 3 4 4 4 They tried to turn around 3 3 4 4 4 4 id title year rating studio category 1 TMNT 1988 9.5 x action 2 Goonies 1989 9.0 y adventure 3 The Postman 1992 3.5 z boring 4 Alien 1994 8.5 a awesome id tag 1 id producer film_id kids tag_id id actor 1 film_id1 producer_id 2 1 awesome 1 1 film_id Mel Gibson actor_id 2 1 2 1 3 2 sci-fi 2 2 Shredder 1 1 3 2 3 2 3 3 3 2 Sigourney Weaver 2 4 3 4 3 4 4 3 3 4 4 4 4 29Thursday, March 10, 2011 29

SELECT `dvd_scenes`.`scene_id`, `dvds`.`dvd_id`, `dvds`.`name`, `dvds`.`name` AS sort_title, `dvds`.`featured` AS premium, `dvds`.`description`, `dvds`.`date_released`, `dvds`.`channel_id`, `dvds`.`studio_id`, `dvd_scenes_dyn`.`times_viewed` AS sub_order, `dvd_scenes_dyn`.`rating` AS rating, `channels`.`name` AS channel_name, `studios`.`name` AS studio_name, `channels_keywords`.`keywords` AS channel_keywords, GROUP_CONCAT(DISTINCT `categories`.`name` ORDER BY `categories`.`name` DESC SEPARATOR , ) AS categories_list, GROUP_CONCAT(DISTINCT `models`.`name` ORDER BY `models`.`name` DESC SEPARATOR , ) AS models_list FROM `dvd_scenes` INNER JOIN dvds ON dvds.dvd_id = dvd_scenes.dvd_id INNER JOIN dvd_scenes_dyn ON dvd_scenes_dyn.id = dvd_scenes.scene_id LEFT JOIN scenes_categories ON dvd_scenes.scene_id = scenes_categories.scene_id LEFT JOIN categories ON `scenes_categories`.`category_id` = categories.category_id LEFT JOIN studios ON studios.studio_id = dvds.studio_id LEFT JOIN scenes_models ON scenes_models.scene_id = dvd_scenes.scene_id LEFT JOIN `channels` ON channels.channel_id = dvds.channel_id LEFT JOIN `channels_keywords` ON dvds.channel_id = channels_keywords.channel_id LEFT JOIN `models` ON scenes_models.model_id = models.model_id WHERE dvds.is_active = 1 AND dvd_scenes.is_active = 1 AND dvds.date_released < UNIX_TIMESTAMP() AND dvds.date_released > 0 AND dvds.channel_id NOT IN (21,26) 30 GROUP BY scene_idThursday, March 10, 2011 30

sphinx http://www.ﬂickr.com/photos/waldenpond/4237758518/ 31Thursday, March 10, 2011 31

Sphinx Alternatives Lucene With Elastic Search solr 32Thursday, March 10, 2011 32

Assigning Problems 33Thursday, March 10, 2011 33

Appendix Paul Reinheimer - @preinheimer - http://blog.preinheimer.com Memcached - http://memcached.org/ supervisor - http://supervisord.org/ Sphinx - http://sphinxsearch.com/ Gearman - http://gearman.org/ 34Thursday, March 10, 2011 34

5 Stars! 35Thursday, March 10, 2011 35

Feedback http://joind.in/2867 36Thursday, March 10, 2011 36

Kill bottlenecks with gearman, sphinx, and memcached, Confoo 2011

by Bachkoutou Toutou on Mar 10, 2011

Accessibility

Tags

Upload Details

Usage Rights

2 Embeds 145

Statistics

Kill bottlenecks with gearman, sphinx, and memcached, Confoo 2011 — Presentation Transcript

http://www.berejeb.com	142
http://feeds.feedburner.com	3