Hadoop at Last.fm

Hadoop at Last.fm,[object Object],June 2010,[object Object]

About us,[object Object],Last.fm you say?,[object Object]

Last.fm is a,[object Object],music discovery website,[object Object],powered by scrobbling,[object Object],that provides personalized radio,[object Object]

Music discovery website,[object Object],Each month we get:,[object Object],over 40M unique visitors,[object Object],over 500M pageviews,[object Object],Each pageview leads to at least one log line,[object Object],Clicks and other interactions sometimes lead to log lines too,[object Object]

Powered by scrobbling,[object Object],scrobble: skrob·bul (ˈskrɒbəll),[object Object],[verb] To automatically add the tracks you play to your Last.fm profile with a piece of software called a Scrobbler,[object Object],Stats:,[object Object],Up to 800 scrobbles per second,[object Object],More than 40 million scrobbles per day,[object Object],Over 40 billion scrobbles so far,[object Object],Each scrobble leads to a log line,[object Object]

Personalized radio,[object Object],Via flash player, Xbox, desktop and mobile apps,[object Object],Stats:,[object Object],Over 10 million streaming hours per month,[object Object],Over 400 thousand unique stations per day,[object Object],Each stream leads to at least one log line,[object Object]

And it’s not just logs…,[object Object],So we gather a lot of logs, but also:,[object Object],Tags,[object Object],Shouts,[object Object],Journals,[object Object],Wikis,[object Object],Friend connections,[object Object],Fingerprints,[object Object],…,[object Object],Hadoop is the infrastructure we use for storing and processing our flood of data,[object Object]

OUR SETUP,[object Object],How many nodes?,[object Object]

Our herd of elephants,[object Object],Current specs of our production cluster:,[object Object],44 nodes,[object Object],8 cores per node,[object Object],16 GB memory per node,[object Object],4 disks of 1 TB spinning at 7200 RPM per node,[object Object],Unpatched CDH2 using:,[object Object],Fair scheduler with preemption,[object Object],Slightly patched hadoop-lzo,[object Object],RecordIO, Avro,[object Object],Hive, Dumbo, Pig,[object Object]

We often avoid Java with Dumbo,[object Object],def mapper(key, value):,[object Object], for word in value.split():,[object Object], yield word, 1,[object Object],def reducer(key, values):,[object Object], yield key, sum(values),[object Object],if __name__ == "__main__":,[object Object], import dumbo,[object Object],dumbo.run(mapper, reducer, combiner=reducer),[object Object]

Or go even more high-level with Hive,[object Object],hive> CREATE TABLE name_counts (gender STRING, name STRING, occurrences INT);,[object Object],hive> INSERT OVERWRITE TABLE name_counts,[object Object],SELECT lower(sex), lower(split(realname, ‘ ‘)[0]), count(1) FROM meta_user_info,[object Object],WHERE lower(sex) <> ‘n’ GROUP BY lower(sex), lower(split(realname, ‘ ‘)[0]);,[object Object],hive> CREATE TABLE gender_likelihoods (name STRING, gender STRING, likelihood FLOAT);,[object Object],hive> INSERT OVERWRITE TABLE gender_likelihoods,[object Object],SELECT b.name, b.gender, b.occurrences / a.occurrences FROM,[object Object],(SELECT name, sum(occurrences) as occurrences FROM name_counts GROUP BY name) a JOIN name_countsb ON (a.name = b.name);,[object Object],hive> SELECT * FROM gender_likelihoods WHERE (name = ‘klaas’) OR (name = ‘sam’);,[object Object],klaasm 0.99038464 ,[object Object],klaasf 0.009615385,[object Object],samm 0.7578873,[object Object],samf 0.24211268,[object Object]

Mixed usage of tools is common,[object Object],def starter(prog):,[object Object], month = prog.delopt(“month”) # is expected to be YYYY/MM,[object Object],hql = “INSERT OVERWRITE DIRECTORY ‘cool/stuff_hive’ ...”.format(month),[object Object], if os.system(‘hive –e “{0}”’.format(hql)) != 0:,[object Object], raise dumbo.Error("hive query failed"),[object Object],prog.addopt(“input”, “cool/stuff_hive”) # will be text delimited by ‘01’,[object Object],prog.addopt(“output”, “cool/stuff/” + month),[object Object],…,[object Object],if __name__ == “__main__”:,[object Object],dumbo.main(runner, starter),[object Object]

Running out of DFS space is common too,[object Object],Possible solutions:,[object Object],Bigger and/or more disks,[object Object],HDFS RAID,[object Object]

Running out of DFS space is common too,[object Object],Data deleted ,[object Object],New nodes,[object Object],More compression and nodes,[object Object],Not finalized yet after upgrade,[object Object],Possible solutions:,[object Object],Bigger and/or more disks,[object Object],HDFS RAID,[object Object]

Hitting I/O and CPU limits is less common,[object Object],Our cluster can be pretty busy at times,[object Object],But DFS space is our main worry,[object Object]

Hitting I/O and CPU limits is less common,[object Object],Upgraded to 0.20,[object Object],Our cluster can be pretty busy at times,[object Object],But DFS space is our main worry,[object Object]

USE CASES,[object Object],What do you use it for?,[object Object]

Things we do with Hadoop,[object Object],Site stats and metrics,[object Object],Charts,[object Object],Reporting,[object Object],Metadata corrections,[object Object],Neighbours,[object Object],Recommendations,[object Object],Indexing for search,[object Object],Evaluations,[object Object],Data insights,[object Object]

And also scaring our ops…,[object Object]

Example: Website traffic stats,[object Object],We compute a lot of site metrics, mostly from apache logs,[object Object]

Example: Website traffic stats,[object Object],Google Chrome is gaining ground,[object Object],We compute a lot of site metrics, mostly from apache logs,[object Object]

Example: Overall charts,[object Object],Charts for a single user can be shown in real time and are computed on the fly,[object Object],But computing overall charts is a pretty big job and is done on Hadoop,[object Object]

Example: World charts,[object Object],This “world chart” for the Belgian band “Hooverphonic” also required Hadoop because it’s based on data from many different users,[object Object]

Example: Overall wave graphs ,[object Object],Overall visualizations also typically require Hadoop for getting the data they visualize,[object Object],The “wave graphs” in our “Best of 2009” newspaper were good examples,[object Object]

Example: Overall wave graphs ,[object Object]

Example: Death and scrobbles graphs,[object Object],−scrobbles,[object Object],− listeners,[object Object]

Example: Radio stats,[object Object],Graphs for several metrics that can be broken down by various attributes,[object Object],Used extensively for A/B testing,[object Object]

Example: Radio stats,[object Object],Graphs for several metrics that can be broken down by various attributes,[object Object],Used extensively for A/B testing,[object Object],Significant differences,[object Object]

Example: Radio stats,[object Object],Graphs for several metrics that can be broken down by various attributes,[object Object],Used extensively for A/B testing,[object Object],Overheated data centre,[object Object],Significant differences,[object Object],DB maintenance that went bad,[object Object]

Thanks!,[object Object],klaas@last.fm @klbosteemarc@last.fm @lanttims@last.fm @roserpens,[object Object]

Hadoop at Last.fm

Hadoop at Last.fm

Recommended

More Related Content

Viewers also liked

Viewers also liked(20)

Similar to Hadoop at Last.fm

Similar to Hadoop at Last.fm(20)

Recently uploaded

Recently uploaded(20)

Hadoop at Last.fm