Downloaded 61 times
![12
#
sudo
addgroup
hadoop
#
sudo
adduser
—ingroup
hadoop
hdfs
#
sudo
adduser
—ingroup
hadoop
yarn
#
cp
/tmp/configs/*.xml
/etc/hadoop/conf/
#
apt-‐get
update
…
[hdfs@sj-‐hadoop-‐b20
~]
$
apt-‐get
install
hadoop-‐hdfs-‐datanode
…
[yarn@sj-‐hadoop-‐b20
~]
$
apt-‐get
install
hadoop-‐yarn-‐nodemanager](https://image.slidesharecdn.com/theevolutionofhadoopatspotify-throughfailuresandpain-150314153821-conversion-gate01/75/The-Evolution-of-Hadoop-at-Spotify-Through-Failures-and-Pain-18-2048.jpg)
![12
#
sudo
addgroup
hadoop
#
sudo
adduser
—ingroup
hadoop
hdfs
#
sudo
adduser
—ingroup
hadoop
yarn
#
cp
/tmp/configs/*.xml
/etc/hadoop/conf/
#
apt-‐get
update
…
[hdfs@sj-‐hadoop-‐b20
~]
$
apt-‐get
install
hadoop-‐hdfs-‐datanode
…
[yarn@sj-‐hadoop-‐b20
~]
$
apt-‐get
install
hadoop-‐yarn-‐nodemanager](https://image.slidesharecdn.com/theevolutionofhadoopatspotify-throughfailuresandpain-150314153821-conversion-gate01/75/The-Evolution-of-Hadoop-at-Spotify-Through-Failures-and-Pain-19-2048.jpg)
![14
[data-‐sci@sj-‐edge-‐a1
~]
$
hdfs
dfs
-‐ls
/data
Found
3
items
drwxr-‐xr-‐x
-‐
hdfs
hadoop
0
2015-‐01-‐01
12:00
lake
drwxr-‐xr-‐x
-‐
hdfs
hadoop
0
2015-‐01-‐01
12:00
pond
drwxr-‐xr-‐x
-‐
hdfs
hadoop
0
2015-‐01-‐01
12:00
ocean
[data-‐sci@sj-‐edge-‐a1
~]
$
hdfs
dfs
-‐ls
/data/lake
Found
1
items
drwxr-‐xr-‐x
-‐
hdfs
hadoop
1321451
2015-‐01-‐01
12:00
boats.txt
[data-‐sci@sj-‐edge-‐a1
~]
$
hdfs
dfs
-‐cat
/data/lake/boats.txt
…](https://image.slidesharecdn.com/theevolutionofhadoopatspotify-throughfailuresandpain-150314153821-conversion-gate01/75/The-Evolution-of-Hadoop-at-Spotify-Through-Failures-and-Pain-21-2048.jpg)
![14
[data-‐sci@sj-‐edge-‐a1
~]
$
hdfs
dfs
-‐ls
/data
Found
3
items
drwxr-‐xr-‐x
-‐
hdfs
hadoop
0
2015-‐01-‐01
12:00
lake
drwxr-‐xr-‐x
-‐
hdfs
hadoop
0
2015-‐01-‐01
12:00
pond
drwxr-‐xr-‐x
-‐
hdfs
hadoop
0
2015-‐01-‐01
12:00
ocean
[data-‐sci@sj-‐edge-‐a1
~]
$
hdfs
dfs
-‐ls
/data/lake
Found
1
items
drwxr-‐xr-‐x
-‐
hdfs
hadoop
1321451
2015-‐01-‐01
12:00
boats.txt
[data-‐sci@sj-‐edge-‐a1
~]
$
hdfs
dfs
-‐cat
/data/lake/boats.txt
…](https://image.slidesharecdn.com/theevolutionofhadoopatspotify-throughfailuresandpain-150314153821-conversion-gate01/75/The-Evolution-of-Hadoop-at-Spotify-Through-Failures-and-Pain-22-2048.jpg)
![[data-‐sci@sj-‐edge-‐a1
~]
$
snakebite
rm
-‐R
/team/disco/
CF/test-‐10/
22](https://image.slidesharecdn.com/theevolutionofhadoopatspotify-throughfailuresandpain-150314153821-conversion-gate01/75/The-Evolution-of-Hadoop-at-Spotify-Through-Failures-and-Pain-34-2048.jpg)
![Goodbye Data (1PB)
[data-‐sci@sj-‐edge-‐a1
~]
$
snakebite
rm
-‐R
/team/disco/
CF/test-‐10/
22
OK:
Deleted
/team/disco](https://image.slidesharecdn.com/theevolutionofhadoopatspotify-throughfailuresandpain-150314153821-conversion-gate01/75/The-Evolution-of-Hadoop-at-Spotify-Through-Failures-and-Pain-35-2048.jpg)
Uploaded byRafał Wojdyła
The Evolution of Hadoop at Spotify - Through Failures and Pain
The document discusses the evolution of Hadoop at Spotify from 2009 to 2015, highlighting the company's growth, challenges, and improvements in their data infrastructure. It details the transition to a more reliable Hadoop setup with dedicated teams and automation tools, as well as lessons learned from failures. Additionally, it emphasizes the importance of user feedback and automation in scaling their data management systems.
More Related Content
Similar to The Evolution of Hadoop at Spotify - Through Failures and Pain
![[DSC Europe 25] Nikola Rajovic - Hardware Technologies Under the Hood: RISC-V...](https://cdn.slidesharecdn.com/ss_thumbnails/o2gptrmtoyqndgoshwgq-dsc2025-tenstorrent-rajovic-251205090438-814685f5-thumbnail.jpg?width=640&height=640&fit=bounds)
![[DSC Europe 25] Dusan Jovicic - AI Story: From on-prem to cloud and back agai...](https://cdn.slidesharecdn.com/ss_thumbnails/8kp49m6uq22ifnbwhfnk-2-251205085715-964d11a6-thumbnail.jpg?width=640&height=640&fit=bounds)
![[DSC Europe 25] Dragana Ilic - AI for Big Data in Astronomy.pptx](https://cdn.slidesharecdn.com/ss_thumbnails/8palya86qaatvjhva1ms-2-dragana-ilic-ai-ilic-251208151906-652b819c-thumbnail.jpg?width=640&height=640&fit=bounds)
![[DSC Europe 25] Petar Zivanov - AI meets documents From chatbots to AI-powere...](https://cdn.slidesharecdn.com/ss_thumbnails/xer2bb6nrdc8pdpev0pc-8-251204082258-7c2fa4a1-thumbnail.jpg?width=640&height=640&fit=bounds)
![[DSC Europe 25] Jim Sterne - Adopting Generative AI Capabilities Into the Ent...](https://cdn.slidesharecdn.com/ss_thumbnails/sxhpofuorcagxsaulkmt-3-251204082258-7e66bc48-thumbnail.jpg?width=640&height=640&fit=bounds)
![[DSC Europe 25] Vid Stimac - Policy Parsimony: Between Oversimplifying and Ov...](https://cdn.slidesharecdn.com/ss_thumbnails/eqlepagzqp2rhg3gbluh-dsc-stimac-251120-251205090438-059e7f54-thumbnail.jpg?width=640&height=640&fit=bounds)
![[DSC Europe 25] Dragan Vucic - Building the Learning Organization - How AI Tr...](https://cdn.slidesharecdn.com/ss_thumbnails/8brigo2sbu6qur6gxrra-7-251205085715-6ae07d24-thumbnail.jpg?width=640&height=640&fit=bounds)
![[DSC Europe 25] Max Talanov - Non digital NNs.pptx](https://cdn.slidesharecdn.com/ss_thumbnails/wif8tr3gtua74qvtopke-non-digital-nns-251205090438-26b0eea6-thumbnail.jpg?width=640&height=640&fit=bounds)
The Evolution of Hadoop at Spotify - Through Failures and Pain
- 1. The Evolution of Hadoopat Spotify Through Failures and Pain Josh Baer (jbx@spotify.com) Rafal Wojdyla (rav@spotify.com) 1 Note: Our views are our own and don't necessarily represent those of Spotify.
- 2. 2 • Growing Pains(2009-2012) • Gaining Focus (2013 - 2014) • The Future (2015+) Overview
- 3.
- 4.
- 5. What is Spotify? •Music Streaming Service • Browse and Discover Millions of Songs, Artists and Albums • Launched in October 2008 • December 2014: • 60 Million Monthly Users • 15 Million Paid Subscribers 5
- 6. What is Spotify? •Data Infrastructure: • 1300 Hadoop Nodes • 42 PB Storage • 20 TB data ingested via Kafka/day • 200 TB generated by Hadoop/day 6
- 7. 7 select artist_id, count(1) fromuser_activities where play_seconds > 30 group by artist_id;
- 8. 7 select artist_id, count(1) fromuser_activities where play_seconds > 30 group by artist_id;
- 9.
- 10. 0 * * * * spotify-‐core hadoop jar hourly_import.jar 15 * * * * spotify-‐core hadoop jar hourly_listeners.jar 30 * * * * spotify-‐analytics hadoop jar user_funnel_hourly.jar * 1 * * * spotify-‐core hadoop jar daily_aggregate.jar * 2 * * * spotify-‐core hadoop jar calculate_royalties.jar */2 22 * * * spotify-‐radio hadoop jar generate_radio.jar 8
- 11. 0 * * * * spotify-‐core hadoop jar hourly_import.jar 15 * * * * spotify-‐core hadoop jar hourly_listeners.jar 30 * * * * spotify-‐analytics hadoop jar user_funnel_hourly.jar * 1 * * * spotify-‐core hadoop jar daily_aggregate.jar * 2 * * * spotify-‐core hadoop jar calculate_royalties.jar */2 22 * * * spotify-‐radio hadoop jar generate_radio.jar 8
- 12. 9 Handles the ‘plumbing’for Hadoop jobs https://github.com/spotify/luigi
- 13.
- 14.
- 15.
- 16.
- 17.
- 18. 12 # sudo addgroup hadoop # sudo adduser —ingroup hadoop hdfs # sudo adduser —ingroup hadoop yarn # cp /tmp/configs/*.xml /etc/hadoop/conf/ # apt-‐get update … [hdfs@sj-‐hadoop-‐b20 ~] $ apt-‐get install hadoop-‐hdfs-‐datanode … [yarn@sj-‐hadoop-‐b20 ~] $ apt-‐get install hadoop-‐yarn-‐nodemanager
- 19. 12 # sudo addgroup hadoop # sudo adduser —ingroup hadoop hdfs # sudo adduser —ingroup hadoop yarn # cp /tmp/configs/*.xml /etc/hadoop/conf/ # apt-‐get update … [hdfs@sj-‐hadoop-‐b20 ~] $ apt-‐get install hadoop-‐hdfs-‐datanode … [yarn@sj-‐hadoop-‐b20 ~] $ apt-‐get install hadoop-‐yarn-‐nodemanager
- 20.
- 21. 14 [data-‐sci@sj-‐edge-‐a1 ~] $ hdfs dfs -‐ls /data Found 3 items drwxr-‐xr-‐x -‐ hdfs hadoop 0 2015-‐01-‐01 12:00 lake drwxr-‐xr-‐x -‐ hdfs hadoop 0 2015-‐01-‐01 12:00 pond drwxr-‐xr-‐x -‐ hdfs hadoop 0 2015-‐01-‐01 12:00 ocean [data-‐sci@sj-‐edge-‐a1 ~] $ hdfs dfs -‐ls /data/lake Found 1 items drwxr-‐xr-‐x -‐ hdfs hadoop 1321451 2015-‐01-‐01 12:00 boats.txt [data-‐sci@sj-‐edge-‐a1 ~] $ hdfs dfs -‐cat /data/lake/boats.txt …
- 22. 14 [data-‐sci@sj-‐edge-‐a1 ~] $ hdfs dfs -‐ls /data Found 3 items drwxr-‐xr-‐x -‐ hdfs hadoop 0 2015-‐01-‐01 12:00 lake drwxr-‐xr-‐x -‐ hdfs hadoop 0 2015-‐01-‐01 12:00 pond drwxr-‐xr-‐x -‐ hdfs hadoop 0 2015-‐01-‐01 12:00 ocean [data-‐sci@sj-‐edge-‐a1 ~] $ hdfs dfs -‐ls /data/lake Found 1 items drwxr-‐xr-‐x -‐ hdfs hadoop 1321451 2015-‐01-‐01 12:00 boats.txt [data-‐sci@sj-‐edge-‐a1 ~] $ hdfs dfs -‐cat /data/lake/boats.txt …
- 23.
- 24. $ time for i in {1..100}; do hadoop fs -‐ls / > /dev/null; done real 3m32.014s user 6m15.891s sys 0m18.821s $ time for i in {1..100}; do snakebite ls / > /dev/null; done real 0m34.760s user 0m29.962s sys 0m4.512s 16
- 25.
- 26. 18 • In 2013,expanded to 200 nodes • Hadoop critical • Needed a team totally focused on it • Created a ‘squad’ with two missions: • Migrate to a new distribution with Yarn • Make Hadoop reliable Forming a team
- 27.
- 28.
- 29.
- 30.
- 31. 19 Hadoop ownerless UpgradesGetting there Squad
- 32. 20 • Alert onservice level problems (i.e. no jobs running) • Keep your alarm channel clean. Beware of alert fatigue. Alerting
- 33. 21 Uhh ohh….. I thinkI made a mistake
- 34. [data-‐sci@sj-‐edge-‐a1 ~] $ snakebite rm -‐R /team/disco/ CF/test-‐10/ 22
- 35. Goodbye Data (1PB) [data-‐sci@sj-‐edge-‐a1 ~] $ snakebite rm -‐R /team/disco/ CF/test-‐10/ 22 OK: Deleted /team/disco
- 36. 23 • “Sit onyour hands before you type” - Wouter de Bie • Users will always want to retain data! • Remove superusers from ‘edgenodes’ • Moving to trash = client-side implementation Lessons Learned
- 37. 24 The Wild WildWest
- 38. 25 • Same hardwareprofile as production cluster • Similar configuration • Staging environment • Reliable Pre-Production Cluster
- 39.
- 40.
- 41.
- 42. 29 • Features: • Datadiscovery • Lineage • Lifecycle management • More • We use it for data movement • Uses Oozie behind the scenes Apache Falcon
- 43. • Most ofour jobs were Hadoop (python) Streaming • Lots of failures, slow performance • Had to find a better way…. 30 Improving Performance
- 44. 31 • Investigated severalframeworks • Selected Crunch: • Real types! • Higher level API • Easier to test • Better performance #JVM_FTW *Dave Whiting’s analysis of systems: http://thewit.ch/scalding_crunchy_pig Improving Performance
- 45.
- 46.
- 47.
- 48. 35 Note: Spotify usersis based on publicly released numbers only
- 49. 36 Explosive Growth • IncreasedSpotify Users • Increased use cases • Increased Engineers
- 50.
- 51. 38 Scaling machines: easy Scalingpeople: hard
- 52.
- 53.
- 54. 41 • Data-discovery tool •Luigi Integration • Find and browse datasets • View schemas • Trace lineage • Open-source plans? :-( Raynor
- 55. 42 Two Takeaways • AutomateEverything • More time to play FIFA build cool tools • Listen to your users • Fail fast, don’t be afraid to scrap work
- 56. 43 Join the Band! Engineerswanted in NYC & Stockholm http://spotify.com/jobs