How Apache Drives Music Recommendations At Spotify

•

12 likes•5,868 views

The slides go through the high-level process of generating personalized playlists for all Spotify's users, using Apache big data products extensively. Presentation given at Apache: Big Data Europe conference on September 29th, 2015 in Budapest.

Technology

HowApache
Drives Music
Recommendations
At Spotify
Josh Baer (jbx@spotify.com)
Note:The view expressed is my own and does not necessarily represent that of Spotify

WhoAm I?
• Technical Product Owner at
Spotify
• Working with batch and fast
processing infrastructure
@l_phant

What is Spotify?
• Music Streaming Service
• Launched in 2008
• Free and PremiumTiers
• Available in 58 Countries

How do we recommend a
personalized playlist of
new music to 75+ million
users?

10.123.133.333 - - [Mon, 3 June 2015 11:31:33 GMT] "GET /api/admin/job/aggregator/status HTTP/1.1" 200 1847 "https://my.analytics.app/
admin" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.81 Safari/537.36"
10.123.133.222 - - [Mon, 3 June 2015 11:31:43 GMT] "GET /api/admin/job/aggregator/status HTTP/1.1" 200 1984 "https://my.analytics.app/
admin" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.81 Safari/537.36”
10.123.133.222 - - [Mon, 3 June 2015 11:33:02 GMT] "GET /dashboard/courses/1291726 HTTP/1.1" 304 - "https://my.analytics.app/admin"
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.81 Safari/537.36"
10.321.145.111 - - [Mon, 3 June 2015 11:33:03 GMT] "GET /api/loggedInUser HTTP/1.1" 304 - "https://my.analytics.app/dashboard/courses/
1291726" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.81 Safari/537.36"
10.112.322.111 - - [Mon, 3 June 2015 11:33:03 GMT] "POST /api/instrumentation/events/new HTTP/1.1" 200 2 "https://my.analytics.app/
dashboard/courses/1291726" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.81
Safari/537.36”
10.123.133.222 - - [Mon, 3 June 2015 11:33:02 GMT] "GET /dashboard/courses/1291726 HTTP/1.1" 304 - "https://my.analytics.app/admin"
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.81 Safari/537.36"
It begins with a log

Apache Kafka at Spotify
•340 Kafka-related nodes
•30TB/day from logs

How do we store TBs of
new data every data?

Apache Hadoop at Spotify
• 1700 Nodes
• 60 PB of Data
• 70TB of Memory
• Over 1 Million jobs run in Q3, 2015

ProcessingGrowth
150%
250%
350%
450%
550%
Q4-2013 Q1-2014 Q2-2014 Q3-2014 Q4-2014 Q1-2015 Q2-2015 Q3-2015
Hadoop at Spotify

Processing Toolbox
• Apache Crunch
• Scalding
• Apache Hive
• Apache Spark
• Apache Storm
• Hadoop Streaming
• Apache Pig

Storage Formats
• ApacheAvro
• Apache Parquet

Collaborative Filtering
Justin Bieber Drake Avicii Major Lazer
Anna Listened Listened
Gustav Listened Listened Listened
Mary Listened Listened Listened Listened
Michael Listened ListenedSuggest

How do we serve new
playlists to all our users
every week?

Apache Cassandra at Spotify
• Number of Clusters: 113
• Number of Machines: 1155
• Largest Cluster: 60 Nodes

Thank YOU for your
contributions to
Apache products!

Spotify Luigi
•Workflow Manager
•Over 150 contributors
•Used by 10s, possibly 100s of companies

Maybe…Apache Luigi?
Sponsors/mentors/contributors wanted!

Think this stuff is interesting?
We have a great time building it!
spotify.com/jobs

Better Spotify ML Presentations
• Algorithmic Music Recommendations at Spotify (Chris Johnson)
• Interactive Recommender Systems with Netflix and Spotify (Chris Johnson)
• Music recommendations @ MLConf 2014 (Erik Bernhardsson)
• Machine learning @ Spotify (Andy Sloane)
• Recommending music on Spotifywith deep learning (Sander Dieleman)
• Scala Data Pipelines @ Spotify (Neville Li)
• Spotify's Music Recommendations LambdaArchitecture (Esh Kumar and Emily Samuels)

What's hot

Collaborative Filtering at Spotify

Erik Bernhardsson

In this presentation, I give an overview of the machine learning algorithms behind Spotify’s extraordinarily popular Discover Weekly playlist. I provide a brief introduction to what the playlist is, explain how music recommendation engines have evolved over time, then break down the three main algorithm types powering Spotify’s recommendations: (1) collaborative filtering, (2) Natural Language Processing (NLP), and (3) Raw audio analysis. Video of the presentation can be found here: https://www.youtube.com/watch?v=PUtYNjInopA

Spotify Discover Weekly: The machine learning behind your music recommendations

Sophia Ciocca

Music recommendations @ MLConf 2014

Erik Bernhardsson

Big data and machine learning @ Spotify

Oscar Carlsson

Spotify: Data center & Backend buildout

David Poblador i Garcia

How data drives spotify

Ali Sarrafi

Personalized Playlists at Spotify

Rohan Agrawal

Scala Data Pipelines @ Spotify

Neville Li

Music Personalization At Spotify

Vidhya Murali

Playlist Recommendations @ Spotify

Nikhil Tibrewal

Spotify uses a range of Machine Learning models to power its music recommendation features including the Discover page, Radio, and Related Artists. Due to the iterative nature of these models they are a natural fit to the Spark computation paradigm and suffer from the IO overhead incurred by Hadoop. In this talk, I review the ALS algorithm for Matrix Factorization with implicit feedback data and how we’ve scaled it up to handle 100s of Billions of data points using Scala, Breeze, and Spark.

Music Recommendations at Scale with Spark

Chris Johnson

Homepage Personalization at Spotify

Oguz Semerci

Nearest neighbors refers to something that is conceptually very simple. For a set of points in some space (possibly many dimensions), we want to find the closest k neighbors quickly. This presentation covers a library called Annoy built my me that that helps you do (approximate) nearest neighbor queries in high dimensional spaces. We're going through vector models, how to measure similarity, and why nearest neighbor queries are useful.

Approximate nearest neighbor methods and vector models – NYC ML meetup

Erik Bernhardsson

Storm at Spotify

Neville Li

Building Data Pipelines for Music Recommendations at Spotify

Vidhya Murali

Are you still building data pipelines with Java and Python? Are you curious about the current buzz in the Big Data community surrounding Scala as a data processing environment? In this talk I'll discuss how Spotify migrated its music recommendations pipeline from Python to Scala. I'll dive into the language specific features that make Scala the ideal candidate for big data processing as well as highlight the rich set of tools and APIs that we take advantage of to process music recommendations for our 50 Million active users including Scalding, Breeze, Kafka, Spark, Parquet, Driven and Zeppelin.

Scala Data Pipelines for Music Recommendations

Chris Johnson

Recommending and searching @ Spotify

Mounia Lalmas-Roelleke

ML+Hadoop at NYC Predictive Analytics

Erik Bernhardsson

These are the slides of my talk at the 2019 Netflix Workshop on Personalization, Recommendation and Search (PRS). This talk is based on previous talks on research we are doing at Spotify, but here I focus on the work we do on personalizing Spotify Home, with respect to success, intent & diversity. The link to the workshop is https://prs2019.splashthat.com/. This is research from various people at Spotify, and has been published at RecSys 2018, CIKM 2018 and WWW (The Web Conference) 2019.

Personalizing the listening experience

Mounia Lalmas-Roelleke

Machine learning @ Spotify - Madison Big Data Meetup

Andy Sloane

What's hot (20)

Collaborative Filtering at Spotify

Spotify Discover Weekly: The machine learning behind your music recommendations

Music recommendations @ MLConf 2014

Big data and machine learning @ Spotify

Spotify: Data center & Backend buildout

How data drives spotify

Personalized Playlists at Spotify

Scala Data Pipelines @ Spotify

Music Personalization At Spotify

Playlist Recommendations @ Spotify

Music Recommendations at Scale with Spark

Homepage Personalization at Spotify

Approximate nearest neighbor methods and vector models – NYC ML meetup

Storm at Spotify

Building Data Pipelines for Music Recommendations at Spotify

Scala Data Pipelines for Music Recommendations

Recommending and searching @ Spotify

ML+Hadoop at NYC Predictive Analytics

Personalizing the listening experience

Machine learning @ Spotify - Madison Big Data Meetup

Viewers also liked

Shortening the feedback loop

Josh Baer

Interactive recommender systems enable the user to steer the received recommendations in the desired direction through explicit interaction with the system. In the larger ecosystem of recommender systems used on a website, it is positioned between a lean-back recommendation experience and an active search for a specific piece of content. Besides this aspect, we will discuss several parts that are especially important for interactive recommender systems, including the following: design of the user interface and its tight integration with the algorithm in the back-end; computational efficiency of the recommender algorithm; as well as choosing the right balance between exploiting the feedback from the user as to provide relevant recommendations, and enabling the user to explore the catalog and steer the recommendations in the desired direction. In particular, we will explore the field of interactive video and music recommendations and their application at Netflix and Spotify. We outline some of the user-experiences built, and discuss the approaches followed to tackle the various aspects of interactive recommendations. We present our insights from user studies and A/B tests. The tutorial targets researchers and practitioners in the field of recommender systems, and will give the participants a unique opportunity to learn about the various aspects of interactive recommender systems in the video and music domain. The tutorial assumes familiarity with the common methods of recommender systems.

Interactive Recommender Systems with Netflix and Spotify

Chris Johnson

At Spotify we collect huge volumes of data for many purposes. Reporting to labels, powering our product features, and analyzing user growth are some of our most common ones. Additionally, we collect many operational metrics related to the responsiveness, utilization and capacity of our servers. To store and process this data, we use scalable and fault-tolerant multi-system infrastructure, and Apache Hadoop is a key part of it. Surprisingly or not, Apache Hadoop generates large amounts of data in the form of logs and metrics that describe its behaviour and performance. To process this data in a scalable and performant manner we use … also Hadoop! During this presentation, I will talk about how we analyze various logs generated by Apache Hadoop using custom scripts (written in Pig or Java/Python MapReduce) and available open-source tools to get data-driven answers to many questions related to the behaviour of our 690-node Hadoop cluster. At Spotify we frequently leverage these tools to learn how fast we are growing, when to buy new nodes, how to calculate the empirical retention policy for each dataset, optimize the scheduler, benchmark the cluster, find its biggest offenders (both people and datasets) and more.

Hadoop Operations Powered By ... Hadoop (Hadoop Summit 2014 Amsterdam)

Adam Kawa

Playlists at Spotify - Using Cassandra to store version controlled objects

Jimmy Mårdell

Spotify's Music Recommendations Lambda Architecture

Esh Vckay

Apache Hadoop In Theory And Practice

Adam Kawa

HCatalog

GetInData

Docker at Spotify - Dockercon14

dotCloud

Empowering Engineering Talent - an update from Spotify

Kevin Goldsmith

How spotify makes product

Ali Sarrafi

Activation: From thinking to tweaking it, how we do it at Spotify

TheFamily

Scaling Operations At Spotify

David Poblador i Garcia

Most of the people think that quality in software development is limited to manual testing on the latest stage before releasing a product. That might be true 20 years ago in the industrial era. But current world is much more dynamic than before. Time to market became the most crucial metric nowadays. Releasing code to production need to be done faster and faster. How to maintain quality on a sufficient level in this fast paced environment? How to find a time to work on quality improvements? Those are two main questions I want to answer during this talk. Do not expect a silver bullet or even receipt to success. But definitely expect a lot of information about continuous delivery/deployment/improvements with a case studies and lessons we learned at Spotify. Spotify Engineering Culture: https://labs.spotify.com/2014/03/27/spotify-engineering-culture-part-1/ https://labs.spotify.com/2014/09/20/spotify-engineering-culture-part-2/ Scaling Agile @ Spotify http://blog.crisp.se/2012/11/14/henrikkniberg/scaling-agile-at-spotify Scaled Agile @ Spotify http://vimeo.com/111131934

Quality Built In @ Spotify

Andrii Dzynia

This was an extended version of the talk that I gave at InfoShare 2016 in GDansk. This version of the talk was presented at ao.com and Think Money in Manchester UK in May 2016. This is a remix of several earlier talks and some new content to tie Spotify's autonomy and continuous improvement culture to it's data-driven product development approach to show the complete picture. As usual, I tend to talk to slides instead of putting a lot of the content into the slides themselves, so sorry if these don't have all the info.

How Spotify Builds Products (Organization. Architecture, Autonomy, Accountabi...

Kevin Goldsmith

Yahoo: Experiences with MySQL GTID and Multi Threaded Replication

Yashada Jadhav

Présentation Spotify: Profils d'écoute et data storytelling @ Radio 2.0 2015 V rencontres Radio 2.0 - Mardi 13 octobre 2015 'La radio 2.0 et vous ? Raconte-moi une histoire' www.rr20.fr 50 intervenants / 10 keynotes / 4 tables rondes / 7 workshops / 4 études exclusives Grands Prix Radio 2.0 / La Radio des Rencontres - Organisateurs : Nicolas Moulard / Actuonda, Xavier Filliol / Editions de l'Octet, Sacem Université, INA - Partenaires Platinum : Mediametrie - Partenaires Grand Prix : Scam, Multivote - Partenaires Gold : Spotify, Deezer, Hyperworld, Musicovery, Targetspot, Triton Digital - Partenaires Média : CBNews, La Lettre Pro de la Radio, RadioPub, Media +, Edition Multimedia, Satellinet, French Web, La Correspondance de la Publicité - La Radio des Rencontres : Broadcast Associés, Radioline, Glowbl

Spotify: Profils d'écoute et data storytelling @ Radio 2.0 2015

ACTUONDA

Music Recommendations at Spotify

Emily Samuels

Impact et attractivité des concerts des radio pour les auditeurs Etude HyperWorld Salon de la Radio 2017 VI Rencontres Radio 2.0 ' Audio Digital Reboot' Lundi 30 janvier 2017 / Grande Halle de la Villette / Salon de la Radio-European Radio Show 45 intervenants : 7 Keynotes / 4 Tables Rondes / 4 Etudes exclusives / 7 Workshops www.rr20.fr Partenaires Gold : Deezer, Spotify, RCS, Blitzr, Sync, Coyote, ACPM, Sacem, Kantar Media Partenaires Communication : Mediamétrie, Scam, INA, Hyperworld Partenaires Media : La Lettre Pro, Radiopub, CBNews, Mind, Influencia, Frenchweb, Satellifax, Media+, Longeur D'ondes, RadioWorld Co-Organisateurs : Les Editions de l'Octet, Actuonda

Impact et attractivité des concerts des radio pour les auditeurs Etude HyperW...

ACTUONDA

Webinar Radio 2.0 Musique et Marques : La Musique en ligne, un nouveau vecteur de comunication ? Présentation de VIF le 14 oct 2016 Dans le cadre du M@MaEvent en collaboration avec l'IRMA. Pour plus d'info sur la Radio 2.0 Webinar Series : http://www.rr20.fr/ Les Webinar Series sont réalisés en partenariat avec Deezer, Spotify, Médiamétrie, INA, RCS, Glowbl, Les Editions de l'Octet, Actuonda, La Lettre Pro, PRP, Minds, Media+, Radiopub

Vif Musique et Marques @ Webinar Radio 2.0 MaMA 2016

ACTUONDA

Reinventing Radio and Music for digital audience at BBC Keynote Mark Friend / BBC Vidéo correspondante : https://youtu.be/TiuA_7w79TQ VI Rencontres Radio 2.0 ' Audio Digital Reboot' Lundi 30 janvier 2017 / Grande Halle de la Villette / Salon de la Radio-European Radio Show 45 intervenants : 7 Keynotes / 4 Tables Rondes / 4 Etudes exclusives / 7 Workshops www.rr20.fr Partenaires Gold : Deezer, Spotify, RCS, Blitzr, Sync, Coyote, ACPM, Sacem, Kantar Media Partenaires Communication : Mediamétrie, Scam, INA, Hyperworld Partenaires Media : La Lettre Pro, Radiopub, CBNews, Mind, Influencia, Frenchweb, Satellifax, Media+, Longeur D'ondes, RadioWorld Co-Organisateurs : Les Editions de l'Octet, Actuonda

Reinventing radio and music for digital audience at BBC by Marc Friend @ Radi...

ACTUONDA

Viewers also liked (20)

Shortening the feedback loop

Interactive Recommender Systems with Netflix and Spotify

Hadoop Operations Powered By ... Hadoop (Hadoop Summit 2014 Amsterdam)

Playlists at Spotify - Using Cassandra to store version controlled objects

Spotify's Music Recommendations Lambda Architecture

Apache Hadoop In Theory And Practice

HCatalog

Docker at Spotify - Dockercon14

Empowering Engineering Talent - an update from Spotify

How spotify makes product

Activation: From thinking to tweaking it, how we do it at Spotify

Scaling Operations At Spotify

Quality Built In @ Spotify

How Spotify Builds Products (Organization. Architecture, Autonomy, Accountabi...

Yahoo: Experiences with MySQL GTID and Multi Threaded Replication

Spotify: Profils d'écoute et data storytelling @ Radio 2.0 2015

Music Recommendations at Spotify

Impact et attractivité des concerts des radio pour les auditeurs Etude HyperW...

Vif Musique et Marques @ Webinar Radio 2.0 MaMA 2016

Reinventing radio and music for digital audience at BBC by Marc Friend @ Radi...

Similar to How Apache Drives Music Recommendations At Spotify

Spotify in the Cloud - An evolution of data infrastructure - Strata NYC

Josh Baer

Shortening the Feedback Loop: How Spotify’s Big Data Ecosystem has evolved to...

Big Data Spain

fluent-plugin-beats at Elasticsearch meetup #14

N Masahiro

Guidelines for productive full stack data engineers

DataWorks Summit

Productive data engineer

Rafał Wojdyła

Music Hackday Boston - The Last.fm API

davidsingleton

How do volunteer open-source projects create and maintain so many compelling, competitive products? What is the Open Source Secret Sauce? Join open-source insider, Ted Husted, as he takes us deep inside the Apache Software Foundation, to show how the sausages are made. In this session, you will learn * Why open source matters; * How open source development works at the ASF; * What makes open source projects successful.

Open source-secret-sauce-rit-2010

Ted Husted

Siphon is a highly available and reliable distributed pub/sub system built using Apache Kafka. It is used to publish, discover and subscribe to near real-time data streams for operational and product intelligence. Siphon is used as a “Databus” by a variety of producers and subscribers in Microsoft, and is compliant with security and privacy requirements. It has a built-in Auditing and Quality control. This session will provide an overview of the use of Kafka at Microsoft, and then deep dive into Siphon. We will describe an important business scenario and talk about the technical details of the system in the context of that scenario. We will also cover the design and implementation of the service, the scale, and real world production experiences from operating the service in the Microsoft cloud environment.

Siphon - Near Real Time Databus Using Kafka, Eric Boyd, Nitin Kumar

confluent

Data minutes #2 Apache Pulsar with MQTT for Edge Computing Lightning - 2022

Timothy Spann

https://arcade.sqlbits.com/sessions/ Sql bits apache nifi 101 Introduction and best practices 11-March-2022 UK https://www.datainmotion.dev/2020/06/no-more-spaghetti-flows.html https://github.com/tspannhw/EverythingApacheNiFi https://www.datainmotion.dev/2020/12/basic-understanding-of-cloudera-flow.html https://www.datainmotion.dev/2020/10/top-25-use-cases-of-cloudera-flow.html In this talk, we will walk step by step through Apache NiFi from the first load to first application. I will include slides, articles and examples to take away as a Quick Start to utilizing Apache NiFi in your real-time dataflows. I will help you get up and running locally on your laptop, Docker or in CDP Public Cloud. I will cover: Terminology Flow Files Version Control Repositories Basic Record Processing Provenance Backpressure Prioritizers System Diagnostics Processors Process Groups Scheduling and Cron Bulletin Board Relationships Routing Tasks Networking Basic Cluster Architecture Listeners Controller Services Remote Ports Handling Errors Funnels Feedback LInk - https://sqlb.it/?7108 ROOM 04 Fri 12:00 - 12:50

Sql bits apache nifi 101 Introduction and best practices

Timothy Spann

The log shipping scene been between us for a long time: from syslog, rsyslog to nowadays Fluentd, Flume and Logstash. Logstash been pushing hard to introduce new features that make the experience better for everyone. At the end of the day, a healthy shipper means a happy sysadmin. The latest Logstash includes persistence to reduce the chance of data loss, monitoring to find how everything is going and configuration management to make your life a lot easier. But wait, there’s more! Offline support, improved shutdown semantics, etc … features that will make your logs shipped and you a rested sysadmin. In this talk we’ll see this features in action thought a real live sensor monitoring example. By the end of the session, you will be able to use the full power of Logstash in your own deployments.

OSDC 2016 - Ingesting Logs with Style by Pere Urbon-Bayes

NETWAYS

Apache Pulsar Community-Jennifer

StreamNative

Yuto Kawamura Building a company-wide data pipeline on Apache Kafka - engineering for 150 billion messages per day Summary: LINE is a messaging service with 200 million monthly active users. Our service architecture evolves daily with various collaborating components. I'll introduce overview of LINE's messaging service architecture, mainly focusing on our company-wide data pipeline infrastructure built upon Apache Kafka which accepts more than 150 billion messages every day, making it one of the largest in the world. In this talk I will introduce how we managed such scale keeping it highly reliable to be capable of being an infrastructure to build services.

Building a company-wide data pipeline on Apache Kafka - engineering for 150 b...

LINE Corporation

Designing a social network offers some exciting challenges to engineers. The system needs to operate at scale, to provide a responsive user experience and to be able to inspect user activity in order to both generate new content and improve how the existing content is delivered. Event Driven Architectures are particularly suitable to handle these kind of challenges and highly scalable messaging systems such as Apache Kafka have been designed specifically to support the requirements of modern high volume applications. In this talk we are describing how the Crowdmix back-end has been designed as an Event Based system running on top of Kafka. We are going to present the overall system architecture and discuss in more detail some of the different sub components processing those events in different fashions, from streaming based processing to batch processing passing through a lambda-style batch and stream cooperation. We are going to conclude describing some lessons learned from our one-year journey in implementing and operating the system

Music streams

Stefano Galarraga

Create Your Own Chatbot with Hubot and CoffeeScript

Rob Scaduto

Trend Micro Big Data Platform and Apache Bigtop

Evans Ye

Spotify: behind the scenes

Ricardo Vice Santos

Cloud lunch and learn real-time streaming in azure

Timothy Spann

Introduction Apache Kafka

Joe Stein

Building a company-wide data pipeline on Apache Kafka - engineering for 150 b...

LINE Corporation

Similar to How Apache Drives Music Recommendations At Spotify (20)

Spotify in the Cloud - An evolution of data infrastructure - Strata NYC

Shortening the Feedback Loop: How Spotify’s Big Data Ecosystem has evolved to...

fluent-plugin-beats at Elasticsearch meetup #14

Guidelines for productive full stack data engineers

Productive data engineer

Music Hackday Boston - The Last.fm API

Open source-secret-sauce-rit-2010

Siphon - Near Real Time Databus Using Kafka, Eric Boyd, Nitin Kumar

Data minutes #2 Apache Pulsar with MQTT for Edge Computing Lightning - 2022

Sql bits apache nifi 101 Introduction and best practices

OSDC 2016 - Ingesting Logs with Style by Pere Urbon-Bayes

Apache Pulsar Community-Jennifer

Building a company-wide data pipeline on Apache Kafka - engineering for 150 b...

Music streams

Create Your Own Chatbot with Hubot and CoffeeScript

Trend Micro Big Data Platform and Apache Bigtop

Spotify: behind the scenes

Cloud lunch and learn real-time streaming in azure

Introduction Apache Kafka

Building a company-wide data pipeline on Apache Kafka - engineering for 150 b...

Recently uploaded

Effective data discovery is crucial for maintaining compliance and mitigating risks in today's rapidly evolving privacy landscape. However, traditional manual approaches often struggle to keep pace with the growing volume and complexity of data. Join us for an insightful webinar where industry leaders from TrustArc and Privya will share their expertise on leveraging AI-powered solutions to revolutionize data discovery. You'll learn how to: - Effortlessly maintain a comprehensive, up-to-date data inventory - Harness code scanning insights to gain complete visibility into data flows leveraging the advantages of code scanning over DB scanning - Simplify compliance by leveraging Privya's integration with TrustArc - Implement proven strategies to mitigate third-party risks Our panel of experts will discuss real-world case studies and share practical strategies for overcoming common data discovery challenges. They'll also explore the latest trends and innovations in AI-driven data management, and how these technologies can help organizations stay ahead of the curve in an ever-changing privacy landscape.

TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery

TrustArc

Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...

Neo4j

Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024

The Digital Insurer

MINDCTI Revenue Release Quarter One 2024

MIND CTI

ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke

Product Anonymous

This presentations targets students or working professionals. You may know Google for search, YouTube, Android, Chrome, and Gmail, but did you know Google has many developer tools, platforms & APIs? This comprehensive yet still high-level overview outlines the most impactful tools for where to run your code, store & analyze your data. It will also inspire you as to what's possible. This talk is 50 minutes in length.

Powerful Google developer tools for immediate impact! (2023-24 C)

wesley chun

Join our latest Connector Corner webinar to discover how UiPath Integration Service revolutionizes API-centric automation in a 'Quote to Cash' process—and how that automation empowers businesses to accelerate revenue generation. A comprehensive demo will explore connecting systems, GenAI, and people, through powerful pre-built connectors designed to speed process cycle times. Speakers: James Dickson, Senior Software Engineer Charlie Greenberg, Host, Product Marketing Manager

Connector Corner: Accelerate revenue generation using UiPath API-centric busi...

DianaGray10

AWS Community Day CPH - Three problems of Terraform

Andrey Devyatkin

MySQL Webinar, presented on the 25th of April, 2024. Summary: MySQL solutions enable the deployment of diverse Database Architectures tailored to specific needs, including High Availability, Disaster Recovery, and Read Scale-Out. With MySQL Shell's AdminAPI, administrators can seamlessly set up, manage, and monitor these solutions, ensuring efficiency and ease of use in their administration. MySQL Router, on the other hand, provides transparent routing from the application traffic to the backend servers in the architectures, requiring minimal configuration. Completely built in-house and supported by Oracle, these solutions have been adopted by enterprises of all sizes for their business-critical applications. In this presentation, we'll delve into various database architecture solutions to help you choose the right one based on your business requirements. Focusing on technical details and the latest features to maximize the potential of these solutions.

Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...

Miguel Araújo

Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024

The Digital Insurer

Increase engagement and revenue with Muvi Live Paywall! In this presentation, we will explore the five key benefits of using Muvi Live Paywall to monetize your live streams. You'll learn how Muvi Live Paywall can help you: Monetize your live content easily: Set up pay-per-view access to your live streams and start generating revenue from your content. Increase audience engagement: Provide exclusive, premium content behind the paywall to keep your viewers engaged. Gain valuable viewer insights: Track viewer data and analytics to better understand your audience and tailor your content accordingly. Reduce content piracy: Muvi Live Paywall's security features help protect your content from unauthorized distribution. Streamline your workflow: The all-in-one platform simplifies the process of managing and monetizing your live streams. With Muvi Live Paywall, you can take control of your live stream monetization and create a sustainable business model for your content. Learn more about Muvi Live Paywall and start generating revenue from your live streams today!

Top 5 Benefits OF Using Muvi Live Paywall For Live Streams

Roshan Dwivedi

Webinar Recording: https://www.panagenda.com/webinars/why-teams-call-analytics-is-critical-to-your-entire-business Nothing is as frustrating and noticeable as being in an important call and being unable to see or hear the other person. Not surprising then, that issues with Teams calls are among the most common problems users call their helpdesk for. Having in depth insight into everything relevant going on at the user’s device, local network, ISP and Microsoft itself during the call is crucial for good Microsoft Teams Call quality support. To ensure a quick and adequate solution and to ensure your users get the most out of their Microsoft 365. But did you know that ‘bad calls’ are also an excellent indicator of other problems arising? Precisely because it is so noticeable!? Like the canary in the mine, bad calls can be early indicators of problems. Problems that might otherwise not have been noticed for a while but can have a big impact on productivity and satisfaction. Join this session by Christoph Adler to learn how true Microsoft Teams call quality analytics helped other organizations troubleshoot bad calls and identify and fix problems that impacted Teams calls or the use of Microsoft365 in general. See what it can do to keep your users happy and productive! In this session we will cover - Why CQD data alone is not enough to troubleshoot call problems - The importance of attributing call problems to the right call participant - What call quality analytics can do to help you quickly find, fix-, and prevent problems - Why having retrospective detailed insights matters - Real life examples of how others have used Microsoft Teams call quality monitoring to problem shoot problems with their ISP, network, device health and more.

Why Teams call analytics are critical to your entire business

panagenda

Automating Google Workspace (GWS) & more with Apps Script

wesley chun

Real Time Object Detection Using Open CV

Khem

In this session, we will delve into strategic approaches for optimizing knowledge management within Microsoft 365, amidst the evolving landscape of Copilot. From leveraging automatic metadata classification and permission governance with SharePoint Premium, to unlocking Viva Engage for the cultivation of knowledge and communities, you will gain actionable insights to bolster your organization's knowledge-sharing initiatives. In this session, we will also explore how to facilitate solutions to enable your employees to find answers and expertise within Microsoft 365. You will leave equipped with practical techniques and a deeper understanding of how there is more to effective knowledge management than just enabling Copilot, but building actual solutions to prepare the knowledge that Copilot and your employees can use.

Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...

Drew Madelung

Scaling API-first – The story of a global engineering organization Ian Reasor, Senior Computer Scientist - Adobe Radu Cotescu, Senior Computer Scientist - Adobe Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe

apidays

Partners Life - Insurer Innovation Award 2024

The Digital Insurer

Created by Mozilla Research in 2012 and now part of Linux Foundation Europe, the Servo project is an experimental rendering engine written in Rust. It combines memory safety and concurrency to create an independent, modular, and embeddable rendering engine that adheres to web standards. Stewardship of Servo moved from Mozilla Research to the Linux Foundation in 2020, where its mission remains unchanged. After some slow years, in 2023 there has been renewed activity on the project, with a roadmap now focused on improving the engine’s CSS 2 conformance, exploring Android support, and making Servo a practical embeddable rendering engine. In this presentation, Rakhi Sharma reviews the status of the project, our recent developments in 2023, our collaboration with Tauri to make Servo an easy-to-use embeddable rendering engine, and our plans for the future to make Servo an alternative web rendering engine for the embedded devices industry. (c) Embedded Open Source Summit 2024 April 16-18, 2024 Seattle, Washington (US) https://events.linuxfoundation.org/embedded-open-source-summit/ https://ossna2024.sched.com/event/1aBNF/a-year-of-servo-reboot-where-are-we-now-rakhi-sharma-igalia

A Year of the Servo Reboot: Where Are We Now?

Igalia

Strategies for Landing an Oracle DBA Job as a Fresher

Remote DBA Services

GenAI Risks & Security Meetup 01052024.pdf

lior mazor

Recently uploaded (20)

TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery

Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...

Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024

MINDCTI Revenue Release Quarter One 2024

ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke

Powerful Google developer tools for immediate impact! (2023-24 C)

Connector Corner: Accelerate revenue generation using UiPath API-centric busi...

AWS Community Day CPH - Three problems of Terraform

Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...

Bajaj Allianz Life Insurance Company - Insurer Innovation Award 2024

Top 5 Benefits OF Using Muvi Live Paywall For Live Streams

Why Teams call analytics are critical to your entire business

Automating Google Workspace (GWS) & more with Apps Script

Real Time Object Detection Using Open CV

Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...

Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe

Partners Life - Insurer Innovation Award 2024

A Year of the Servo Reboot: Where Are We Now?

Strategies for Landing an Oracle DBA Job as a Fresher

GenAI Risks & Security Meetup 01052024.pdf

How Apache Drives Music Recommendations At Spotify

1. HowApache Drives Music Recommendations At Spotify Josh Baer (jbx@spotify.com) Note:The view expressed is my own and does not necessarily represent that of Spotify

2. WhoAm I? • Technical Product Owner at Spotify • Working with batch and fast processing infrastructure @l_phant

3. Music Discovery in the 90s

4. What is Spotify? • Music Streaming Service • Launched in 2008 • Free and PremiumTiers • Available in 58 Countries

5. 75+ MillionActive Users

6. 30+ Million Songs

7. 1+ Billion Plays/Day

8. Music Recommendations with Apache

10.

11. How do we recommend a personalized playlist of new music to 75+ million users?

12. 10.123.133.333 - - [Mon, 3 June 2015 11:31:33 GMT] "GET /api/admin/job/aggregator/status HTTP/1.1" 200 1847 "https://my.analytics.app/ admin" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.81 Safari/537.36" 10.123.133.222 - - [Mon, 3 June 2015 11:31:43 GMT] "GET /api/admin/job/aggregator/status HTTP/1.1" 200 1984 "https://my.analytics.app/ admin" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.81 Safari/537.36” 10.123.133.222 - - [Mon, 3 June 2015 11:33:02 GMT] "GET /dashboard/courses/1291726 HTTP/1.1" 304 - "https://my.analytics.app/admin" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.81 Safari/537.36" 10.321.145.111 - - [Mon, 3 June 2015 11:33:03 GMT] "GET /api/loggedInUser HTTP/1.1" 304 - "https://my.analytics.app/dashboard/courses/ 1291726" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.81 Safari/537.36" 10.112.322.111 - - [Mon, 3 June 2015 11:33:03 GMT] "POST /api/instrumentation/events/new HTTP/1.1" 200 2 "https://my.analytics.app/ dashboard/courses/1291726" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.81 Safari/537.36” 10.123.133.222 - - [Mon, 3 June 2015 11:33:02 GMT] "GET /dashboard/courses/1291726 HTTP/1.1" 304 - "https://my.analytics.app/admin" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.81 Safari/537.36" It begins with a log

13.

14.

15. Apache Kafka at Spotify •340 Kafka-related nodes •30TB/day from logs

16.

17. How do we store TBs of new data every data?

18.

19. Apache Hadoop at Spotify • 1700 Nodes • 60 PB of Data • 70TB of Memory • Over 1 Million jobs run in Q3, 2015

20. ProcessingGrowth 150% 250% 350% 450% 550% Q4-2013 Q1-2014 Q2-2014 Q3-2014 Q4-2014 Q1-2015 Q2-2015 Q3-2015 Hadoop at Spotify

21. Processing Toolbox • Apache Crunch • Scalding • Apache Hive • Apache Spark • Apache Storm • Hadoop Streaming • Apache Pig

22. Storage Formats • ApacheAvro • Apache Parquet

23. How do we personalize the playlists?

24.

25. Collaborative Filtering Justin Bieber Drake Avicii Major Lazer Anna Listened Listened Gustav Listened Listened Listened Mary Listened Listened Listened Listened Michael Listened ListenedSuggest

26.

27. How do we serve new playlists to all our users every week?

28.

29. Apache Cassandra at Spotify • Number of Clusters: 113 • Number of Machines: 1155 • Largest Cluster: 60 Nodes

30. Driven By Data

31. Driven ByApache

32. Thank YOU for your contributions to Apache products!

33. One Last Thing…

34. Spotify Luigi •Workflow Manager •Over 150 contributors •Used by 10s, possibly 100s of companies

35. Maybe…Apache Luigi? Sponsors/mentors/contributors wanted!

36. Think this stuff is interesting? We have a great time building it! spotify.com/jobs

37. Better Spotify ML Presentations • Algorithmic Music Recommendations at Spotify (Chris Johnson) • Interactive Recommender Systems with Netflix and Spotify (Chris Johnson) • Music recommendations @ MLConf 2014 (Erik Bernhardsson) • Machine learning @ Spotify (Andy Sloane) • Recommending music on Spotifywith deep learning (Sander Dieleman) • Scala Data Pipelines @ Spotify (Neville Li) • Spotify's Music Recommendations LambdaArchitecture (Esh Kumar and Emily Samuels)

How Apache Drives Music Recommendations At Spotify

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to How Apache Drives Music Recommendations At Spotify

Similar to How Apache Drives Music Recommendations At Spotify (20)

Recently uploaded

Recently uploaded (20)

How Apache Drives Music Recommendations At Spotify