Spotify Teknikdagarna
•Download as PPTX, PDF•
0 likes•236 views
Spotify generates 150 terabytes of user data every day from over 75 million users interacting with the service through streaming music, clicks, and other activities. To make sense of this massive amount of data, Spotify utilizes a Hadoop cluster with 1600 nodes across multiple data centers to combine, store, and analyze the data using techniques like ALS matrix factorization to power personalized music recommendations.
Report
Share
Report
Share
Recommended
Yaniv Mor - Xplenty - big data new physics
Xplenty provides a cloud-based, Hadoop-as-a-Service solution that allows organizations of any size to easily harness the power of big data without needing Hadoop skills. Their platform includes visualization tools and support for both structured and unstructured data stored in cloud databases.
CARARE: Data adventures in heritage science
The document discusses challenges related to managing large volumes of heritage science data. It notes that while technology has increased data richness and re-use, it has also increased issues around data quality. Better quality content is now needed, including higher resolution data and richer metadata. The document explores approaches to assessing metadata quality, such as completeness and accuracy metrics. It proposes using machine learning techniques like clustering to identify topics needing enrichment based on textual features of metadata elements. The overall goal is to improve processes for quality-driven metadata enrichment.
Spotify architecture - Pressing play
Slides from a talk I gave at Scandinavian Developers Conference 2012 on the architecture of Spotify. The slides follows a story of playing a track and the steps to get there.
Spotify: Playing for millions, tuning for more
Barcelona Developers Conference presentation by Nick Barkas and David Poblador i Garcia, 18 November 2011. How we manage a huge collection of servers and some of the technologies we use for building a scalable, high performance music streaming service.
JavaScript @ Spotify (Felipe Ribeiro Technology Stream)
Lviv IT Arena is a conference specially designed for programmers, designers, developers, top managers, inverstors, entrepreneurs and startuppers. Annually it takes place at the beginning of October in Lviv at Arena Lviv stadium. In 2016 the conference gathered more than 1800 participants and over 100 speakers from companies like Microsoft, Philips, Twitter, UBER and IBM. More details about the conference at itarena.lviv.ua.
Evolution of Spotify's ad architecture (Qcon 2016 Shanghai)
Evolution of Spotify's ad architecture over the years to keep up with changing business needs, build new products and leverage improvement in cloud and infrastructure capabilities.
MILLIONS EVENT DELIVERY WITH CLOUD PUB / SUB
This document provides an overview of Google Cloud Pub/Sub including key concepts, benefits, and usage examples. It discusses why message queues are important, early players in the space, and how Cloud Pub/Sub provides scalability, flexibility and reliability. The document also summarizes Spotify's migration from their old event delivery system to using Cloud Pub/Sub and the results of producer load and consumer stability tests.
Spotify: Data center & Backend buildout
In this talk Emil Fredriksson and David Poblador i Garcia explain how Spotify builds its infrastructure in order to deliver millions of songs to millions of users.
We explain how we manage to support our development teams to build features by developing a highly scalable infrastructure.
Recommended
Yaniv Mor - Xplenty - big data new physics
Xplenty provides a cloud-based, Hadoop-as-a-Service solution that allows organizations of any size to easily harness the power of big data without needing Hadoop skills. Their platform includes visualization tools and support for both structured and unstructured data stored in cloud databases.
CARARE: Data adventures in heritage science
The document discusses challenges related to managing large volumes of heritage science data. It notes that while technology has increased data richness and re-use, it has also increased issues around data quality. Better quality content is now needed, including higher resolution data and richer metadata. The document explores approaches to assessing metadata quality, such as completeness and accuracy metrics. It proposes using machine learning techniques like clustering to identify topics needing enrichment based on textual features of metadata elements. The overall goal is to improve processes for quality-driven metadata enrichment.
Spotify architecture - Pressing play
Slides from a talk I gave at Scandinavian Developers Conference 2012 on the architecture of Spotify. The slides follows a story of playing a track and the steps to get there.
Spotify: Playing for millions, tuning for more
Barcelona Developers Conference presentation by Nick Barkas and David Poblador i Garcia, 18 November 2011. How we manage a huge collection of servers and some of the technologies we use for building a scalable, high performance music streaming service.
JavaScript @ Spotify (Felipe Ribeiro Technology Stream)
Lviv IT Arena is a conference specially designed for programmers, designers, developers, top managers, inverstors, entrepreneurs and startuppers. Annually it takes place at the beginning of October in Lviv at Arena Lviv stadium. In 2016 the conference gathered more than 1800 participants and over 100 speakers from companies like Microsoft, Philips, Twitter, UBER and IBM. More details about the conference at itarena.lviv.ua.
Evolution of Spotify's ad architecture (Qcon 2016 Shanghai)
Evolution of Spotify's ad architecture over the years to keep up with changing business needs, build new products and leverage improvement in cloud and infrastructure capabilities.
MILLIONS EVENT DELIVERY WITH CLOUD PUB / SUB
This document provides an overview of Google Cloud Pub/Sub including key concepts, benefits, and usage examples. It discusses why message queues are important, early players in the space, and how Cloud Pub/Sub provides scalability, flexibility and reliability. The document also summarizes Spotify's migration from their old event delivery system to using Cloud Pub/Sub and the results of producer load and consumer stability tests.
Spotify: Data center & Backend buildout
In this talk Emil Fredriksson and David Poblador i Garcia explain how Spotify builds its infrastructure in order to deliver millions of songs to millions of users.
We explain how we manage to support our development teams to build features by developing a highly scalable infrastructure.
Ads Personalization at Spotify - NYC Data Engineering 10/23
Spotify engineers (Kinshuk Mishra and Noel Cody) share their experiences about building personalized ad experiences for users through iterative engineering and product development. The slide explains their process of continuous problem discovery, hypothesis generation, product development and experimentation. They deep dive into the specific ad personalization problems Spotify is solving and explain their data infrastructure technology stack in detail. They also explain how they've experimented various product hypothesis and iteratively evolved their infrastructure to keep up with the product requirements.
Cross platform web app development
This document discusses Spotify's use of web apps for cross-platform development. Spotify uses web technologies like HTML, CSS, and JavaScript to build apps that can run across desktop, web, and mobile clients. Key aspects discussed include using iframes for sandboxing apps, cross-origin communication via postMessage, and strategies for communication between different browser tabs using localStorage. Performance is also addressed, such as techniques to reduce iframe overhead and improve notification delays.
Google Cloud Platform: Prototype ->Production-> Planet scale
As one of Big Data’s Founding Fathers, Google explored the technological changes we faced over the past 10 years and present their solutions to the new data challenges within the Google Cloud ecosystem
From stream to recommendation using apache beam with cloud pubsub and cloud d...
Spotify talk at GCP NEXT 2016, March 24, 2016.
Google Cloud Pubsub, Dataflow, BigQuery and Scio.
https://github.com/spotify/scio
Data at Spotify
Danielle Jabin is a data engineer at Spotify who works on A/B testing infrastructure. She describes Spotify's big data landscape, which includes over 40 million active users generating 1.5 TB of compressed data per day. Spotify collects this user data using Kafka for high-volume data collection, processes it using Hadoop on a large cluster, and stores aggregates in databases like PostgreSQL and Cassandra for analytics and visualization.
Google Cloud Platform Empowers TensorFlow and Machine Learning
Google Cloud Platform empowers TensorFlow and machine learning by providing scalable computing resources and APIs. It allows developers to build neural networks with TensorFlow, and easily integrate pre-trained machine learning models into applications using Cloud Vision and Speech APIs. Cloud Machine Learning offers a managed service for distributed TensorFlow training and prediction at scale in the cloud.
Spotify: behind the scenes
Slides from my guest lecture at Uppsala University, class of Distributed Systems, on the 10th of February 2011
The Web3 Data Economy: Ocean Protocol
1. The document discusses the need for a "Web3 Data Economy" using blockchain and tokens to open up data access and incentivize data sharing, similar to how cryptocurrencies have opened up and incentivized money transfers.
2. It proposes that Ocean Protocol can serve as the foundational platform and token for this new data economy, fulfilling similar roles that Ethereum plays in the cryptocurrency economy like being the unit of exchange and hosting applications.
3. The goal is to connect data owners with AI companies and researchers to provide more open access to data and tools while protecting privacy and properly incentivizing all parties, helping to shift power away from current large data platforms.
The evolution of the big data platform @ Netflix (OSCON 2015)
The document summarizes the evolution of Netflix's big data platform to meet the challenges of their growing scale. Key points include:
- Netflix now has over 65 million members in over 50 countries and supports over 1000 devices. They stream over 10 billion hours of content per quarter.
- Their traditional business intelligence stack could no longer meet the demands of scale. They transitioned to using AWS services like S3 for storage and open source tools like Kafka, Cassandra, and Parquet to enable real-time analytics and machine learning on their massive data volumes.
- Netflix has adopted an open source-first strategy and contributes back to the community as their own tools evolve to meet processing needs and achieve the necessary scale to
Big data and APIs for PHP developers - SXSW 2011
Slides from the Big Data presentation by Bradley Holt, Laura Thomson, Julie Steele, Eli White, Dennis Yang, and David Zülke
Data Monetization
This presentation focuses on the “Data - Big Data - Bigger Data” and the Challenges, Opportunities and Solutions from these trends.
What are the Challenges this massive data brings to the table?
What are the opportunities this data provide ?
Some solutions on how to handle this data.
Sound cloud - User & Partner Conference - AT Internet
This document discusses SoundCloud's use of Amazon Redshift for big data analytics. It summarizes how SoundCloud uses Hadoop to store large amounts of listener data from multiple sources, but faced challenges with data silos and slow access. It migrated to using Amazon Redshift for its data warehouse which provides fast query performance on petabytes of data. It developed ETL processes to load data from source systems into Redshift, build a data model, and create aggregated fact tables and reporting data cubes for exploration and reporting.
MAPCI at Skåne:Sthlm 2013
Presentation of the newly created research and innovation institute in Lund as well as the background and rationale for doing so.
Stream Reasoning: a summary of ten years of research and a vision for the nex...
Stream Reasoning: a summary of ten years of research and a vision for the nex...Emanuele Della Valle
Stream reasoning studies the application of inference techniques to data characterised by being highly dynamic. It can find application in several settings, from Smart Cities to Industry 4.0, from Internet of Things to Social Media analytics. This year stream reasoning turns ten, and this talk analyses its growth. In the first part, it traces the main results obtained so far, by presenting the most prominent studies. It starts by an overview of the most relevant studies developed in the context of semantic web, and then it extends the analysis to include contributions from adjacent areas, such as database and artificial intelligence. Looking at the past is useful to prepare for the future: the second part presents a set of open challenges and issues that stream reasoning will face in the next future.A Empresa na Era da Informação Extrema
This document discusses how algorithms can perform complex tasks like composing music and driving cars. It also discusses how companies like Amazon and Netflix generate huge amounts of data daily and use Amazon Web Services like S3, EMR and MapReduce to analyze and gain insights from petabytes of data. Technologies like Hadoop and MapReduce allow massive datasets to be processed across large clusters of servers in parallel.
Presentation on BigData by Swapnaja
The document discusses big data and how it is being generated from various sources like social media, sensors, and mobile devices. It describes the key characteristics of big data known as the three V's - volume, velocity and variety. It then explains how Hadoop uses HDFS for storage and MapReduce for processing large datasets in parallel across clusters of computers. The conclusion states that big data presents both opportunities and challenges for industries in creating value from large and diverse datasets.
Big Data Analytics(Intro,Hadoop Map Reduce,Mahout,K-means clustering,H-base)
Big Data Analytics(Intro,Hadoop Map Reduce,Mahout,K-means clustering,H-base)MIT College Of Engineering,Pune
A very categorized presentation about big data analytics Various topics like Introduction to Big Data,Hadoop,HDFS Map Reduce, Mahout,K-means Algorithm,H-Base are explained very clearly in simple language for everyone to understand easily.Big Data Use Cases and Solutions in the AWS Cloud
The AWS cloud computing platform has disrupted big data. Managing big data applications used to be for only well-funded research organizations and large corporations, but not any longer. Hear from Ben Butler, Big Data Solutions Marketing Manager for AWS, to learn how our customers are using big data services in the AWS cloud to innovate faster than ever before. Not only is AWS technology available to everyone, but it is self-service, on-demand, and featuring innovative technology and flexible pricing models at low cost with no commitments. Learn from customer success stories, as Ben shares real-world case studies describing the specific big data challenges being solved on AWS. We will conclude with a discussion around the tutorials, public datasets, test drives, and our grants program - all of the resources needed to get you started quickly.
Big Data: Come usare le Machine Learning per migliorare il business
L'obiettivo della presentazione è quello di sensibilizzare le aziende nella necessità di introdurre progetti di Big Data in azienda per riconoscere in modo puntuale il proprio utente.
Il digital ci permette di raccogliere l'imporonta digitale degli utenti ma richiede di ottenere la loro fiducia per comprendere i loro interessi, abitudini, network etc.
Oggi analizziamo dati in tempo reale ma dobbiamo essere in grado, per competere sull mercato, di prevedere gli interessi ed esigenze degli utenti.
Per fare questo ci servono sistemi di Machine Learning in grado di apprendere dai dati ed evolversi automaticamente.
Durante la presentazione sono stati illustrati i metodi di apprendimento supervisionato e non supervisionato e come il Servizio Machine Learning di Amazon Web Services sia in grado di semplificare un'attività complessa come la costruzione di un modello di Machine Learning.
Big Data
Big data refers to large and complex datasets that are difficult to process using traditional database management tools. It is characterized by high volume, velocity, and variety of both structured and unstructured data. The volume of data being created is growing exponentially - from 2003 until now, the amount of data created each day has increased from 5 billion gigabytes to the same amount being created every 10 minutes. Big data tools are used by companies like eBay, Amazon, and Walmart to handle high volumes of customer transactions and queries, with some companies storing petabytes or more of data. Common big data software includes Hadoop, Spark, and Splunk, while real-world examples demonstrate big data's use in fields like healthcare, retail, and social
Big Data Analytics(concepts of hadoop mapreduce,mahout,k-means clustering,hbase)
Big Data Analytics(concepts of hadoop mapreduce,mahout,k-means clustering,hbase)MIT College Of Engineering,Pune
A very categorized presentation about big data analytics with very easy language to understand the basics easily..Itunes
Web 2.0 refers to a transition of the World Wide Web from static websites to dynamic applications that emphasize online collaboration and sharing. RSS allows users to subscribe to favorite websites and view updated content. Social networking represents relationships and connections between individuals. Wikis allow easy collaborative editing of content without registration. Ajax makes web pages more interactive by exchanging small amounts of data without reloading pages.
More Related Content
Viewers also liked
Ads Personalization at Spotify - NYC Data Engineering 10/23
Spotify engineers (Kinshuk Mishra and Noel Cody) share their experiences about building personalized ad experiences for users through iterative engineering and product development. The slide explains their process of continuous problem discovery, hypothesis generation, product development and experimentation. They deep dive into the specific ad personalization problems Spotify is solving and explain their data infrastructure technology stack in detail. They also explain how they've experimented various product hypothesis and iteratively evolved their infrastructure to keep up with the product requirements.
Cross platform web app development
This document discusses Spotify's use of web apps for cross-platform development. Spotify uses web technologies like HTML, CSS, and JavaScript to build apps that can run across desktop, web, and mobile clients. Key aspects discussed include using iframes for sandboxing apps, cross-origin communication via postMessage, and strategies for communication between different browser tabs using localStorage. Performance is also addressed, such as techniques to reduce iframe overhead and improve notification delays.
Google Cloud Platform: Prototype ->Production-> Planet scale
As one of Big Data’s Founding Fathers, Google explored the technological changes we faced over the past 10 years and present their solutions to the new data challenges within the Google Cloud ecosystem
From stream to recommendation using apache beam with cloud pubsub and cloud d...
Spotify talk at GCP NEXT 2016, March 24, 2016.
Google Cloud Pubsub, Dataflow, BigQuery and Scio.
https://github.com/spotify/scio
Data at Spotify
Danielle Jabin is a data engineer at Spotify who works on A/B testing infrastructure. She describes Spotify's big data landscape, which includes over 40 million active users generating 1.5 TB of compressed data per day. Spotify collects this user data using Kafka for high-volume data collection, processes it using Hadoop on a large cluster, and stores aggregates in databases like PostgreSQL and Cassandra for analytics and visualization.
Google Cloud Platform Empowers TensorFlow and Machine Learning
Google Cloud Platform empowers TensorFlow and machine learning by providing scalable computing resources and APIs. It allows developers to build neural networks with TensorFlow, and easily integrate pre-trained machine learning models into applications using Cloud Vision and Speech APIs. Cloud Machine Learning offers a managed service for distributed TensorFlow training and prediction at scale in the cloud.
Spotify: behind the scenes
Slides from my guest lecture at Uppsala University, class of Distributed Systems, on the 10th of February 2011
Viewers also liked (7)
Ads Personalization at Spotify - NYC Data Engineering 10/23
Ads Personalization at Spotify - NYC Data Engineering 10/23
Google Cloud Platform: Prototype ->Production-> Planet scale
Google Cloud Platform: Prototype ->Production-> Planet scale
From stream to recommendation using apache beam with cloud pubsub and cloud d...
From stream to recommendation using apache beam with cloud pubsub and cloud d...
Google Cloud Platform Empowers TensorFlow and Machine Learning
Google Cloud Platform Empowers TensorFlow and Machine Learning
Similar to Spotify Teknikdagarna
The Web3 Data Economy: Ocean Protocol
1. The document discusses the need for a "Web3 Data Economy" using blockchain and tokens to open up data access and incentivize data sharing, similar to how cryptocurrencies have opened up and incentivized money transfers.
2. It proposes that Ocean Protocol can serve as the foundational platform and token for this new data economy, fulfilling similar roles that Ethereum plays in the cryptocurrency economy like being the unit of exchange and hosting applications.
3. The goal is to connect data owners with AI companies and researchers to provide more open access to data and tools while protecting privacy and properly incentivizing all parties, helping to shift power away from current large data platforms.
The evolution of the big data platform @ Netflix (OSCON 2015)
The document summarizes the evolution of Netflix's big data platform to meet the challenges of their growing scale. Key points include:
- Netflix now has over 65 million members in over 50 countries and supports over 1000 devices. They stream over 10 billion hours of content per quarter.
- Their traditional business intelligence stack could no longer meet the demands of scale. They transitioned to using AWS services like S3 for storage and open source tools like Kafka, Cassandra, and Parquet to enable real-time analytics and machine learning on their massive data volumes.
- Netflix has adopted an open source-first strategy and contributes back to the community as their own tools evolve to meet processing needs and achieve the necessary scale to
Big data and APIs for PHP developers - SXSW 2011
Slides from the Big Data presentation by Bradley Holt, Laura Thomson, Julie Steele, Eli White, Dennis Yang, and David Zülke
Data Monetization
This presentation focuses on the “Data - Big Data - Bigger Data” and the Challenges, Opportunities and Solutions from these trends.
What are the Challenges this massive data brings to the table?
What are the opportunities this data provide ?
Some solutions on how to handle this data.
Sound cloud - User & Partner Conference - AT Internet
This document discusses SoundCloud's use of Amazon Redshift for big data analytics. It summarizes how SoundCloud uses Hadoop to store large amounts of listener data from multiple sources, but faced challenges with data silos and slow access. It migrated to using Amazon Redshift for its data warehouse which provides fast query performance on petabytes of data. It developed ETL processes to load data from source systems into Redshift, build a data model, and create aggregated fact tables and reporting data cubes for exploration and reporting.
MAPCI at Skåne:Sthlm 2013
Presentation of the newly created research and innovation institute in Lund as well as the background and rationale for doing so.
Stream Reasoning: a summary of ten years of research and a vision for the nex...
Stream Reasoning: a summary of ten years of research and a vision for the nex...Emanuele Della Valle
Stream reasoning studies the application of inference techniques to data characterised by being highly dynamic. It can find application in several settings, from Smart Cities to Industry 4.0, from Internet of Things to Social Media analytics. This year stream reasoning turns ten, and this talk analyses its growth. In the first part, it traces the main results obtained so far, by presenting the most prominent studies. It starts by an overview of the most relevant studies developed in the context of semantic web, and then it extends the analysis to include contributions from adjacent areas, such as database and artificial intelligence. Looking at the past is useful to prepare for the future: the second part presents a set of open challenges and issues that stream reasoning will face in the next future.A Empresa na Era da Informação Extrema
This document discusses how algorithms can perform complex tasks like composing music and driving cars. It also discusses how companies like Amazon and Netflix generate huge amounts of data daily and use Amazon Web Services like S3, EMR and MapReduce to analyze and gain insights from petabytes of data. Technologies like Hadoop and MapReduce allow massive datasets to be processed across large clusters of servers in parallel.
Presentation on BigData by Swapnaja
The document discusses big data and how it is being generated from various sources like social media, sensors, and mobile devices. It describes the key characteristics of big data known as the three V's - volume, velocity and variety. It then explains how Hadoop uses HDFS for storage and MapReduce for processing large datasets in parallel across clusters of computers. The conclusion states that big data presents both opportunities and challenges for industries in creating value from large and diverse datasets.
Big Data Analytics(Intro,Hadoop Map Reduce,Mahout,K-means clustering,H-base)
Big Data Analytics(Intro,Hadoop Map Reduce,Mahout,K-means clustering,H-base)MIT College Of Engineering,Pune
A very categorized presentation about big data analytics Various topics like Introduction to Big Data,Hadoop,HDFS Map Reduce, Mahout,K-means Algorithm,H-Base are explained very clearly in simple language for everyone to understand easily.Big Data Use Cases and Solutions in the AWS Cloud
The AWS cloud computing platform has disrupted big data. Managing big data applications used to be for only well-funded research organizations and large corporations, but not any longer. Hear from Ben Butler, Big Data Solutions Marketing Manager for AWS, to learn how our customers are using big data services in the AWS cloud to innovate faster than ever before. Not only is AWS technology available to everyone, but it is self-service, on-demand, and featuring innovative technology and flexible pricing models at low cost with no commitments. Learn from customer success stories, as Ben shares real-world case studies describing the specific big data challenges being solved on AWS. We will conclude with a discussion around the tutorials, public datasets, test drives, and our grants program - all of the resources needed to get you started quickly.
Big Data: Come usare le Machine Learning per migliorare il business
L'obiettivo della presentazione è quello di sensibilizzare le aziende nella necessità di introdurre progetti di Big Data in azienda per riconoscere in modo puntuale il proprio utente.
Il digital ci permette di raccogliere l'imporonta digitale degli utenti ma richiede di ottenere la loro fiducia per comprendere i loro interessi, abitudini, network etc.
Oggi analizziamo dati in tempo reale ma dobbiamo essere in grado, per competere sull mercato, di prevedere gli interessi ed esigenze degli utenti.
Per fare questo ci servono sistemi di Machine Learning in grado di apprendere dai dati ed evolversi automaticamente.
Durante la presentazione sono stati illustrati i metodi di apprendimento supervisionato e non supervisionato e come il Servizio Machine Learning di Amazon Web Services sia in grado di semplificare un'attività complessa come la costruzione di un modello di Machine Learning.
Big Data
Big data refers to large and complex datasets that are difficult to process using traditional database management tools. It is characterized by high volume, velocity, and variety of both structured and unstructured data. The volume of data being created is growing exponentially - from 2003 until now, the amount of data created each day has increased from 5 billion gigabytes to the same amount being created every 10 minutes. Big data tools are used by companies like eBay, Amazon, and Walmart to handle high volumes of customer transactions and queries, with some companies storing petabytes or more of data. Common big data software includes Hadoop, Spark, and Splunk, while real-world examples demonstrate big data's use in fields like healthcare, retail, and social
Big Data Analytics(concepts of hadoop mapreduce,mahout,k-means clustering,hbase)
Big Data Analytics(concepts of hadoop mapreduce,mahout,k-means clustering,hbase)MIT College Of Engineering,Pune
A very categorized presentation about big data analytics with very easy language to understand the basics easily..Itunes
Web 2.0 refers to a transition of the World Wide Web from static websites to dynamic applications that emphasize online collaboration and sharing. RSS allows users to subscribe to favorite websites and view updated content. Social networking represents relationships and connections between individuals. Wikis allow easy collaborative editing of content without registration. Ajax makes web pages more interactive by exchanging small amounts of data without reloading pages.
Itunes
Web 2.0 refers to a transition of the World Wide Web from static websites to dynamic applications that emphasize online collaboration and sharing. RSS allows users to subscribe to favorite websites and view updated content. Social networking represents relationships and connections between individuals. Wikis allow easy collaborative editing of content without registration. Ajax makes web pages more interactive by exchanging small amounts of data without reloading pages.
Big Data
This document discusses big data use cases that use Amazon Elastic MapReduce (EMR) and Hadoop. It provides examples of companies like a big box retailer, Etsy, Yelp, and Foursquare using EMR to perform clickstream analysis, recommendations, process large amounts of log data, and analyze user check-in patterns at massive scales impossible without these cloud big data tools. EMR allows these companies to easily experiment and process massive datasets across large clusters of servers in an on-demand, flexible, and cost-effective manner in the cloud.
Big data use cases in the cloud presentation
This document discusses big data use cases that use Amazon Elastic MapReduce (EMR) and Hadoop on the cloud. It provides examples of companies like a big box retailer, Etsy, Yelp, and Foursquare using EMR to perform clickstream analysis, recommendations, process large amounts of log data, and analyze user check-in patterns from terabytes of data. EMR allows these companies to easily run Hadoop jobs and experiments on large, scalable cloud clusters and lower the costs of operating distributed data systems.
Getting your head around big data
The document provides an overview of the history of data from ancient times to modern technologies and the challenges of harnessing big data. It discusses how data is generated from a variety of sources at different frequencies and velocities and not all data is useful or accurate, highlighting issues of veracity. It then presents solutions for storing, processing, analyzing and visualizing large datasets including MapReduce, high-scale databases, distributed processing, data visualization tools, and end-to-end platforms.
A Brief History Of Data
This document provides a brief history of data from ancient times to the present day. It discusses how humans started counting and recording data visually over 20,000 years ago. Written language emerged around 3,500 BC allowing data to be recorded and transmitted. Major developments include the first library in 1250 BC to store data in mass, the origin of maps in 1150 BC, and using numbers and logic to derive insights from 100 BC to 350 BC. Significant milestones from 1600 to 1900 include the development of statistics, computers, programming languages, data standards, and the internet. Today's "big data" landscape is characterized by volume, variety, velocity, and veracity of data being created. The future will involve understanding data through insights
Similar to Spotify Teknikdagarna (20)
The evolution of the big data platform @ Netflix (OSCON 2015)
The evolution of the big data platform @ Netflix (OSCON 2015)
Sound cloud - User & Partner Conference - AT Internet
Sound cloud - User & Partner Conference - AT Internet
Stream Reasoning: a summary of ten years of research and a vision for the nex...
Stream Reasoning: a summary of ten years of research and a vision for the nex...
Big Data Analytics(Intro,Hadoop Map Reduce,Mahout,K-means clustering,H-base)
Big Data Analytics(Intro,Hadoop Map Reduce,Mahout,K-means clustering,H-base)
Big Data: Come usare le Machine Learning per migliorare il business
Big Data: Come usare le Machine Learning per migliorare il business
Big Data Analytics(concepts of hadoop mapreduce,mahout,k-means clustering,hbase)
Big Data Analytics(concepts of hadoop mapreduce,mahout,k-means clustering,hbase)
Spotify Teknikdagarna
- 1. How to make sense of 150 TB of data, every day Sebastian Widlund widlund@spotify.com
- 2. M 27M 53M 80M 2009 2010 2011 2012 2013 May'14 May'15 Spotify 75 million users 20 million paying users Founded in 2008 3+ billion dollars paid to rights holders 30+ million tracks 1.5+ billion playlists 1500+ employees
- 3. “Big data is a broad term for data sets so large or complex that traditional data processing applications are inadequate.” What is big data?
- 4. 14 TB of user/service-related log data per day Streams/clicks/interactions are being logged Expands to 150 TB every day Combining data sources
- 5. We utilise a cluster of 1600 nodes
- 6. Hadoop Data Center European Data Center American Data Center Internet Internet7 TB/day 7 TB/day Client Client
- 9. Approximate 60M users x 4M songs with 40 latent factors, ALS In short, minimise the cost function:
- 10. 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 Time of day Sunday Monday Tuesday Wednesday Thursday Friday Saturday
Editor's Notes
- Svårt att definiera exakt hur många låtar. Deduplicering, vi har ett eget team som jobbar med det 30 miljoner faktiska låtar, när man räknat bort dubbletter.
- High Volume, High Velocity, High Variety Katalog, 30 miljoner rader (inte så mycket). 30^2 (n^2) jämförelse.
- billions of lines of data every day Anonymizing data, making sure that all data is according to privacy concerns. One machine, 160mb/s, 10 days to read in 150 TB of data So how do we do it?
- 60 PB of disk space 68 TB of RAM (42gb per server) 30k CPU cores
- Logs sent from clients - Sent to EU/US data centre
- Very complex Lots of different services App is a small part of everything “The app does not just work”
- Example of the “Discovery” Logs come from the client, pass through hadoop into a service that recommends music and surfaces back to user
- Example of the “Discovery” Logs come from the client, pass through hadoop into a service that recommends music and surfaces back to user
- Track usage, but important for breakdown. The reason we save a lot of data