Spotify's journey to GCP

•

0 likes•722 views

Spotify is moving their entire backend to GCP. You will get to hear their challenges, problems and also success stories and what to think about when choosing and moving services into the cloud.

Technology

~# whoami
Alexey Lapitsky / @vzctl
alexey@spotify.com
Site Reliability Engineer @Spotify
PM for the service track of the GCP migration

Agenda
● History
● Why GCP?
● The biggest mistake
● Learnings
● Questions!

Spotify’s cloud history
2010
Manual VM provisioning
2012
Cloudstack, reached ~700 instances
2013
Amazon EC2
2015
Google Cloud Platform

Current status
● 4 data centers, ~10k physical servers
● Micro-service architecture, ~1k different services
● Mostly commodity hardware (Dell)
● Fully automated, self-service server provisioning
Moving everything to GCP

Why GCP?
● BigData
● Culture
● Timing
● Partnership

Spoticloud
CLI
REST
DB
Cloudstack
AWS
Puppet
DNS
Server
● CLI Tool + REST Server
● User authentication with ssh agent: github.com/spotify/crtauth
SSH
Jumphost

Looks awesome!
● No need to distribute AWS accounts
● No need to manage ACLs
● Extremely easy to use
~# spcloud create --name web --project alexey

… or not really :(
● Testing environment, shamelessly misused for production
● No accountability or a proper ownership tracking
○ 60% of instances were wasting money
● Access only to the basic cloud primitives
○ Which are already available on-premise
Currently being deprecated.

Things we learned
● VPN is tricky
● GCP network is great
● Keep an eye on quotas
● Live VM migration!
● Think about ACL mgmt in advance
● BigTable, GCS, BigQuery are awesome

What's hot

Industrial Light & Magic

Amazon Web Services

Managing Your Cloud Spend With PlanForCloud - RightScale Compute 2013

RightScale

Serverless and AI: Orit Nissan-Messing, Iguazio, Serverless NYC 2018

iguazio

Building the Serverless Container Experience: Kevin McGrath, Spotinst, Server...

iguazio

Real-world architectures are different from code samples and small examples, and as we build more complex and mature Serverless architectures, we often encounter unexpected challenges. This talk will start by discussing the challenges involved in building data processing architectures using stateless infrastructures. We'll review patterns from event-driven architectures, see how they apply to Serverless architectures and propose practical solutions to common pain-points.

The Problem is Data: Gwen Shapira, Confluent, Serverless NYC 2018

iguazio

Embracing Serverless with Google

Joseph Lust

Although Hadoop might be the first thing to come in your mind when you think of processing large data sets, it is not always the best solution for your Big Data problems. Hadoop might be the right choice for batch-processing big data, but when it comes to real-time data processing there are other architectures and tools to consider. This TechTalk shows the need behind solving real time data problems and explains the Lambda architecture, mentioning Druid as an example, and the simpler and less expensive "event sourcing model".

JOSA TechTalk - Lambda architecture and real-time processing

Mahmoud Jalajel

IronSource Atom - Redshift - Lessons Learned

Idan Tohami

Building big data applications on AWS by Ran Tessler

Idan Tohami

A novel building infrastructure as code

Yu-Lin Huang

Trying out the Go language with Google App Engine

Lynn Langit

Although a giant player in anything software related Google Cloud still feels a bit under appreciated. How did it get where it is now? What are its core strengths? Most of all, we want to provide a glimpse of the future in determining major shifts in Cloud computing. Every company is a data company but their data still remains under-utilised due to a lack of execution power, let’s find this power. Due to Cloud pricing models efficient software engineering is gaining in importance, let’s unlock this efficiency. Hybrid and multi-Cloud is easily one of the largest investment domains in the Cloud world. Let’s find out why and see how we can stay vendor neutral as much as possible.

Openbar Kontich // Google Cloud: past, present and the (oh so sweet) future b...

Openbar

Cost Optimization Best Practices from Trend Micro

Cliff Chao-kuan Lu

Is Serverless just running functions in a cloud? It’s more than that! Serverless computing refers to the concept of building and running applications that do not require server management. It describes a deployment model where applications, bundled as one or more functions, are uploaded to a platform and then executed, scaled, and billed in response to the exact demand needed at the moment. During the talk I’ll show how to use Knative both on Kubernetes and on OpenShift platform. Hopefully we will see why your organization should consider using Knative as one of its primary deployments models on hybrid cloud world.

Knative, Serverless on Kubernetes, and Openshift

Chris Suszyński

Serverless Architecture GCP In Production

Oliver Fierro

Terraform

Otto Jongerius

Using HashiCorp’s Terraform to build your infrastructure on AWS - Pop-up Loft...

Amazon Web Services

Hands on Compute Engine

Simon Su

NLUUG print conference May 26 2016

Igmar Palsenberg

Skillenza is back with another game-changing virtual hackathon for you. Seize this amazing opportunity to create projects on serverless architecture. For those of you who are not acquainted with it, serverless architectures are system designs that use third-party services to build and run applications. As developers, this helps you to gain better scalability and flexibility without needing any administration to manage infrastructure. So you can build quicker and at a reduced cost as well. https://skillenza.com/challenge/build-with-serverless-online-hackathon-aws

Skillenza Build with Serverless Challenge - Advanced Serverless Concepts

Dhaval Nagar

What's hot (20)

Industrial Light & Magic

Managing Your Cloud Spend With PlanForCloud - RightScale Compute 2013

Serverless and AI: Orit Nissan-Messing, Iguazio, Serverless NYC 2018

Building the Serverless Container Experience: Kevin McGrath, Spotinst, Server...

The Problem is Data: Gwen Shapira, Confluent, Serverless NYC 2018

Embracing Serverless with Google

JOSA TechTalk - Lambda architecture and real-time processing

IronSource Atom - Redshift - Lessons Learned

Building big data applications on AWS by Ran Tessler

A novel building infrastructure as code

Trying out the Go language with Google App Engine

Openbar Kontich // Google Cloud: past, present and the (oh so sweet) future b...

Cost Optimization Best Practices from Trend Micro

Knative, Serverless on Kubernetes, and Openshift

Serverless Architecture GCP In Production

Terraform

Using HashiCorp’s Terraform to build your infrastructure on AWS - Pop-up Loft...

Hands on Compute Engine

NLUUG print conference May 26 2016

Skillenza Build with Serverless Challenge - Advanced Serverless Concepts

Viewers also liked

How to migrate workloads to the google cloud platform

actualtechmedia

Google cloud big data summit master gcp big data summit la - 10-20-2015

Raj Babu

Scala Data Pipelines @ Spotify

Neville Li

From stream to recommendation using apache beam with cloud pubsub and cloud d...

Neville Li

Using CI for continuous delivery Part 3

Vishal Biyani

Using CI for continuous delivery Part 4

Vishal Biyani

Using CI for continuous delivery Part 2

Vishal Biyani

Using CI for continuous delivery Part 1

Vishal Biyani

Data at Spotify

Danielle Jabin

Big Data At Spotify

Adam Kawa

Netflix Global Cloud Architecture

Adrian Cockcroft

Growing up with agile - how the Spotify 'model' has evolved

Peter Antman

Viewers also liked (12)

How to migrate workloads to the google cloud platform

Google cloud big data summit master gcp big data summit la - 10-20-2015

Scala Data Pipelines @ Spotify

From stream to recommendation using apache beam with cloud pubsub and cloud d...

Using CI for continuous delivery Part 3

Using CI for continuous delivery Part 4

Using CI for continuous delivery Part 2

Using CI for continuous delivery Part 1

Data at Spotify

Big Data At Spotify

Netflix Global Cloud Architecture

Growing up with agile - how the Spotify 'model' has evolved

Similar to Spotify's journey to GCP

Introduction to serverless computing on Google Cloud

wesley chun

Balkan - data eng meetup - data fusion

Balkan Misirli

GCP-pde.pdf

NirajKumar938204

Cloud Native Use Cases / Case Studies - KubeCon 2019 San Diego - RECAP

Krishna-Kumar

Google I/O 2016 Recap - Google Cloud Platform News Update

Simon Su

The journey of Moving from AWS ELK to GCP Data Pipeline

Randy Huang

Jareks talk is about tools and mechanism we developed and used to improve productivity and teamwork in our team (of 6 currently) while developing 70+ operators for Airflow over more than 6 months. We developed an "Airflow Breeze" simplified development environment which cuts down the time to become productive Apache Airflow developer from days to minutes. It is part of Airflow Improvement Proposals: AIP-10 Multi-layered and multi-stage official Airflow image AIP-7 Simplified development workflow

It's a Breeze to develop Airflow (Cloud Native Warsaw)

Jarek Potiuk

This talk is about tools and mechanism we developed and used to improve productivity and teamwork in our team (of 6 currently) while developing 70+ operators for Airflow over more than 6 months. We developed an "Airflow Breeze" simplified development environment which cuts down the time to become productive Apache Airflow developer from days to minutes. It is part of Airflow Improvement Proposals: AIP-10 Multi-layered and multi-stage official Airflow image AIP-7 Simplified development workflow

It's a Breeze to develop Apache Airflow (London Apache Airflow meetup)

Jarek Potiuk

So you want to write a cloud function

Elad Hirsch

6 years ago, I wrote my first "serverless" application: Fewbytes' clients timesheets interface. This was an AppEngine application, and its architecture was similar to other server applications - clients communicating with a central piece of code which talks on their behalf to other components. But today, with the growing power and sophistication of Cloud platforms, our familiar architectural patterns fail us. To fully unleash the power of the Cloud, a new type of architectural pattern is required, one in which there is no central server and many peers cooperate. We need a truly Cloud Native architecture: Total Cloud Immersion

Total cloud immersion

Avishai Ish-Shalom

Mad scalability: Scaling when you are not Google

Abel Muíño

Traveloka's journey to no ops streaming analytics

Rendy Bambang Junior

Summary: In this talk you’ll learn how to implement and deploy a basic serverless Python application. --- Serverless is a concept that has recently raised to popularity, boosted by the drive to financially optimize usage of computing power in cloud environments while reducing maintenance efforts. The following topics will be covered in this talk: - What is a serverless application? - What are the benefits of the serverless execution model? - What is AWS Lambda - How to implement a basic Python serverless application with AWS Lambda? - How to implement a serverless Python based Webservice using Zappa

PyConIE 2017 Writing and deploying serverless python applications

Cesar Cardenas Desales

CloudKit

Arun Nagarajan

This 1-hour presentation is meant to give univeresity hackathoners a deeper yes still high-level overview of Google Cloud and its developer APIs with the purpose of inspiring students to consider these products for their hacks. It follows and dives deeper into the products introduced at the opening ceremony lightning talk. Of particular focus are the serverless and machine learning platforms & APIs... tools that have an immediate impact on projects, alleviating the need to manage VMs, operating systems, etc., as well as dispensing with the need to have expertise with machine learning.

Powerful Google Cloud tools for your hack

wesley chun

Scale with a smile with Google Cloud Platform At DevConTLV (June 2014)

Ido Green

Introduction to DevOps and the Practical Use Cases at Credit OK

Kriangkrai Chaonithi

PyConIT 2018 Writing and deploying serverless python applications

Cesar Cardenas Desales

GDG London Workshop: Build GCP infrastructure with Terraform

Pradeep Bhadani

Cloud Computing

Omar Fathy

Similar to Spotify's journey to GCP (20)

Introduction to serverless computing on Google Cloud

Balkan - data eng meetup - data fusion

GCP-pde.pdf

Cloud Native Use Cases / Case Studies - KubeCon 2019 San Diego - RECAP

Google I/O 2016 Recap - Google Cloud Platform News Update

The journey of Moving from AWS ELK to GCP Data Pipeline

It's a Breeze to develop Airflow (Cloud Native Warsaw)

It's a Breeze to develop Apache Airflow (London Apache Airflow meetup)

So you want to write a cloud function

Total cloud immersion

Mad scalability: Scaling when you are not Google

Traveloka's journey to no ops streaming analytics

PyConIE 2017 Writing and deploying serverless python applications

CloudKit

Powerful Google Cloud tools for your hack

Scale with a smile with Google Cloud Platform At DevConTLV (June 2014)

Introduction to DevOps and the Practical Use Cases at Credit OK

PyConIT 2018 Writing and deploying serverless python applications

GDG London Workshop: Build GCP infrastructure with Terraform

Cloud Computing

Recently uploaded

Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving

Edi Saputra

Partners Life - Insurer Innovation Award 2024

The Digital Insurer

Building Digital Trust in a Digital Economy Veronica Tan, Director - Cyber Security Agency of Singapore Apidays Singapore 2024: Connecting Customers, Business and Technology (April 17 & 18, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...

apidays

Webinar Recording: https://www.panagenda.com/webinars/why-teams-call-analytics-is-critical-to-your-entire-business Nothing is as frustrating and noticeable as being in an important call and being unable to see or hear the other person. Not surprising then, that issues with Teams calls are among the most common problems users call their helpdesk for. Having in depth insight into everything relevant going on at the user’s device, local network, ISP and Microsoft itself during the call is crucial for good Microsoft Teams Call quality support. To ensure a quick and adequate solution and to ensure your users get the most out of their Microsoft 365. But did you know that ‘bad calls’ are also an excellent indicator of other problems arising? Precisely because it is so noticeable!? Like the canary in the mine, bad calls can be early indicators of problems. Problems that might otherwise not have been noticed for a while but can have a big impact on productivity and satisfaction. Join this session by Christoph Adler to learn how true Microsoft Teams call quality analytics helped other organizations troubleshoot bad calls and identify and fix problems that impacted Teams calls or the use of Microsoft365 in general. See what it can do to keep your users happy and productive! In this session we will cover - Why CQD data alone is not enough to troubleshoot call problems - The importance of attributing call problems to the right call participant - What call quality analytics can do to help you quickly find, fix-, and prevent problems - Why having retrospective detailed insights matters - Real life examples of how others have used Microsoft Teams call quality monitoring to problem shoot problems with their ISP, network, device health and more.

Why Teams call analytics are critical to your entire business

panagenda

This presentations targets students or working professionals. You may know Google for search, YouTube, Android, Chrome, and Gmail, but did you know Google has many developer tools, platforms & APIs? This comprehensive yet still high-level overview outlines the most impactful tools for where to run your code, store & analyze your data. It will also inspire you as to what's possible. This talk is 50 minutes in length.

Powerful Google developer tools for immediate impact! (2023-24 C)

wesley chun

The value of a flexible API Management solution for Open Banking Steve Melan, Manager for IT Innovation and Architecture - State's and Saving's Bank of Luxembourg Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - The value of a flexible API Management solution for O...

apidays

Discord is a free app offering voice, video, and text chat functionalities, primarily catering to the gaming community. It serves as a hub for users to create and join servers tailored to their interests. Discord’s ecosystem comprises servers, each functioning as a distinct online community with its own channels dedicated to specific topics or activities. Users can engage in text-based discussions, voice calls, or video chats within these channels. Understanding Discord Servers Discord servers are virtual spaces where users congregate to interact, share content, and build communities. Servers may revolve around gaming, hobbies, interests, or fandoms, providing a platform for like-minded individuals to connect. Communication Features Discord offers a range of communication tools, including text channels for messaging, voice channels for real-time audio conversations, and video channels for face-to-face interactions. These features facilitate seamless communication and collaboration. What Does NSFW Mean? The acronym NSFW stands for “Not Safe For Work,” indicating content that may be inappropriate for professional or public settings. NSFW Content NSFW content encompasses material that is sexually explicit, violent, or otherwise graphic in nature. It often includes nudity, profanity, or depictions of sensitive topics.

Understanding Discord NSFW Servers A Guide for Responsible Users.pdf

UK Journal

Axa Assurance Maroc - Insurer Innovation Award 2024

The Digital Insurer

Scaling API-first – The story of a global engineering organization

Radu Cotescu

Effective data discovery is crucial for maintaining compliance and mitigating risks in today's rapidly evolving privacy landscape. However, traditional manual approaches often struggle to keep pace with the growing volume and complexity of data. Join us for an insightful webinar where industry leaders from TrustArc and Privya will share their expertise on leveraging AI-powered solutions to revolutionize data discovery. You'll learn how to: - Effortlessly maintain a comprehensive, up-to-date data inventory - Harness code scanning insights to gain complete visibility into data flows leveraging the advantages of code scanning over DB scanning - Simplify compliance by leveraging Privya's integration with TrustArc - Implement proven strategies to mitigate third-party risks Our panel of experts will discuss real-world case studies and share practical strategies for overcoming common data discovery challenges. They'll also explore the latest trends and innovations in AI-driven data management, and how these technologies can help organizations stay ahead of the curve in an ever-changing privacy landscape.

TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery

TrustArc

Increase engagement and revenue with Muvi Live Paywall! In this presentation, we will explore the five key benefits of using Muvi Live Paywall to monetize your live streams. You'll learn how Muvi Live Paywall can help you: Monetize your live content easily: Set up pay-per-view access to your live streams and start generating revenue from your content. Increase audience engagement: Provide exclusive, premium content behind the paywall to keep your viewers engaged. Gain valuable viewer insights: Track viewer data and analytics to better understand your audience and tailor your content accordingly. Reduce content piracy: Muvi Live Paywall's security features help protect your content from unauthorized distribution. Streamline your workflow: The all-in-one platform simplifies the process of managing and monetizing your live streams. With Muvi Live Paywall, you can take control of your live stream monetization and create a sustainable business model for your content. Learn more about Muvi Live Paywall and start generating revenue from your live streams today!

Top 5 Benefits OF Using Muvi Live Paywall For Live Streams

Roshan Dwivedi

The Good, the Bad and the Governed - Why is governance a dirty word? David O'Neill, Chief Operating Officer - APIContext Apidays New York 2024: The API Economy in the AI Era (April 30 & May 1, 2024) ------ Check out our conferences at https://www.apidays.global/ Do you want to sponsor or talk at one of our conferences? https://apidays.typeform.com/to/ILJeAaV8 Learn more on APIscene, the global media made by the community for the community: https://www.apiscene.io Explore the API ecosystem with the API Landscape: https://apilandscape.apiscene.io/

Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...

apidays

Automating Google Workspace (GWS) & more with Apps Script

wesley chun

Following the popularity of "Cloud Revolution: Exploring the New Wave of Serverless Spatial Data," we're thrilled to announce this much-anticipated encore webinar. In this sequel, we'll dive deeper into the Cloud-Native realm by uncovering practical applications and FME support for these new formats, including COGs, COPC, FlatGeoBuf, GeoParquet, STAC, and ZARR. Building on the foundation laid by industry leaders Michelle Roby of Radiant Earth and Chris Holmes of Planet in the first webinar, this second part offers an in-depth look at the real-world application and behind-the-scenes dynamics of these cutting-edge formats. We will spotlight specific use-cases and workflows, showcasing their efficiency and relevance in practical scenarios. Discover the vast possibilities each format holds, highlighted through detailed discussions and demonstrations. Our expert speakers will dissect the key aspects and provide critical takeaways for effective use, ensuring attendees leave with a thorough understanding of how to apply these formats in their own projects. Elevate your understanding of how FME supports these cutting-edge technologies, enhancing your ability to manage, share, and analyze spatial data. Whether you're building on knowledge from our initial session or are new to the serverless spatial data landscape, this webinar is your gateway to mastering cloud-native formats in your workflows.

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME

Safe Software

As privacy and data protection regulations evolve rapidly, organizations operating in multiple jurisdictions face mounting challenges to ensure compliance and safeguard customer data. With state-specific privacy laws coming up in multiple states this year, it is essential to understand what their unique data protection regulations will require clearly. How will data privacy evolve in the US in 2024? How to stay compliant? Our panellists will guide you through the intricacies of these states' specific data privacy laws, clarifying complex legal frameworks and compliance requirements. This webinar will review: - The essential aspects of each state's privacy landscape and the latest updates - Common compliance challenges faced by organizations operating in multiple states and best practices to achieve regulatory adherence - Valuable insights into potential changes to existing regulations and prepare your organization for the evolving landscape

TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments

TrustArc

A Domino Admins Adventures (Engage 2024)

Gabriella Davis

Top 10 Most Downloaded Games on Play Store in 2024

SynarionITSolutions

Three things you will take away from the session: • How to run an effective tenant-to-tenant migration • Best practices for before, during, and after migration • Tips for using migration as a springboard to prepare for Copilot in Microsoft 365 Main ideas: Migration Overview: The presentation covers the current reality of cross-tenant migrations, the triggers, phases, best practices, and benefits of a successful tenant migration Considerations: When considering a migration, it is important to consider the migration scope, performance, customization, flexibility, user-friendly interface, automation, monitoring, support, training, scalability, data integrity, data security, cost, and licensing structure Next Wave: The next wave of change includes the launch of Copilot, which requires businesses to be prepared for upcoming changes related to Copilot and the cloud, and to consolidate data and tighten governance ShareGate: ShareGate can help with pre-migration analysis, configurable migration tool, and automated, end-user driven collaborative governance

Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff

sammart93

Boost Fertility New Invention Ups Success Rates.pdf

sudhanshuwaghmare1

MySQL Webinar, presented on the 25th of April, 2024. Summary: MySQL solutions enable the deployment of diverse Database Architectures tailored to specific needs, including High Availability, Disaster Recovery, and Read Scale-Out. With MySQL Shell's AdminAPI, administrators can seamlessly set up, manage, and monitor these solutions, ensuring efficiency and ease of use in their administration. MySQL Router, on the other hand, provides transparent routing from the application traffic to the backend servers in the architectures, requiring minimal configuration. Completely built in-house and supported by Oracle, these solutions have been adopted by enterprises of all sizes for their business-critical applications. In this presentation, we'll delve into various database architecture solutions to help you choose the right one based on your business requirements. Focusing on technical details and the latest features to maximize the potential of these solutions.

Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...

Miguel Araújo

Recently uploaded (20)

Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving

Partners Life - Insurer Innovation Award 2024

Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...

Why Teams call analytics are critical to your entire business

Powerful Google developer tools for immediate impact! (2023-24 C)

Apidays New York 2024 - The value of a flexible API Management solution for O...

Understanding Discord NSFW Servers A Guide for Responsible Users.pdf

Axa Assurance Maroc - Insurer Innovation Award 2024

Scaling API-first – The story of a global engineering organization

TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery

Top 5 Benefits OF Using Muvi Live Paywall For Live Streams

Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...

Automating Google Workspace (GWS) & more with Apps Script

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME

TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments

A Domino Admins Adventures (Engage 2024)

Top 10 Most Downloaded Games on Play Store in 2024

Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff

Boost Fertility New Invention Ups Success Rates.pdf

Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...

Spotify's journey to GCP

1. Spotify’s journey to GCP

2. ~# whoami Alexey Lapitsky / @vzctl alexey@spotify.com Site Reliability Engineer @Spotify PM for the service track of the GCP migration

3. Agenda ● History ● Why GCP? ● The biggest mistake ● Learnings ● Questions!

4. Spotify’s cloud history 2010 Manual VM provisioning 2012 Cloudstack, reached ~700 instances 2013 Amazon EC2 2015 Google Cloud Platform

5. Current status ● 4 data centers, ~10k physical servers ● Micro-service architecture, ~1k different services ● Mostly commodity hardware (Dell) ● Fully automated, self-service server provisioning Moving everything to GCP

6. Focus

7. Why GCP?

8. Why GCP? ● BigData ● Culture ● Timing ● Partnership

9. Our worst mistake

10. Spoticloud CLI REST DB Cloudstack AWS Puppet DNS Server ● CLI Tool + REST Server ● User authentication with ssh agent: github.com/spotify/crtauth SSH Jumphost

11. Looks awesome! ● No need to distribute AWS accounts ● No need to manage ACLs ● Extremely easy to use ~# spcloud create --name web --project alexey

12. … or not really :( ● Testing environment, shamelessly misused for production ● No accountability or a proper ownership tracking ○ 60% of instances were wasting money ● Access only to the basic cloud primitives ○ Which are already available on-premise Currently being deprecated.

13. GCP Migration

14. Cross-project networking (XPN)

15.

16. Things we learned ● VPN is tricky ● GCP network is great ● Keep an eye on quotas ● Live VM migration! ● Think about ACL mgmt in advance ● BigTable, GCS, BigQuery are awesome

17. Questions?

Spotify's journey to GCP

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (12)

Similar to Spotify's journey to GCP

Similar to Spotify's journey to GCP (20)

Recently uploaded

Recently uploaded (20)

Spotify's journey to GCP