(ENT209) Netflix Cloud Migration, DevOps and Distributed Systems | AWS re:Invent 2014

•

15 likes•7,371 views

Netflix's migration to the cloud as our primary streaming control plane was paralleled by our move from traditional IT and centralized operations to a more decentralized DevOps organizational model. In this session, we explore the relationship between technical infrastructure and organization and how to find the right balance of centralized and decentralized operations. We also cover the rationale, goals, strategies, and technologies applied to accomplish this daunting task. We reflect on where we stand today and how we've realized many of our goals.

Technology

Single points of failure: Expensive robust hardware
Blade Webserver
Non-scalable load
Monolithic app
Embedded PL/SQL
business logic
Potential Problems:
Oracle on
High Perf
Disc Array
Multi-proc
DB Server

1999
2007
2008
2009
2010
2011
2012
2013
2014
2015

Oracle on
High Perf
Disc Array
Multi-proc
DB Server
Blade Webserver
Amazon EC2 Webserver
Middle Tier
Apps
Amazon EC2
API server
Amazon SimpleDB

Amazon SimpleDB
cassandra
Amazon DynamoDB

1
10
100
1000
99.999%
99.99%
99.9%
99%
Rate of Change
Availability
Frontier ofavailability/change

1
10
100
1000
99.999%
99.99%
99.9%
99%
Rate of Change
Availability

•Performance
•Scale
•Availability (redundancy, dependencies, configurations, etc.)
•Security
•Privacy
•Retiring tech-debtIf it matters, don’t delegate it!

by the line manager responsible for the –ilitydebts
Features,
Testing
Availability
Scalability
Performance
Tech-debt
…tradeoff…

Please give us your feedback on this session.
Complete session evaluations and earn re:Invent swag.
http://bit.ly/awsevals

Many companies are adopting Apache Kafka to power their data pipelines, including LinkedIn, Netflix, and Airbnb. Kafka’s ability to handle high throughput real-time data makes it a perfect fit for solving the data integration problem, acting as the common buffer for all your data and bridging the gap between streaming and batch systems. However, building a data pipeline around Kafka today can be challenging because it requires combining a wide variety of tools to collect data from disparate data systems. One tool streams updates from your database to Kafka, another imports logs, and yet another exports to HDFS. As a result, building a data pipeline can take significant engineering effort and has high operational overhead because all these different tools require ongoing monitoring and maintenance. Additionally, some of the tools are simply a poor fit for the job: the fragmented nature of the data integration tools ecosystem lead to creative but misguided solutions such as misusing stream processing frameworks for data integration purposes. We describe the design and implementation of Kafka Connect, Kafka’s new tool for scalable, fault-tolerant data import and export. First we’ll discuss some existing tools in the space and why they fall short when applied to data integration at large scale. Next, we will explore Kafka Connect’s design and how it compares to systems with similar goals, discussing key design decisions that trade off between ease of use for connector developers, operational complexity, and reuse of existing connectors. Finally, we’ll discuss how standardizing on Kafka Connect can ultimately lead to simplifying your entire data pipeline, making ETL into your data warehouse and enabling stream processing applications as simple as adding another Kafka connector. eventbrite_kafka_summit_event_logo_v3-035858-edited.png

Have I Been Pwned and Cloudflare

StefnJkullSigurarson1

Event Driven-Architecture from a Scalability perspective

Jonas Bonér

Blueprint for omnichannel integration architecture

Eric D. Schabell

Agile integration is a broadly scoped discussion around how to use all the services and power contained in your organization’s current architecture. While the topic is interesting in its own right, let's take a deeper look at a specific solution within the integration context: providing an omnichannel customer experience. Omnichannel is the integration and orchestration of channels to make the experience of customer engagement across all channels as efficient as engagement with 1 channel. Webinar recording available: http://redhat.com/en/events/webinar/blueprint-omnichannel-integration-architecture Original internal slides: https://docs.google.com/presentation/d/1dLEPLLiMADb5q9AUnmqxdp8L7UsiovzzalJ1nJPNNkw

elasticRTC -- how to have your own WebRTC cloud scaling to be billions in min...

Luis Lopez

Creating WebRTC applications is simple, but making them scalable in a robust and efficient way isn’t. Due to this, many application developers do not dare to use their own infrastructures and instead prefer to leverage third party clouds for creating their scalable WebRTC services. These clouds are private and usually expose some kind of API that developers consume for accessing WebRTC communication capabilities such as group-communications, media recording or media transcoding. This model is very convenient for creating simple vertical applications, but it also has some drawbacks. First, these clouds billing models are based on pay-per-minute schemes that are not always compatible with the business models desired by application developers. Second, the exposed APIs tend to be restrictive and limit the freedom of developers for innovating. Third, these clouds, being fully private, cannot be extended or customized for specific needs developers may have. In this talk, we propose an alternative solution to private WebRTC PaaS models based on leveraging novel cloud orchestration technologies such as Cloud Formation or Heat Templates for enabling developers to create their very own platform on top of public clouds. We analyze the technological ingredients required for enabling such models and present some experiments and production deployments showing how any developer may bring her own IaaS cloud for growing on it a full featured WebRTC platform with elastic scalability and full control on the billing and on the underlying technology evolution.

Three Pillars, Zero Answers: Rethinking Observability

DevOps.com

Observability has never been more important: the complexity of microservices makes it harder and harder to answer basic questions about system behavior. The conventional wisdom claims that Metrics, Logging and Tracing are “the three pillars” of observability… yet software organizations check these three boxes and are still grasping at straws during emergencies. In this session, we’ll illustrate the problem with the three pillars: metrics, logs, and traces are just data – they are the fuel, not the car.

OCR 's Functions

prithvi764

Ranger’s pluggable architecture allows resource access policy administration and enforcement for standard and custom services from a “single pane of glass”. Apache Ranger has a rich Authorization Model, which provides the mechanism to author Policy in a Ranger Admin Server and serves as policy decision and audit point in authorizing user’s resource access within various components of Hadoop ecosystem. This session will provide a deep dive into Ranger framework and a cook-book for extending Ranger to do authorization / auditing on resource access to external applications, including technical details of Rest APIs, Ranger policy engine and enriching authorization requests, with a demo of a sample application.We will then demonstrate a real-world example of how Ranger has simplified security enforcement for Hadoop-native MPP SQL engine like Apache HAWQ (incubating),which previously used its built-in Postgres-like authorization mechanisms. The integration design includes a Ranger Plugin Service that allows transparent authorization API calls between C-based Apache HAWQ and Java-based Apache Ranger.

Microservices

Meysam Javadi

Apache Kafka as Event Streaming Platform for Microservice Architectures

Kai Wähner

This session introduces Apache Kafka, an event-driven open source streaming platform. Apache Kafka goes far beyond scalable, high volume messaging. In addition, you can leverage Kafka Connect for integration and the Kafka Streams API for building lightweight stream processing microservices in autonomous teams. The Confluent Platform adds further components such as a Schema Registry, REST Proxy, KSQL, Clients for different programming languages and Connectors for different technologies. The session discusses how tech giants like LinkedIn, Ebay or Airbnb leverage Apache Kafka as event streaming platform to solve various different business problems and how to create a scalable, flexible microservice architecture. A live demo shows how you can easily process and analyze streams of events using Apache Kafka and KSQL.

MicroService Architecture

Fred George

SOA, service-oriented architectures, burst on the scene in the new millennium as the latest technology to support application growth. In concert with the Web, SOA ushered in new paradigms for structuring enterprise applications. At the Forward Internet Group in London, we are implementing SOA in unusual ways. Rather than a few, businessrelated services being implemented per the original vision, we have developed systems made of myriads of very small, usually shortlived services. In this workshop, we will start by exploring the evolution of SOA implementations by the speaker. In particular, lessons learned from each implementation will be discussed, and reapplication of these lessons on the next implementation. Challenges (and even failures) will be explicitly identified. We will arrive at a model of the current systems: An environment of very small services that are loosely coupled into a complex system. We explore the demise of acceptance tests in this complex environment, and the clever replacement of business metrics in their stead. Finally, we will conclude with the surprising programmer development process impacts of this architecture. Indeed, bedrock principles of Agile have been rendered unnecessary, something that equally surprised us. (Presented at Agile India 2013)

Pulsar in the Lakehouse: Apache Pulsar™ with Apache Spark™ and Delta Lake - P...

StreamNative

In this session, we provide an overview of the “Lakehouse” architecture and how Apache Pulsar™ can be used to support this architecture through integrations with the Apache Spark™ and Delta Lake to build your reliable data lake. We will also discuss the current state of Pulsar + Spark & Delta Lake connectors and discuss real world use cases and present the roadmap on what you can expect in the future of integrations between Spark, Delta Lake, and Pulsar communities.

Segmenting your Network for Security - The Good, the Bad and the Ugly

AlgoSec

NETWORK PLANNING AND DESIGN, University of Colombo School of Computing

Kafka Summit 2018: Monitoring and Orchestration of Your Microservices Landsca...

Bernd Ruecker

SAM Software Asset Management

BearingPoint

Report on mini project(Student database handling using RMI)

shraddha mane

Event-Driven Transformation in Banking and FSI

Solace

Complex event flows in distributed systems

Bernd Ruecker

Secure Kafka at scale in true multi-tenant environment ( Vishnu Balusu & Asho...

confluent

Application teams in JPMC have started shifting towards building event driven architectures and real time steaming pipelines and Kafka has been at core in this journey. As application teams have started adopting Kafka rapidly, need for a centrally managed Kafka as a service has emerged. We have started delivering Kafka as a service in early 2018 and running in production for more than an year now operating 80+ clusters (and growing) in all environments together. One of the key requirements is to provide truly segregated, secured multi-tenant environment with RBAC model while satisfying financial regulations and controls at the same time. Operating clusters at large scale requires scalable self-service capabilities and cluster management orchestration. In this talk we will present - Our experiences in delivering and operating secured, multi-tenant and resilient Kafka clusters at scale. - Internals of our service framework/control plane which enables self-service capabilities for application teams, cluster build/patch orchestration and capacity management capabilities for TSE/admin teams. - Our approach in enabling automated Cross Datacenter failover for application teams using service framework and confluent replicator.

Netflix: From Zero to Production-Ready in Minutes (QCon 2017)

Tim Bozarth

Slides from Tim Bozarth's (@timbozarth) QCon 2017 presentation (https://qconnewyork.com/ny2017/presentation/zero-production-ready-minutes) Abstract: The fabric of Netflix's approach to building new highly-available services is evolving. The Runtime Platform Team is focused on improving developer productivity while simultaneously making it simpler to build and maintain the high-availability services that Netflix expects. Starting with application generation, and leveraging a new approach to communication between services (RPC), we're simplifying what's needed to build a fast, reliable, and optimized service capable of delivering a fantastic customer experience. We'll be sharing how Netflix is enabling engineers to go from "zero" to "production ready" in minutes - incorporating best-practices learned through years in the cloud. We will also share the story of transitioning from our home-grown RPC machinery to open-source standards, how we recognized when it was the right time to walk away from our own creations, and how our new approach is improving team velocity across Netflix engineering.

Microservices Tutorial for Beginners | Microservices Architecture | Microserv...

Edureka!

( Microservices Architecture Training: https://www.edureka.co/microservices-architecture-training ) This Edureka's tutorial will help you understand what are microservices and their practical implementation using Spring Boot. In this video, you will learn the following: 1. Monolithic Architecture 2. Challenges Of Monolithic Architecture 3. What are Microservices 4. Microservice Architecture Components 5. Hands-On Using Spring Boot

Mastering Chaos - A Netflix Guide to Microservices

C4Media

Video and slides synchronized, mp3 and slide download available at URL http://bit.ly/2hkPH7v. Josh Evans talks about the chaotic and vibrant world of microservices at Netflix. He starts with the basics, - the anatomy of a microservice, the challenges around distributed systems, and the benefits. Then he builds on that foundation exploring the cultural, architectural, and operational methods that lead to microservice mastery. Filmed at qconsf.com. Josh Evans is Director of Operations Engineering at Netflix, with experience in e-commerce, tools, testing, and operations. For the past three years he has led an organization that creates, integrates, and evangelizes proven technical solutions and practices like continuous delivery, real-time operational insight, and chaos engineering to achieve operational excellence at scale.

Experimental Farm - part 1

zzptichka

MongoDB Days UK: Building an Enterprise Data Fabric at Royal Bank of Scotland...

MongoDB

Presented by Michael Fulke, Development Team Lead, Royal Bank of Scotland Experience level: Beginner When addressing common investment banking use-cases, incumbent application architectures have proven themselves to be complex, difficult to maintain and expensive. Driven by the apparently competing pressures of cost and agility, RBS used MongoDB to build a common enterprise data fabric which is underpinning several core trading platforms. In this session, you will learn how RBS has successfully integrated MongoDB into a wider Java-based architecture, built with a strong open source bias.

Network Virtualization

Kingston Smiler

Pervasive Web Application ArchitectureUC San Diego

Cloudscape progress v3Sandra Wills

Smau Milano 2016 - Michele Finelli

SMAU

What's hot

Extending Apache Ranger Authorization Beyond Hadoop: Review of Apache Ranger ...

DataWorks Summit

Microservices

Meysam Javadi

Apache Kafka as Event Streaming Platform for Microservice Architectures

Kai Wähner

MicroService Architecture

Fred George

Pulsar in the Lakehouse: Apache Pulsar™ with Apache Spark™ and Delta Lake - P...

StreamNative

Segmenting your Network for Security - The Good, the Bad and the Ugly

AlgoSec

NETWORK PLANNING AND DESIGN, University of Colombo School of Computing

Kafka Summit 2018: Monitoring and Orchestration of Your Microservices Landsca...

Bernd Ruecker

SAM Software Asset Management

BearingPoint

Report on mini project(Student database handling using RMI)

shraddha mane

Event-Driven Transformation in Banking and FSI

Solace

Complex event flows in distributed systems

Bernd Ruecker

Secure Kafka at scale in true multi-tenant environment ( Vishnu Balusu & Asho...

confluent

Netflix: From Zero to Production-Ready in Minutes (QCon 2017)

Tim Bozarth

Microservices Tutorial for Beginners | Microservices Architecture | Microserv...

Edureka!

Mastering Chaos - A Netflix Guide to Microservices

C4Media

Experimental Farm - part 1

zzptichka

MongoDB Days UK: Building an Enterprise Data Fabric at Royal Bank of Scotland...

MongoDB

Network Virtualization

Kingston Smiler

Pervasive Web Application ArchitectureUC San Diego

What's hot (20)

Extending Apache Ranger Authorization Beyond Hadoop: Review of Apache Ranger ...

Microservices

Apache Kafka as Event Streaming Platform for Microservice Architectures

MicroService Architecture

Pulsar in the Lakehouse: Apache Pulsar™ with Apache Spark™ and Delta Lake - P...

Segmenting your Network for Security - The Good, the Bad and the Ugly

NETWORK PLANNING AND DESIGN,

Kafka Summit 2018: Monitoring and Orchestration of Your Microservices Landsca...

SAM Software Asset Management

Report on mini project(Student database handling using RMI)

Event-Driven Transformation in Banking and FSI

Complex event flows in distributed systems

Secure Kafka at scale in true multi-tenant environment ( Vishnu Balusu & Asho...

Netflix: From Zero to Production-Ready in Minutes (QCon 2017)

Microservices Tutorial for Beginners | Microservices Architecture | Microserv...

Mastering Chaos - A Netflix Guide to Microservices

Experimental Farm - part 1

MongoDB Days UK: Building an Enterprise Data Fabric at Royal Bank of Scotland...

Network Virtualization

Pervasive Web Application Architecture

Viewers also liked

Cloudscape progress v3Sandra Wills

Smau Milano 2016 - Michele Finelli

SMAU

White Paper - Cloudscape 2015

Microsoft India

NetflixOSS for Triangle Devops Oct 2013

aspyker

Spring Cloud Netflix OSS

Steve Hall

Slides from the April 27th, 2016 Milwaukee Java Meetup held at DigitalMeasures in Milwaukee WI. Example code and configuration at: https://github.com/shall11672/mkej-config https://github.com/shall11672/mkej-configServer https://github.com/shall11672/mkej-eurekaService https://github.com/shall11672/mkej-greetingService https://github.com/shall11672/mkej-menuService https://github.com/shall11672/mkej-apiGateway https://github.com/shall11672/mkej-monitoringApplication

Optimizing the Ops in DevOps

Gordon Haff

As DevOps practices have been put into wide use, it's become evident that developers and operations aren't merging to become one discipline. Nor is operations simply going away. Rather, DevOps is leading software development and operations - together with other practices such as security - to collaborate and coexist with less overhead and conflict than in the past. In his session at @DevOpsSummit at 19th Cloud Expo, Gordon Haff, Red Hat Technology Evangelist, will discuss what modern operational practices look like in a world in which applications are more loosely coupled, are developed using DevOps approaches, and are deployed on software-defined, and often containerized, infrastructures - and where operations itself is increasingly another "as a service" capability from the perspective of developers. How does the operations tool chest change? How does the required skill set differ? How are the interactions between operations and other IT and business organizations different from in the past? How can operations provide the confidence to the entire organization that this new pipeline is still delivering non-functional requirements such as regulatory compliance and a secure and certified operating environment? How does operations safely consume vendor and upstream dependencies while meeting developer desires for the latest and greatest? Operations is more important than ever for a business to derive value from its IT organization. But the roles and the goals of operations are significantly different than they were historically.

(BDT403) Netflix's Next Generation Big Data Platform | AWS re:Invent 2014

(ENT209) Netflix Cloud Migration, DevOps and Distributed Systems | AWS re:Invent 2014

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to (ENT209) Netflix Cloud Migration, DevOps and Distributed Systems | AWS re:Invent 2014

Similar to (ENT209) Netflix Cloud Migration, DevOps and Distributed Systems | AWS re:Invent 2014 (20)

More from Amazon Web Services

More from Amazon Web Services (20)

Recently uploaded

Recently uploaded (20)

(ENT209) Netflix Cloud Migration, DevOps and Distributed Systems | AWS re:Invent 2014