SlideShare a Scribd company logo
Introduction:
Building Using The NetflixOSS
Architecture
May 2013
Adrian Cockcroft
@adrianco #netflixcloud @NetflixOSS
http://www.linkedin.com/in/adriancockcroft
Presentation vs. Tutorial
• Presentation
– Short duration, focused subject
– One presenter to many anonymous audience
– A few questions at the end
• Tutorial
– Time to explore in and around the subject
– Tutor gets to know the audience
– Discussion, rat-holes, “bring out your dead”
Introduction – Who are you?
Netflix Open Source Cloud Prize
Cloud Native – More details
NetflixOSS – Cloud Native On-Ramp
Adrian Cockcroft
• Director, Architecture for Cloud Systems, Netflix Inc.
– Previously Director for Personalization Platform
• Distinguished Availability Engineer, eBay Inc. 2004-7
– Founding member of eBay Research Labs
• Distinguished Engineer, Sun Microsystems Inc. 1988-2004
– 2003-4 Chief Architect High Performance Technical Computing
– 2001 Author: Capacity Planning for Web Services
– 1999 Author: Resource Management
– 1995 & 1998 Author: Sun Performance and Tuning
– 1996 Japanese Edition of Sun Performance and Tuning
• SPARC & Solarisパフォーマンスチューニング (サンソフトプレスシリー
ズ)
• More
– Twitter @adrianco – Blog http://perfcap.blogspot.com
– Presentations at http://www.slideshare.net/adrianco
Attendee Introductions
• Who are you, where do you work
• Why are you here today, what do you need
• “Bring out your dead”
– Do you have a specific problem or question?
– One sentence elevator pitch
• What instrument do you play?
Boosting the @NetflixOSS Ecosystem
See netflix.github.com
In 2012 Netflix Engineering won this..
We’d like to give out prizes too
But what for?
Contributions to NetflixOSS!
Shared under Apache license
Located on github
How long do you have?
Entries open March 13th
Entries close September 15th
Six months…
Who can win?
Almost anyone, anywhere…
Except current or former Netflix or
AWS employees
Who decides who wins?
Nominating Committee
Panel of Judges
Judges
Aino Corry
Program Chair for Qcon/GOTO
Martin Fowler
Chief Scientist ThoughtworksSimon Wardley
Strategist
Yury Izrailevsky
VP Cloud Netflix
Werner Vogels
CTO Amazon Joe Weinman
SVP Telx, Author “Cloudonomics”
What are Judges Looking For?
Eligible, Apache 2.0 licensed
NetflixOSS project pull requests
Original and useful contribution to NetflixOSS
Good code quality and structure
Documentation on how to build and run it
Code that successfully builds and passes a test suite
Evidence that code is in use by other projects, or is running in production
A large number of watchers, stars and forks on github
What do you win?
One winner in each of the 10 categories
Ticket and expenses to attend AWS
Re:Invent 2013 in Las Vegas
A Trophy
How do you enter?
Get a (free) github account
Fork github.com/netflix/cloud-prize
Send us your email address
Describe and build your entry
Twitter #cloudprize
Entrants
Netflix
Engineering
Six Judges Winners
Nominations
Conforms to
Rules
Working
Code
Community
Traction
Categories
Registration
Opened
March 13
Github
Apache
Licensed
Contributions
Github
Close Entries
September 15
Github
Award
Ceremony
Dinner
November
AWS
Re:Invent
Ten Prize
Categories
$10K cash
$5K AWS
AWS
Re:Invent
Tickets
Trophy
Cloud Native
Recap the keynote in much more
detail and discussion
A new engineering challenge
Construct a highly agile and highly
available service from ephemeral and
often broken components
Inspiration
Netflix Streaming
A Cloud Native Application based on
an open source platform
Netflix Member Web Site Home Page
Personalization Driven – How Does It Work?
How Netflix Streaming Works
Customer Device
(PC, PS3, TV…)
Web Site or
Discovery API
User Data
Personalization
Streaming API
DRM
QoS Logging
OpenConnect
CDN Boxes
CDN
Management and
Steering
Content Encoding
Consumer
Electronics
AWS Cloud
Services
CDN Edge
Locations
Real Web Server Dependencies Flow
(Netflix Home page business transaction as seen by AppDynamics)
Start Here
memcached
Cassandra
Web service
S3 bucket
Personalization movie group choosers
(for US, Canada and Latam)
Each icon is
three to a few
hundred
instances
across three
AWS zones
New Cloud Native Patterns
Micro-services and Chaos engines
Highly available systems composed
from ephemeral components
Open Source is the default
Some Strategic Questions
What changed…
The AWS Question
Why does Netflix use AWS when
Amazon Prime is a competitor?
Netflix vs. Amazon Prime
• Do retailers competing with Amazon use AWS?
– Yes, lots of them, Netflix is no different
• Does Prime have a platform advantage?
– No, because Netflix gets to run on AWS
• Does Netflix take Amazon Prime seriously?
– Yes, but so far Prime isn’t impacting our business
Amazon Video 1.31%
18x Prime
25x Prime
Nov
2012
Streaming
Bandwidth
March
2013
Mean
Bandwidth
+39% 6mo
The Google Cloud Question
Why doesn’t Netflix use Google
Cloud as well as AWS?
Google Cloud – Wait and See
Pro’s
• Cloud Native
• Huge scale for internal apps
• Exposing internal services
• Nice clean API model
• Starting a price war
• Fast for what it does
• Rapid start & minute billing
Con’s
• In beta until last week
• No big customers yet
• Missing many key features
• Different arch model
• Missing billing options
• No SSD or huge instances
• Zone maintenance windows
But: Anyone interested is welcome to port NetflixOSS components to Google Cloud
Cloud Wars: Price and Performance
AWS vs.
GCS War
Private
Cloud
What Changed:
Everyone using
AWS or GCS gets
the price cuts and
performance
improvements, as
they happen. No
need to switch
vendor.
No Change:
Locked in for
three years.
The DIY Question
Why doesn’t Netflix build and run its
own cloud?
Fitting Into Public Scale
Public
Grey
Area
Private
1,000 Instances 100,000 Instances
Netflix FacebookStartups
How big is Public?
AWS upper bound estimate based on the number of public IP Addresses
Every provisioned instance gets a public IP by default
AWS Maximum Possible Instance Count 3.7 Million
Growth >10x in Three Years, >2x Per Annum
The Alternative Supplier
Question
What if there is no clear leader for a
feature, or AWS doesn’t have what
we need?
Things We Don’t Use AWS For
SaaS Applications – Pagerduty, Appdynamics
Content Delivery Service
DNS Service
CDN Scale
AWS CloudFront
Akamai
Limelight
Level 3
Netflix
Openconnect
YouTube
Gigabits Terabits
NetflixFacebookStartups
Content Delivery Service
Open Source Hardware Design + FreeBSD, bird, nginx
see openconnect.netflix.com
DNS Service
AWS Route53 is missing too many features
Multiple vendor strategy Dyn, Ultra, Route53
Abstracted (broken) DNS APIs with Denominator
Availability Questions
Is it running yet?
How many places is it running in?
How far apart are those places?
Netflix Outages
• Running very fast with scissors
– Mostly self inflicted – bugs, mistakes from pace of change
– Some caused by AWS bugs and mistakes
• Incident Life-cycle Management by Platform Team
– No runbooks, no operational changes by the SREs
– Tools to identify what broke and call the right developer
• Next step is multi-region
– Investigating and building in stages during 2013
– Could have prevented some of our 2012 outages
Managing Multi-Region Availability
Cassandra Replicas
Zone A
Cassandra Replicas
Zone B
Cassandra Replicas
Zone C
Regional Load Balancers
Cassandra Replicas
Zone A
Cassandra Replicas
Zone B
Cassandra Replicas
Zone C
Regional Load Balancers
UltraDNS
DynECT
DNS
AWS
Route53
Denominator – manage traffic via multiple DNS providers
Denominator
Cloud Native Big Data
Size the cluster to the data
Size the cluster to the questions
Never wait for space or answers
Netflix Dataoven
Data Warehouse
Over 2 Petabytes
Ursula
Aegisthus
Data Pipelines
From cloud
Services
~100 Billion
Events/day
From C*
Terabytes of
Dimension
data
Hadoop Clusters – AWS EMR
1300 nodes 800 nodes Multiple 150 nodes Nightly
RDS
Metadata
Gateways
Tools
Cloud Native Patterns
Master copies of data are cloud resident
Dynamically provisioned micro-services
Services are distributed and ephemeral
Cloud Native Architecture
Distributed Quorum
NoSQL Datastores
Autoscaled Micro
Services
Autoscaled Micro
Services
Clients Things
JVM JVM
JVM JVM
Cassandra Cassandra Cassandra
Memcached
JVM
Zone A Zone B Zone C
Non-Native Cloud Architecture
Datacenter
Dinosaurs
Cloudy
Buffer
Agile Mobile
Mammals
iOS/Android
App Servers
MySQL Legacy Apps
How to get to Cloud Native?
Freedom and Responsibility for Developers
Decentralize and Automate Ops Activities
Integrate DevOps into the Business Organization
Four Transitions
• Management: Integrated Roles in a Single Organization
– Business, Development, Operations -> BusDevOps
• Developers: Denormalized Data – NoSQL
– Decentralized, scalable, available, polyglot
• Responsibility from Ops to Dev: Continuous Delivery
– Decentralized small daily production updates
• Responsibility from Ops to Dev: Agile Infrastructure - Cloud
– Hardware in minutes, provisioned directly by developers
Netflix BusDevOps Organization
Chief Product
Officer
VP Product
Management
Directors
Product
VP UI
Engineering
Directors
Development
Developers +
DevOps
UI Data
Sources
AWS
VP Discovery
Engineering
Directors
Development
Developers +
DevOps
Discovery
Data Sources
AWS
VP Platform
Directors
Platform
Developers +
DevOps
Platform
Data Sources
AWS
Denormalized, independently
updated and scaled data
Cloud, independently updated
and scaled infrastructure
Code, independently updated
continuous delivery
Decentralized Deployment
Asgard
http://techblog.netflix.com/2012/06/asgard-web-based-cloud-management-and.html
Ephemeral Instances
• Largest services are autoscaled
• Average lifetime of an instance is 36 hours
P
u
s
h
Autoscale Up
Autoscale Down
A Cloud Native Open Source Platform
See netflix.github.com
Three Questions
Why is Netflix doing this?
How does it all fit together?
What is coming next?
Beware of Geeks Bearing Gifts: Strategies for an
Increasingly Open Economy
Simon Wardley - Researcher at the Leading Edge Forum
How did Netflix get ahead?
Netflix BusDevOps Org
• Doing it since 2009
• SaaS Applications
• PaaS for agility
• Public IaaS for AWS features
• Big data in the cloud
• Integrating many APIs
• FOSS from github
• Renting hardware for 1hr
• Coding in Java/Groovy/Scala
Traditional IT Operations
• Taking their time
• Pilot private cloud projects
• Beta quality installations
• Small scale
• Integrating several vendors
• Paying big $ for software
• Paying big $ for consulting
• Buying hardware for 3yrs
• Hacking at scripts
Netflix Platform Evolution
Bleeding Edge
Innovation
Common
Pattern
Shared
Pattern
2009-2010 2011-2012 2013-2014
Netflix ended up several years ahead of the
industry, but it’s becoming commoditized now
Making it easy to follow
Exploring the wild west each time vs. laying down a shared route
Establish our
solutions as Best
Practices / Standards
Hire, Retain and
Engage Top
Engineers
Build up Netflix
Technology Brand
Benefit from a
shared ecosystem
Goals
How does it all fit together?
Example Application – RSS Reader
Github
NetflixOSS
Source
AWS
Base AMI
Maven
Central
Cloudbees
Jenkins
Aminator
Bakery
Dynaslave
AWS Build
Slaves
Asgard
(+ Frigga)
Console
AWS
Baked AMIs
Odin
Orchestration
API
AWS
Account
NetflixOSS Continuous Build and Deployment
AWS Account
Asgard Console
Archaius
Config Service
Cross region
Priam C*
Pytheas
Dashboards
Atlas
Monitoring
Genie, Lipstick
Hadoop Services
AWS Usage
Cost Monitoring
Multiple AWS Regions
Eureka Registry
Exhibitor ZK
Edda History
Simian Army
3 AWS Zones
Application
Clusters
Autoscale Groups
Instances
Priam
Cassandra
Persistent Storage
Evcache
Memcached
Ephemeral Storage
NetflixOSS Services Scope
•Baked AMI – Tomcat, Apache, your code
•Governator – Guice based dependency injection
•Archaius – dynamic configuration properties client
•Eureka - service registration client
Initialization
•Karyon - Base Server for inbound requests
•RxJava – Reactive pattern
•Hystrix/Turbine – dependencies and real-time status
•Ribbon - REST Client for outbound calls
Service
Requests
•Astyanax – Cassandra client and pattern library
•Evcache – Zone aware Memcached client
•Curator – Zookeeper patterns
•Denominator – DNS routing abstraction
Data Access
•Blitz4j – non-blocking logging
•Servo – metrics export for autoscaling
•Atlas – high volume instrumentation
Logging
NetflixOSS Instance Libraries
•CassJmeter – Load testing for Cassandra
•Circus Monkey – Test account reservation rebalancingTest Tools
•Janitor Monkey – Cleans up unused resources
•Efficiency Monkey
•Doctor Monkey
•Howler Monkey – Complains about AWS limits
Maintenance
•Chaos Monkey – Kills Instances
•Chaos Gorilla – Kills Availability Zones
•Chaos Kong – Kills Regions
•Latency Monkey – Latency and error injection
Availability
•Security Monkey – security group and S3 bucket permissions
•Conformity Monkey – architectural pattern warningsSecurity
NetflixOSS Testing and Automation
More Use Cases
More
Features
Better portability
Higher availability
Easier to deploy
Contributions from end users
Contributions from vendors
What’s Coming Next?
Vendor Driven Portability
Interest in using NetflixOSS for Enterprise Private Clouds
“It’s done when it runs Asgard”
Functionally complete
Demonstrated March
Release 3.3 in 2Q13
Some vendor interest
Needs AWS compatible Autoscaler
Some vendor interest
Many missing features
“Confused” AWS API strategy
AWS 2009
Baseline features needed to support NetflixOSS
Eucalyptus 3.3
Functionality and scale now, portability coming
Moving from parts to a platform in 2013
Netflix is fostering a cloud native ecosystem
Rapid Evolution - Low MTBIAMSH
(Mean Time Between Idea And Making Stuff Happen)
Takeaway
NetflixOSS makes it easier for everyone to become Cloud Native
@adrianco #netflixcloud @NetflixOSS

More Related Content

What's hot

Netflix Global Cloud Architecture
Netflix Global Cloud ArchitectureNetflix Global Cloud Architecture
Netflix Global Cloud Architecture
Adrian Cockcroft
 
Gluecon keynote
Gluecon keynoteGluecon keynote
Gluecon keynote
Adrian Cockcroft
 
Netflix Cloud Platform Building Blocks
Netflix Cloud Platform Building BlocksNetflix Cloud Platform Building Blocks
Netflix Cloud Platform Building Blocks
Sudhir Tonse
 
Netflix in the Cloud
Netflix in the CloudNetflix in the Cloud
Netflix in the Cloud
Adrian Cockcroft
 
Netflix Velocity Conference 2011
Netflix Velocity Conference 2011Netflix Velocity Conference 2011
Netflix Velocity Conference 2011
Adrian Cockcroft
 
Performance architecture for cloud connect
Performance architecture for cloud connectPerformance architecture for cloud connect
Performance architecture for cloud connect
Adrian Cockcroft
 
Netflix in the cloud 2011
Netflix in the cloud 2011Netflix in the cloud 2011
Netflix in the cloud 2011
Adrian Cockcroft
 
Cloud Architecture Tutorial - Platform Component Architecture (2of3)
Cloud Architecture Tutorial - Platform Component Architecture (2of3)Cloud Architecture Tutorial - Platform Component Architecture (2of3)
Cloud Architecture Tutorial - Platform Component Architecture (2of3)
Adrian Cockcroft
 
Global Netflix - HPTS Workshop - Scaling Cassandra benchmark to over 1M write...
Global Netflix - HPTS Workshop - Scaling Cassandra benchmark to over 1M write...Global Netflix - HPTS Workshop - Scaling Cassandra benchmark to over 1M write...
Global Netflix - HPTS Workshop - Scaling Cassandra benchmark to over 1M write...
Adrian Cockcroft
 
Architectures for High Availability - QConSF
Architectures for High Availability - QConSFArchitectures for High Availability - QConSF
Architectures for High Availability - QConSF
Adrian Cockcroft
 
Cloud Architecture Tutorial - Running in the Cloud (3of3)
Cloud Architecture Tutorial - Running in the Cloud (3of3)Cloud Architecture Tutorial - Running in the Cloud (3of3)
Cloud Architecture Tutorial - Running in the Cloud (3of3)
Adrian Cockcroft
 
MED202 Netflix’s Transcoding Transformation - AWS re: Invent 2012
MED202 Netflix’s Transcoding Transformation - AWS re: Invent 2012MED202 Netflix’s Transcoding Transformation - AWS re: Invent 2012
MED202 Netflix’s Transcoding Transformation - AWS re: Invent 2012
Amazon Web Services
 
Netflix and Open Source
Netflix and Open SourceNetflix and Open Source
Netflix and Open Source
Adrian Cockcroft
 
ARC203 Highly Available Architecture at Netflix - AWS re: Invent 2012
ARC203 Highly Available Architecture at Netflix - AWS re: Invent 2012ARC203 Highly Available Architecture at Netflix - AWS re: Invent 2012
ARC203 Highly Available Architecture at Netflix - AWS re: Invent 2012
Amazon Web Services
 
Intuit CTOF 2011 - Netflix for Mobile in the Cloud
Intuit CTOF 2011 - Netflix for Mobile in the CloudIntuit CTOF 2011 - Netflix for Mobile in the Cloud
Intuit CTOF 2011 - Netflix for Mobile in the Cloud
Sid Anand
 
ARC301 Intro to Chaos Monkey & the Simian Army - AWS re: Invent 2012
ARC301 Intro to Chaos Monkey & the Simian Army - AWS re: Invent 2012ARC301 Intro to Chaos Monkey & the Simian Army - AWS re: Invent 2012
ARC301 Intro to Chaos Monkey & the Simian Army - AWS re: Invent 2012
Amazon Web Services
 
Asgard, the Grails App that Deploys Netflix to the Cloud
Asgard, the Grails App that Deploys Netflix to the CloudAsgard, the Grails App that Deploys Netflix to the Cloud
Asgard, the Grails App that Deploys Netflix to the Cloud
Joe Sondow
 
AWS Re:Invent - Optimizing Costs with AWS
AWS Re:Invent -  Optimizing Costs with AWSAWS Re:Invent -  Optimizing Costs with AWS
AWS Re:Invent - Optimizing Costs with AWS
Coburn Watson
 
High Availability in the Cloud - Architectural Best Practices
High Availability in the Cloud - Architectural Best PracticesHigh Availability in the Cloud - Architectural Best Practices
High Availability in the Cloud - Architectural Best Practices
RightScale
 
ENT101 Embracing the Cloud - AWS re: Invent 2012
ENT101 Embracing the Cloud - AWS re: Invent 2012ENT101 Embracing the Cloud - AWS re: Invent 2012
ENT101 Embracing the Cloud - AWS re: Invent 2012
Amazon Web Services
 

What's hot (20)

Netflix Global Cloud Architecture
Netflix Global Cloud ArchitectureNetflix Global Cloud Architecture
Netflix Global Cloud Architecture
 
Gluecon keynote
Gluecon keynoteGluecon keynote
Gluecon keynote
 
Netflix Cloud Platform Building Blocks
Netflix Cloud Platform Building BlocksNetflix Cloud Platform Building Blocks
Netflix Cloud Platform Building Blocks
 
Netflix in the Cloud
Netflix in the CloudNetflix in the Cloud
Netflix in the Cloud
 
Netflix Velocity Conference 2011
Netflix Velocity Conference 2011Netflix Velocity Conference 2011
Netflix Velocity Conference 2011
 
Performance architecture for cloud connect
Performance architecture for cloud connectPerformance architecture for cloud connect
Performance architecture for cloud connect
 
Netflix in the cloud 2011
Netflix in the cloud 2011Netflix in the cloud 2011
Netflix in the cloud 2011
 
Cloud Architecture Tutorial - Platform Component Architecture (2of3)
Cloud Architecture Tutorial - Platform Component Architecture (2of3)Cloud Architecture Tutorial - Platform Component Architecture (2of3)
Cloud Architecture Tutorial - Platform Component Architecture (2of3)
 
Global Netflix - HPTS Workshop - Scaling Cassandra benchmark to over 1M write...
Global Netflix - HPTS Workshop - Scaling Cassandra benchmark to over 1M write...Global Netflix - HPTS Workshop - Scaling Cassandra benchmark to over 1M write...
Global Netflix - HPTS Workshop - Scaling Cassandra benchmark to over 1M write...
 
Architectures for High Availability - QConSF
Architectures for High Availability - QConSFArchitectures for High Availability - QConSF
Architectures for High Availability - QConSF
 
Cloud Architecture Tutorial - Running in the Cloud (3of3)
Cloud Architecture Tutorial - Running in the Cloud (3of3)Cloud Architecture Tutorial - Running in the Cloud (3of3)
Cloud Architecture Tutorial - Running in the Cloud (3of3)
 
MED202 Netflix’s Transcoding Transformation - AWS re: Invent 2012
MED202 Netflix’s Transcoding Transformation - AWS re: Invent 2012MED202 Netflix’s Transcoding Transformation - AWS re: Invent 2012
MED202 Netflix’s Transcoding Transformation - AWS re: Invent 2012
 
Netflix and Open Source
Netflix and Open SourceNetflix and Open Source
Netflix and Open Source
 
ARC203 Highly Available Architecture at Netflix - AWS re: Invent 2012
ARC203 Highly Available Architecture at Netflix - AWS re: Invent 2012ARC203 Highly Available Architecture at Netflix - AWS re: Invent 2012
ARC203 Highly Available Architecture at Netflix - AWS re: Invent 2012
 
Intuit CTOF 2011 - Netflix for Mobile in the Cloud
Intuit CTOF 2011 - Netflix for Mobile in the CloudIntuit CTOF 2011 - Netflix for Mobile in the Cloud
Intuit CTOF 2011 - Netflix for Mobile in the Cloud
 
ARC301 Intro to Chaos Monkey & the Simian Army - AWS re: Invent 2012
ARC301 Intro to Chaos Monkey & the Simian Army - AWS re: Invent 2012ARC301 Intro to Chaos Monkey & the Simian Army - AWS re: Invent 2012
ARC301 Intro to Chaos Monkey & the Simian Army - AWS re: Invent 2012
 
Asgard, the Grails App that Deploys Netflix to the Cloud
Asgard, the Grails App that Deploys Netflix to the CloudAsgard, the Grails App that Deploys Netflix to the Cloud
Asgard, the Grails App that Deploys Netflix to the Cloud
 
AWS Re:Invent - Optimizing Costs with AWS
AWS Re:Invent -  Optimizing Costs with AWSAWS Re:Invent -  Optimizing Costs with AWS
AWS Re:Invent - Optimizing Costs with AWS
 
High Availability in the Cloud - Architectural Best Practices
High Availability in the Cloud - Architectural Best PracticesHigh Availability in the Cloud - Architectural Best Practices
High Availability in the Cloud - Architectural Best Practices
 
ENT101 Embracing the Cloud - AWS re: Invent 2012
ENT101 Embracing the Cloud - AWS re: Invent 2012ENT101 Embracing the Cloud - AWS re: Invent 2012
ENT101 Embracing the Cloud - AWS re: Invent 2012
 

Viewers also liked

CMG2013 Workshop: Netflix Cloud Native, Capacity, Performance and Cost Optimi...
CMG2013 Workshop: Netflix Cloud Native, Capacity, Performance and Cost Optimi...CMG2013 Workshop: Netflix Cloud Native, Capacity, Performance and Cost Optimi...
CMG2013 Workshop: Netflix Cloud Native, Capacity, Performance and Cost Optimi...
Adrian Cockcroft
 
Flowcon (added to for CMG) Keynote talk on how Speed Wins and how Netflix is ...
Flowcon (added to for CMG) Keynote talk on how Speed Wins and how Netflix is ...Flowcon (added to for CMG) Keynote talk on how Speed Wins and how Netflix is ...
Flowcon (added to for CMG) Keynote talk on how Speed Wins and how Netflix is ...
Adrian Cockcroft
 
Bottleneck analysis - Devopsdays Silicon Valley 2013
Bottleneck analysis - Devopsdays Silicon Valley 2013Bottleneck analysis - Devopsdays Silicon Valley 2013
Bottleneck analysis - Devopsdays Silicon Valley 2013
Adrian Cockcroft
 
Cassandra Performance and Scalability on AWS
Cassandra Performance and Scalability on AWSCassandra Performance and Scalability on AWS
Cassandra Performance and Scalability on AWS
Adrian Cockcroft
 
Yow Conference Dec 2013 Netflix Workshop Slides with Notes
Yow Conference Dec 2013 Netflix Workshop Slides with NotesYow Conference Dec 2013 Netflix Workshop Slides with Notes
Yow Conference Dec 2013 Netflix Workshop Slides with Notes
Adrian Cockcroft
 
Speeding Up Innovation
Speeding Up InnovationSpeeding Up Innovation
Speeding Up Innovation
Adrian Cockcroft
 
MicroServices at Netflix - challenges of scale
MicroServices at Netflix - challenges of scaleMicroServices at Netflix - challenges of scale
MicroServices at Netflix - challenges of scale
Sudhir Tonse
 
Operational Insight: Concepts and Examples
Operational Insight: Concepts and ExamplesOperational Insight: Concepts and Examples
Operational Insight: Concepts and Examples
royrapoport
 
Netflix Development Patterns for Scale, Performance & Availability (DMG206) |...
Netflix Development Patterns for Scale, Performance & Availability (DMG206) |...Netflix Development Patterns for Scale, Performance & Availability (DMG206) |...
Netflix Development Patterns for Scale, Performance & Availability (DMG206) |...
Amazon Web Services
 
Keeping Movies Running Amid Thunderstorms!
Keeping Movies Running Amid Thunderstorms!Keeping Movies Running Amid Thunderstorms!
Keeping Movies Running Amid Thunderstorms!
Sid Anand
 
Java Microservices with Netflix OSS & Spring
Java Microservices with Netflix OSS & Spring Java Microservices with Netflix OSS & Spring
Java Microservices with Netflix OSS & Spring
Conor Svensson
 
Cloud Tech III: Actionable Metrics
Cloud Tech III: Actionable MetricsCloud Tech III: Actionable Metrics
Cloud Tech III: Actionable Metrics
royrapoport
 
SSL Certificate Expiration and Howler Monkey's Inception
SSL Certificate Expiration and Howler Monkey's InceptionSSL Certificate Expiration and Howler Monkey's Inception
SSL Certificate Expiration and Howler Monkey's Inception
royrapoport
 
Canary Analyze All the Things
Canary Analyze All the ThingsCanary Analyze All the Things
Canary Analyze All the Things
royrapoport
 
Python Through the Back Door: Netflix Presentation at CodeMash 2014
Python Through the Back Door: Netflix Presentation at CodeMash 2014Python Through the Back Door: Netflix Presentation at CodeMash 2014
Python Through the Back Door: Netflix Presentation at CodeMash 2014
royrapoport
 

Viewers also liked (15)

CMG2013 Workshop: Netflix Cloud Native, Capacity, Performance and Cost Optimi...
CMG2013 Workshop: Netflix Cloud Native, Capacity, Performance and Cost Optimi...CMG2013 Workshop: Netflix Cloud Native, Capacity, Performance and Cost Optimi...
CMG2013 Workshop: Netflix Cloud Native, Capacity, Performance and Cost Optimi...
 
Flowcon (added to for CMG) Keynote talk on how Speed Wins and how Netflix is ...
Flowcon (added to for CMG) Keynote talk on how Speed Wins and how Netflix is ...Flowcon (added to for CMG) Keynote talk on how Speed Wins and how Netflix is ...
Flowcon (added to for CMG) Keynote talk on how Speed Wins and how Netflix is ...
 
Bottleneck analysis - Devopsdays Silicon Valley 2013
Bottleneck analysis - Devopsdays Silicon Valley 2013Bottleneck analysis - Devopsdays Silicon Valley 2013
Bottleneck analysis - Devopsdays Silicon Valley 2013
 
Cassandra Performance and Scalability on AWS
Cassandra Performance and Scalability on AWSCassandra Performance and Scalability on AWS
Cassandra Performance and Scalability on AWS
 
Yow Conference Dec 2013 Netflix Workshop Slides with Notes
Yow Conference Dec 2013 Netflix Workshop Slides with NotesYow Conference Dec 2013 Netflix Workshop Slides with Notes
Yow Conference Dec 2013 Netflix Workshop Slides with Notes
 
Speeding Up Innovation
Speeding Up InnovationSpeeding Up Innovation
Speeding Up Innovation
 
MicroServices at Netflix - challenges of scale
MicroServices at Netflix - challenges of scaleMicroServices at Netflix - challenges of scale
MicroServices at Netflix - challenges of scale
 
Operational Insight: Concepts and Examples
Operational Insight: Concepts and ExamplesOperational Insight: Concepts and Examples
Operational Insight: Concepts and Examples
 
Netflix Development Patterns for Scale, Performance & Availability (DMG206) |...
Netflix Development Patterns for Scale, Performance & Availability (DMG206) |...Netflix Development Patterns for Scale, Performance & Availability (DMG206) |...
Netflix Development Patterns for Scale, Performance & Availability (DMG206) |...
 
Keeping Movies Running Amid Thunderstorms!
Keeping Movies Running Amid Thunderstorms!Keeping Movies Running Amid Thunderstorms!
Keeping Movies Running Amid Thunderstorms!
 
Java Microservices with Netflix OSS & Spring
Java Microservices with Netflix OSS & Spring Java Microservices with Netflix OSS & Spring
Java Microservices with Netflix OSS & Spring
 
Cloud Tech III: Actionable Metrics
Cloud Tech III: Actionable MetricsCloud Tech III: Actionable Metrics
Cloud Tech III: Actionable Metrics
 
SSL Certificate Expiration and Howler Monkey's Inception
SSL Certificate Expiration and Howler Monkey's InceptionSSL Certificate Expiration and Howler Monkey's Inception
SSL Certificate Expiration and Howler Monkey's Inception
 
Canary Analyze All the Things
Canary Analyze All the ThingsCanary Analyze All the Things
Canary Analyze All the Things
 
Python Through the Back Door: Netflix Presentation at CodeMash 2014
Python Through the Back Door: Netflix Presentation at CodeMash 2014Python Through the Back Door: Netflix Presentation at CodeMash 2014
Python Through the Back Door: Netflix Presentation at CodeMash 2014
 

Similar to Gluecon 2013 - NetflixOSS Cloud Native Tutorial Introduction

AWS re:Invent 2016: Discovery Channel's Broadcast Workflows and Channel Origi...
AWS re:Invent 2016: Discovery Channel's Broadcast Workflows and Channel Origi...AWS re:Invent 2016: Discovery Channel's Broadcast Workflows and Channel Origi...
AWS re:Invent 2016: Discovery Channel's Broadcast Workflows and Channel Origi...
Amazon Web Services
 
Cloud Architecture Tutorial - Why and What (1of 3)
Cloud Architecture Tutorial - Why and What (1of 3) Cloud Architecture Tutorial - Why and What (1of 3)
Cloud Architecture Tutorial - Why and What (1of 3)
Adrian Cockcroft
 
Introducing to serverless computing and AWS lambda - Israel Clouds Meetup
Introducing to serverless computing and AWS lambda - Israel Clouds MeetupIntroducing to serverless computing and AWS lambda - Israel Clouds Meetup
Introducing to serverless computing and AWS lambda - Israel Clouds Meetup
Boaz Ziniman
 
AWS Cloud Technology And Future of Faster Modern Architecture
AWS Cloud Technology And Future of Faster Modern ArchitectureAWS Cloud Technology And Future of Faster Modern Architecture
AWS Cloud Technology And Future of Faster Modern Architecture
TMME - TECH MEETUP FOR MYANMAR ENGINEERS IN JP
 
(SPOT302) Availability: The New Kind of Innovator’s Dilemma
(SPOT302) Availability: The New Kind of Innovator’s Dilemma(SPOT302) Availability: The New Kind of Innovator’s Dilemma
(SPOT302) Availability: The New Kind of Innovator’s Dilemma
Amazon Web Services
 
Netflix Cloud Architecture and Open Source
Netflix Cloud Architecture and Open SourceNetflix Cloud Architecture and Open Source
Netflix Cloud Architecture and Open Source
aspyker
 
10 Pro Tips for Scaling Your Startup from 0-10M Users
10 Pro Tips for Scaling Your Startup from 0-10M Users10 Pro Tips for Scaling Your Startup from 0-10M Users
10 Pro Tips for Scaling Your Startup from 0-10M Users
Amazon Web Services
 
Scaling to millions of users with Amazon CloudFront - April 2017 AWS Online T...
Scaling to millions of users with Amazon CloudFront - April 2017 AWS Online T...Scaling to millions of users with Amazon CloudFront - April 2017 AWS Online T...
Scaling to millions of users with Amazon CloudFront - April 2017 AWS Online T...
Amazon Web Services
 
Web Scale Applications using NeflixOSS Cloud Platform
Web Scale Applications using NeflixOSS Cloud PlatformWeb Scale Applications using NeflixOSS Cloud Platform
Web Scale Applications using NeflixOSS Cloud Platform
Sudhir Tonse
 
WIN401_Migrating Microsoft Applications to AWS
WIN401_Migrating Microsoft Applications to AWSWIN401_Migrating Microsoft Applications to AWS
WIN401_Migrating Microsoft Applications to AWS
Amazon Web Services
 
Triangle Devops Meetup 10/2015
Triangle Devops Meetup 10/2015Triangle Devops Meetup 10/2015
Triangle Devops Meetup 10/2015
aspyker
 
Netflix in the Cloud at SV Forum
Netflix in the Cloud at SV ForumNetflix in the Cloud at SV Forum
Netflix in the Cloud at SV Forum
Adrian Cockcroft
 
The Best of re:invent 2016
The Best of re:invent 2016The Best of re:invent 2016
The Best of re:invent 2016
Amazon Web Services
 
Serverless Microservices - Real life story of a Web App that uses AWS Lambda
Serverless Microservices - Real life story of a Web App that uses AWS LambdaServerless Microservices - Real life story of a Web App that uses AWS Lambda
Serverless Microservices - Real life story of a Web App that uses AWS Lambda
Mitoc Group
 
AWS re:Invent 2016: The State of Serverless Computing (SVR311)
AWS re:Invent 2016: The State of Serverless Computing (SVR311)AWS re:Invent 2016: The State of Serverless Computing (SVR311)
AWS re:Invent 2016: The State of Serverless Computing (SVR311)
Amazon Web Services
 
The Effect of Cloud Computing on Growth of Startups and Entrepreneurship
The Effect of Cloud Computing on Growth of Startups and EntrepreneurshipThe Effect of Cloud Computing on Growth of Startups and Entrepreneurship
The Effect of Cloud Computing on Growth of Startups and Entrepreneurship
Sohail Abbasi
 
Rackspace Best Practices for DevOps on AWS
Rackspace Best Practices for DevOps on AWSRackspace Best Practices for DevOps on AWS
Rackspace Best Practices for DevOps on AWS
Amazon Web Services
 
Webinar aws 101 a walk through the aws cloud- introduction to cloud computi...
Webinar aws 101   a walk through the aws cloud- introduction to cloud computi...Webinar aws 101   a walk through the aws cloud- introduction to cloud computi...
Webinar aws 101 a walk through the aws cloud- introduction to cloud computi...
Amazon Web Services
 
Gluecon Monitoring Microservices and Containers: A Challenge
Gluecon Monitoring Microservices and Containers: A ChallengeGluecon Monitoring Microservices and Containers: A Challenge
Gluecon Monitoring Microservices and Containers: A Challenge
Adrian Cockcroft
 
Deploy Deep Learning Models on Amazon ECS - DevDay Austin 2017
Deploy Deep Learning Models on Amazon ECS - DevDay Austin 2017Deploy Deep Learning Models on Amazon ECS - DevDay Austin 2017
Deploy Deep Learning Models on Amazon ECS - DevDay Austin 2017
Amazon Web Services
 

Similar to Gluecon 2013 - NetflixOSS Cloud Native Tutorial Introduction (20)

AWS re:Invent 2016: Discovery Channel's Broadcast Workflows and Channel Origi...
AWS re:Invent 2016: Discovery Channel's Broadcast Workflows and Channel Origi...AWS re:Invent 2016: Discovery Channel's Broadcast Workflows and Channel Origi...
AWS re:Invent 2016: Discovery Channel's Broadcast Workflows and Channel Origi...
 
Cloud Architecture Tutorial - Why and What (1of 3)
Cloud Architecture Tutorial - Why and What (1of 3) Cloud Architecture Tutorial - Why and What (1of 3)
Cloud Architecture Tutorial - Why and What (1of 3)
 
Introducing to serverless computing and AWS lambda - Israel Clouds Meetup
Introducing to serverless computing and AWS lambda - Israel Clouds MeetupIntroducing to serverless computing and AWS lambda - Israel Clouds Meetup
Introducing to serverless computing and AWS lambda - Israel Clouds Meetup
 
AWS Cloud Technology And Future of Faster Modern Architecture
AWS Cloud Technology And Future of Faster Modern ArchitectureAWS Cloud Technology And Future of Faster Modern Architecture
AWS Cloud Technology And Future of Faster Modern Architecture
 
(SPOT302) Availability: The New Kind of Innovator’s Dilemma
(SPOT302) Availability: The New Kind of Innovator’s Dilemma(SPOT302) Availability: The New Kind of Innovator’s Dilemma
(SPOT302) Availability: The New Kind of Innovator’s Dilemma
 
Netflix Cloud Architecture and Open Source
Netflix Cloud Architecture and Open SourceNetflix Cloud Architecture and Open Source
Netflix Cloud Architecture and Open Source
 
10 Pro Tips for Scaling Your Startup from 0-10M Users
10 Pro Tips for Scaling Your Startup from 0-10M Users10 Pro Tips for Scaling Your Startup from 0-10M Users
10 Pro Tips for Scaling Your Startup from 0-10M Users
 
Scaling to millions of users with Amazon CloudFront - April 2017 AWS Online T...
Scaling to millions of users with Amazon CloudFront - April 2017 AWS Online T...Scaling to millions of users with Amazon CloudFront - April 2017 AWS Online T...
Scaling to millions of users with Amazon CloudFront - April 2017 AWS Online T...
 
Web Scale Applications using NeflixOSS Cloud Platform
Web Scale Applications using NeflixOSS Cloud PlatformWeb Scale Applications using NeflixOSS Cloud Platform
Web Scale Applications using NeflixOSS Cloud Platform
 
WIN401_Migrating Microsoft Applications to AWS
WIN401_Migrating Microsoft Applications to AWSWIN401_Migrating Microsoft Applications to AWS
WIN401_Migrating Microsoft Applications to AWS
 
Triangle Devops Meetup 10/2015
Triangle Devops Meetup 10/2015Triangle Devops Meetup 10/2015
Triangle Devops Meetup 10/2015
 
Netflix in the Cloud at SV Forum
Netflix in the Cloud at SV ForumNetflix in the Cloud at SV Forum
Netflix in the Cloud at SV Forum
 
The Best of re:invent 2016
The Best of re:invent 2016The Best of re:invent 2016
The Best of re:invent 2016
 
Serverless Microservices - Real life story of a Web App that uses AWS Lambda
Serverless Microservices - Real life story of a Web App that uses AWS LambdaServerless Microservices - Real life story of a Web App that uses AWS Lambda
Serverless Microservices - Real life story of a Web App that uses AWS Lambda
 
AWS re:Invent 2016: The State of Serverless Computing (SVR311)
AWS re:Invent 2016: The State of Serverless Computing (SVR311)AWS re:Invent 2016: The State of Serverless Computing (SVR311)
AWS re:Invent 2016: The State of Serverless Computing (SVR311)
 
The Effect of Cloud Computing on Growth of Startups and Entrepreneurship
The Effect of Cloud Computing on Growth of Startups and EntrepreneurshipThe Effect of Cloud Computing on Growth of Startups and Entrepreneurship
The Effect of Cloud Computing on Growth of Startups and Entrepreneurship
 
Rackspace Best Practices for DevOps on AWS
Rackspace Best Practices for DevOps on AWSRackspace Best Practices for DevOps on AWS
Rackspace Best Practices for DevOps on AWS
 
Webinar aws 101 a walk through the aws cloud- introduction to cloud computi...
Webinar aws 101   a walk through the aws cloud- introduction to cloud computi...Webinar aws 101   a walk through the aws cloud- introduction to cloud computi...
Webinar aws 101 a walk through the aws cloud- introduction to cloud computi...
 
Gluecon Monitoring Microservices and Containers: A Challenge
Gluecon Monitoring Microservices and Containers: A ChallengeGluecon Monitoring Microservices and Containers: A Challenge
Gluecon Monitoring Microservices and Containers: A Challenge
 
Deploy Deep Learning Models on Amazon ECS - DevDay Austin 2017
Deploy Deep Learning Models on Amazon ECS - DevDay Austin 2017Deploy Deep Learning Models on Amazon ECS - DevDay Austin 2017
Deploy Deep Learning Models on Amazon ECS - DevDay Austin 2017
 

Recently uploaded

"$10 thousand per minute of downtime: architecture, queues, streaming and fin...
"$10 thousand per minute of downtime: architecture, queues, streaming and fin..."$10 thousand per minute of downtime: architecture, queues, streaming and fin...
"$10 thousand per minute of downtime: architecture, queues, streaming and fin...
Fwdays
 
Introducing BoxLang : A new JVM language for productivity and modularity!
Introducing BoxLang : A new JVM language for productivity and modularity!Introducing BoxLang : A new JVM language for productivity and modularity!
Introducing BoxLang : A new JVM language for productivity and modularity!
Ortus Solutions, Corp
 
inQuba Webinar Mastering Customer Journey Management with Dr Graham Hill
inQuba Webinar Mastering Customer Journey Management with Dr Graham HillinQuba Webinar Mastering Customer Journey Management with Dr Graham Hill
inQuba Webinar Mastering Customer Journey Management with Dr Graham Hill
LizaNolte
 
PRODUCT LISTING OPTIMIZATION PRESENTATION.pptx
PRODUCT LISTING OPTIMIZATION PRESENTATION.pptxPRODUCT LISTING OPTIMIZATION PRESENTATION.pptx
PRODUCT LISTING OPTIMIZATION PRESENTATION.pptx
christinelarrosa
 
"What does it really mean for your system to be available, or how to define w...
"What does it really mean for your system to be available, or how to define w..."What does it really mean for your system to be available, or how to define w...
"What does it really mean for your system to be available, or how to define w...
Fwdays
 
What is an RPA CoE? Session 1 – CoE Vision
What is an RPA CoE?  Session 1 – CoE VisionWhat is an RPA CoE?  Session 1 – CoE Vision
What is an RPA CoE? Session 1 – CoE Vision
DianaGray10
 
Harnessing the Power of NLP and Knowledge Graphs for Opioid Research
Harnessing the Power of NLP and Knowledge Graphs for Opioid ResearchHarnessing the Power of NLP and Knowledge Graphs for Opioid Research
Harnessing the Power of NLP and Knowledge Graphs for Opioid Research
Neo4j
 
QR Secure: A Hybrid Approach Using Machine Learning and Security Validation F...
QR Secure: A Hybrid Approach Using Machine Learning and Security Validation F...QR Secure: A Hybrid Approach Using Machine Learning and Security Validation F...
QR Secure: A Hybrid Approach Using Machine Learning and Security Validation F...
AlexanderRichford
 
Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors
Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectorsConnector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors
Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors
DianaGray10
 
A Deep Dive into ScyllaDB's Architecture
A Deep Dive into ScyllaDB's ArchitectureA Deep Dive into ScyllaDB's Architecture
A Deep Dive into ScyllaDB's Architecture
ScyllaDB
 
Christine's Product Research Presentation.pptx
Christine's Product Research Presentation.pptxChristine's Product Research Presentation.pptx
Christine's Product Research Presentation.pptx
christinelarrosa
 
Demystifying Knowledge Management through Storytelling
Demystifying Knowledge Management through StorytellingDemystifying Knowledge Management through Storytelling
Demystifying Knowledge Management through Storytelling
Enterprise Knowledge
 
Call Girls Chandigarh🔥7023059433🔥Agency Profile Escorts in Chandigarh Availab...
Call Girls Chandigarh🔥7023059433🔥Agency Profile Escorts in Chandigarh Availab...Call Girls Chandigarh🔥7023059433🔥Agency Profile Escorts in Chandigarh Availab...
Call Girls Chandigarh🔥7023059433🔥Agency Profile Escorts in Chandigarh Availab...
manji sharman06
 
QA or the Highway - Component Testing: Bridging the gap between frontend appl...
QA or the Highway - Component Testing: Bridging the gap between frontend appl...QA or the Highway - Component Testing: Bridging the gap between frontend appl...
QA or the Highway - Component Testing: Bridging the gap between frontend appl...
zjhamm304
 
"Scaling RAG Applications to serve millions of users", Kevin Goedecke
"Scaling RAG Applications to serve millions of users",  Kevin Goedecke"Scaling RAG Applications to serve millions of users",  Kevin Goedecke
"Scaling RAG Applications to serve millions of users", Kevin Goedecke
Fwdays
 
LF Energy Webinar: Carbon Data Specifications: Mechanisms to Improve Data Acc...
LF Energy Webinar: Carbon Data Specifications: Mechanisms to Improve Data Acc...LF Energy Webinar: Carbon Data Specifications: Mechanisms to Improve Data Acc...
LF Energy Webinar: Carbon Data Specifications: Mechanisms to Improve Data Acc...
DanBrown980551
 
Apps Break Data
Apps Break DataApps Break Data
Apps Break Data
Ivo Velitchkov
 
From Natural Language to Structured Solr Queries using LLMs
From Natural Language to Structured Solr Queries using LLMsFrom Natural Language to Structured Solr Queries using LLMs
From Natural Language to Structured Solr Queries using LLMs
Sease
 
Session 1 - Intro to Robotic Process Automation.pdf
Session 1 - Intro to Robotic Process Automation.pdfSession 1 - Intro to Robotic Process Automation.pdf
Session 1 - Intro to Robotic Process Automation.pdf
UiPathCommunity
 
Poznań ACE event - 19.06.2024 Team 24 Wrapup slidedeck
Poznań ACE event - 19.06.2024 Team 24 Wrapup slidedeckPoznań ACE event - 19.06.2024 Team 24 Wrapup slidedeck
Poznań ACE event - 19.06.2024 Team 24 Wrapup slidedeck
FilipTomaszewski5
 

Recently uploaded (20)

"$10 thousand per minute of downtime: architecture, queues, streaming and fin...
"$10 thousand per minute of downtime: architecture, queues, streaming and fin..."$10 thousand per minute of downtime: architecture, queues, streaming and fin...
"$10 thousand per minute of downtime: architecture, queues, streaming and fin...
 
Introducing BoxLang : A new JVM language for productivity and modularity!
Introducing BoxLang : A new JVM language for productivity and modularity!Introducing BoxLang : A new JVM language for productivity and modularity!
Introducing BoxLang : A new JVM language for productivity and modularity!
 
inQuba Webinar Mastering Customer Journey Management with Dr Graham Hill
inQuba Webinar Mastering Customer Journey Management with Dr Graham HillinQuba Webinar Mastering Customer Journey Management with Dr Graham Hill
inQuba Webinar Mastering Customer Journey Management with Dr Graham Hill
 
PRODUCT LISTING OPTIMIZATION PRESENTATION.pptx
PRODUCT LISTING OPTIMIZATION PRESENTATION.pptxPRODUCT LISTING OPTIMIZATION PRESENTATION.pptx
PRODUCT LISTING OPTIMIZATION PRESENTATION.pptx
 
"What does it really mean for your system to be available, or how to define w...
"What does it really mean for your system to be available, or how to define w..."What does it really mean for your system to be available, or how to define w...
"What does it really mean for your system to be available, or how to define w...
 
What is an RPA CoE? Session 1 – CoE Vision
What is an RPA CoE?  Session 1 – CoE VisionWhat is an RPA CoE?  Session 1 – CoE Vision
What is an RPA CoE? Session 1 – CoE Vision
 
Harnessing the Power of NLP and Knowledge Graphs for Opioid Research
Harnessing the Power of NLP and Knowledge Graphs for Opioid ResearchHarnessing the Power of NLP and Knowledge Graphs for Opioid Research
Harnessing the Power of NLP and Knowledge Graphs for Opioid Research
 
QR Secure: A Hybrid Approach Using Machine Learning and Security Validation F...
QR Secure: A Hybrid Approach Using Machine Learning and Security Validation F...QR Secure: A Hybrid Approach Using Machine Learning and Security Validation F...
QR Secure: A Hybrid Approach Using Machine Learning and Security Validation F...
 
Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors
Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectorsConnector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors
Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors
 
A Deep Dive into ScyllaDB's Architecture
A Deep Dive into ScyllaDB's ArchitectureA Deep Dive into ScyllaDB's Architecture
A Deep Dive into ScyllaDB's Architecture
 
Christine's Product Research Presentation.pptx
Christine's Product Research Presentation.pptxChristine's Product Research Presentation.pptx
Christine's Product Research Presentation.pptx
 
Demystifying Knowledge Management through Storytelling
Demystifying Knowledge Management through StorytellingDemystifying Knowledge Management through Storytelling
Demystifying Knowledge Management through Storytelling
 
Call Girls Chandigarh🔥7023059433🔥Agency Profile Escorts in Chandigarh Availab...
Call Girls Chandigarh🔥7023059433🔥Agency Profile Escorts in Chandigarh Availab...Call Girls Chandigarh🔥7023059433🔥Agency Profile Escorts in Chandigarh Availab...
Call Girls Chandigarh🔥7023059433🔥Agency Profile Escorts in Chandigarh Availab...
 
QA or the Highway - Component Testing: Bridging the gap between frontend appl...
QA or the Highway - Component Testing: Bridging the gap between frontend appl...QA or the Highway - Component Testing: Bridging the gap between frontend appl...
QA or the Highway - Component Testing: Bridging the gap between frontend appl...
 
"Scaling RAG Applications to serve millions of users", Kevin Goedecke
"Scaling RAG Applications to serve millions of users",  Kevin Goedecke"Scaling RAG Applications to serve millions of users",  Kevin Goedecke
"Scaling RAG Applications to serve millions of users", Kevin Goedecke
 
LF Energy Webinar: Carbon Data Specifications: Mechanisms to Improve Data Acc...
LF Energy Webinar: Carbon Data Specifications: Mechanisms to Improve Data Acc...LF Energy Webinar: Carbon Data Specifications: Mechanisms to Improve Data Acc...
LF Energy Webinar: Carbon Data Specifications: Mechanisms to Improve Data Acc...
 
Apps Break Data
Apps Break DataApps Break Data
Apps Break Data
 
From Natural Language to Structured Solr Queries using LLMs
From Natural Language to Structured Solr Queries using LLMsFrom Natural Language to Structured Solr Queries using LLMs
From Natural Language to Structured Solr Queries using LLMs
 
Session 1 - Intro to Robotic Process Automation.pdf
Session 1 - Intro to Robotic Process Automation.pdfSession 1 - Intro to Robotic Process Automation.pdf
Session 1 - Intro to Robotic Process Automation.pdf
 
Poznań ACE event - 19.06.2024 Team 24 Wrapup slidedeck
Poznań ACE event - 19.06.2024 Team 24 Wrapup slidedeckPoznań ACE event - 19.06.2024 Team 24 Wrapup slidedeck
Poznań ACE event - 19.06.2024 Team 24 Wrapup slidedeck
 

Gluecon 2013 - NetflixOSS Cloud Native Tutorial Introduction

  • 1. Introduction: Building Using The NetflixOSS Architecture May 2013 Adrian Cockcroft @adrianco #netflixcloud @NetflixOSS http://www.linkedin.com/in/adriancockcroft
  • 2. Presentation vs. Tutorial • Presentation – Short duration, focused subject – One presenter to many anonymous audience – A few questions at the end • Tutorial – Time to explore in and around the subject – Tutor gets to know the audience – Discussion, rat-holes, “bring out your dead”
  • 3. Introduction – Who are you? Netflix Open Source Cloud Prize Cloud Native – More details NetflixOSS – Cloud Native On-Ramp
  • 4. Adrian Cockcroft • Director, Architecture for Cloud Systems, Netflix Inc. – Previously Director for Personalization Platform • Distinguished Availability Engineer, eBay Inc. 2004-7 – Founding member of eBay Research Labs • Distinguished Engineer, Sun Microsystems Inc. 1988-2004 – 2003-4 Chief Architect High Performance Technical Computing – 2001 Author: Capacity Planning for Web Services – 1999 Author: Resource Management – 1995 & 1998 Author: Sun Performance and Tuning – 1996 Japanese Edition of Sun Performance and Tuning • SPARC & Solarisパフォーマンスチューニング (サンソフトプレスシリー ズ) • More – Twitter @adrianco – Blog http://perfcap.blogspot.com – Presentations at http://www.slideshare.net/adrianco
  • 5. Attendee Introductions • Who are you, where do you work • Why are you here today, what do you need • “Bring out your dead” – Do you have a specific problem or question? – One sentence elevator pitch • What instrument do you play?
  • 6. Boosting the @NetflixOSS Ecosystem See netflix.github.com
  • 7. In 2012 Netflix Engineering won this..
  • 8. We’d like to give out prizes too But what for? Contributions to NetflixOSS! Shared under Apache license Located on github
  • 9.
  • 10. How long do you have? Entries open March 13th Entries close September 15th Six months…
  • 11. Who can win? Almost anyone, anywhere… Except current or former Netflix or AWS employees
  • 12. Who decides who wins? Nominating Committee Panel of Judges
  • 13. Judges Aino Corry Program Chair for Qcon/GOTO Martin Fowler Chief Scientist ThoughtworksSimon Wardley Strategist Yury Izrailevsky VP Cloud Netflix Werner Vogels CTO Amazon Joe Weinman SVP Telx, Author “Cloudonomics”
  • 14. What are Judges Looking For? Eligible, Apache 2.0 licensed NetflixOSS project pull requests Original and useful contribution to NetflixOSS Good code quality and structure Documentation on how to build and run it Code that successfully builds and passes a test suite Evidence that code is in use by other projects, or is running in production A large number of watchers, stars and forks on github
  • 15. What do you win? One winner in each of the 10 categories Ticket and expenses to attend AWS Re:Invent 2013 in Las Vegas A Trophy
  • 16. How do you enter? Get a (free) github account Fork github.com/netflix/cloud-prize Send us your email address Describe and build your entry Twitter #cloudprize
  • 17. Entrants Netflix Engineering Six Judges Winners Nominations Conforms to Rules Working Code Community Traction Categories Registration Opened March 13 Github Apache Licensed Contributions Github Close Entries September 15 Github Award Ceremony Dinner November AWS Re:Invent Ten Prize Categories $10K cash $5K AWS AWS Re:Invent Tickets Trophy
  • 18. Cloud Native Recap the keynote in much more detail and discussion
  • 19. A new engineering challenge Construct a highly agile and highly available service from ephemeral and often broken components
  • 21. Netflix Streaming A Cloud Native Application based on an open source platform
  • 22. Netflix Member Web Site Home Page Personalization Driven – How Does It Work?
  • 23. How Netflix Streaming Works Customer Device (PC, PS3, TV…) Web Site or Discovery API User Data Personalization Streaming API DRM QoS Logging OpenConnect CDN Boxes CDN Management and Steering Content Encoding Consumer Electronics AWS Cloud Services CDN Edge Locations
  • 24. Real Web Server Dependencies Flow (Netflix Home page business transaction as seen by AppDynamics) Start Here memcached Cassandra Web service S3 bucket Personalization movie group choosers (for US, Canada and Latam) Each icon is three to a few hundred instances across three AWS zones
  • 25. New Cloud Native Patterns Micro-services and Chaos engines Highly available systems composed from ephemeral components Open Source is the default
  • 27. The AWS Question Why does Netflix use AWS when Amazon Prime is a competitor?
  • 28. Netflix vs. Amazon Prime • Do retailers competing with Amazon use AWS? – Yes, lots of them, Netflix is no different • Does Prime have a platform advantage? – No, because Netflix gets to run on AWS • Does Netflix take Amazon Prime seriously? – Yes, but so far Prime isn’t impacting our business
  • 29. Amazon Video 1.31% 18x Prime 25x Prime Nov 2012 Streaming Bandwidth March 2013 Mean Bandwidth +39% 6mo
  • 30. The Google Cloud Question Why doesn’t Netflix use Google Cloud as well as AWS?
  • 31. Google Cloud – Wait and See Pro’s • Cloud Native • Huge scale for internal apps • Exposing internal services • Nice clean API model • Starting a price war • Fast for what it does • Rapid start & minute billing Con’s • In beta until last week • No big customers yet • Missing many key features • Different arch model • Missing billing options • No SSD or huge instances • Zone maintenance windows But: Anyone interested is welcome to port NetflixOSS components to Google Cloud
  • 32. Cloud Wars: Price and Performance AWS vs. GCS War Private Cloud What Changed: Everyone using AWS or GCS gets the price cuts and performance improvements, as they happen. No need to switch vendor. No Change: Locked in for three years.
  • 33. The DIY Question Why doesn’t Netflix build and run its own cloud?
  • 34. Fitting Into Public Scale Public Grey Area Private 1,000 Instances 100,000 Instances Netflix FacebookStartups
  • 35. How big is Public? AWS upper bound estimate based on the number of public IP Addresses Every provisioned instance gets a public IP by default AWS Maximum Possible Instance Count 3.7 Million Growth >10x in Three Years, >2x Per Annum
  • 36. The Alternative Supplier Question What if there is no clear leader for a feature, or AWS doesn’t have what we need?
  • 37. Things We Don’t Use AWS For SaaS Applications – Pagerduty, Appdynamics Content Delivery Service DNS Service
  • 38. CDN Scale AWS CloudFront Akamai Limelight Level 3 Netflix Openconnect YouTube Gigabits Terabits NetflixFacebookStartups
  • 39. Content Delivery Service Open Source Hardware Design + FreeBSD, bird, nginx see openconnect.netflix.com
  • 40. DNS Service AWS Route53 is missing too many features Multiple vendor strategy Dyn, Ultra, Route53 Abstracted (broken) DNS APIs with Denominator
  • 41. Availability Questions Is it running yet? How many places is it running in? How far apart are those places?
  • 42.
  • 43. Netflix Outages • Running very fast with scissors – Mostly self inflicted – bugs, mistakes from pace of change – Some caused by AWS bugs and mistakes • Incident Life-cycle Management by Platform Team – No runbooks, no operational changes by the SREs – Tools to identify what broke and call the right developer • Next step is multi-region – Investigating and building in stages during 2013 – Could have prevented some of our 2012 outages
  • 44. Managing Multi-Region Availability Cassandra Replicas Zone A Cassandra Replicas Zone B Cassandra Replicas Zone C Regional Load Balancers Cassandra Replicas Zone A Cassandra Replicas Zone B Cassandra Replicas Zone C Regional Load Balancers UltraDNS DynECT DNS AWS Route53 Denominator – manage traffic via multiple DNS providers Denominator
  • 45. Cloud Native Big Data Size the cluster to the data Size the cluster to the questions Never wait for space or answers
  • 46. Netflix Dataoven Data Warehouse Over 2 Petabytes Ursula Aegisthus Data Pipelines From cloud Services ~100 Billion Events/day From C* Terabytes of Dimension data Hadoop Clusters – AWS EMR 1300 nodes 800 nodes Multiple 150 nodes Nightly RDS Metadata Gateways Tools
  • 47. Cloud Native Patterns Master copies of data are cloud resident Dynamically provisioned micro-services Services are distributed and ephemeral
  • 48. Cloud Native Architecture Distributed Quorum NoSQL Datastores Autoscaled Micro Services Autoscaled Micro Services Clients Things JVM JVM JVM JVM Cassandra Cassandra Cassandra Memcached JVM Zone A Zone B Zone C
  • 49. Non-Native Cloud Architecture Datacenter Dinosaurs Cloudy Buffer Agile Mobile Mammals iOS/Android App Servers MySQL Legacy Apps
  • 50. How to get to Cloud Native? Freedom and Responsibility for Developers Decentralize and Automate Ops Activities Integrate DevOps into the Business Organization
  • 51. Four Transitions • Management: Integrated Roles in a Single Organization – Business, Development, Operations -> BusDevOps • Developers: Denormalized Data – NoSQL – Decentralized, scalable, available, polyglot • Responsibility from Ops to Dev: Continuous Delivery – Decentralized small daily production updates • Responsibility from Ops to Dev: Agile Infrastructure - Cloud – Hardware in minutes, provisioned directly by developers
  • 52. Netflix BusDevOps Organization Chief Product Officer VP Product Management Directors Product VP UI Engineering Directors Development Developers + DevOps UI Data Sources AWS VP Discovery Engineering Directors Development Developers + DevOps Discovery Data Sources AWS VP Platform Directors Platform Developers + DevOps Platform Data Sources AWS Denormalized, independently updated and scaled data Cloud, independently updated and scaled infrastructure Code, independently updated continuous delivery
  • 55. Ephemeral Instances • Largest services are autoscaled • Average lifetime of an instance is 36 hours P u s h Autoscale Up Autoscale Down
  • 56. A Cloud Native Open Source Platform See netflix.github.com
  • 57. Three Questions Why is Netflix doing this? How does it all fit together? What is coming next?
  • 58. Beware of Geeks Bearing Gifts: Strategies for an Increasingly Open Economy Simon Wardley - Researcher at the Leading Edge Forum
  • 59. How did Netflix get ahead? Netflix BusDevOps Org • Doing it since 2009 • SaaS Applications • PaaS for agility • Public IaaS for AWS features • Big data in the cloud • Integrating many APIs • FOSS from github • Renting hardware for 1hr • Coding in Java/Groovy/Scala Traditional IT Operations • Taking their time • Pilot private cloud projects • Beta quality installations • Small scale • Integrating several vendors • Paying big $ for software • Paying big $ for consulting • Buying hardware for 3yrs • Hacking at scripts
  • 60. Netflix Platform Evolution Bleeding Edge Innovation Common Pattern Shared Pattern 2009-2010 2011-2012 2013-2014 Netflix ended up several years ahead of the industry, but it’s becoming commoditized now
  • 61. Making it easy to follow Exploring the wild west each time vs. laying down a shared route
  • 62. Establish our solutions as Best Practices / Standards Hire, Retain and Engage Top Engineers Build up Netflix Technology Brand Benefit from a shared ecosystem Goals
  • 63. How does it all fit together?
  • 65. Github NetflixOSS Source AWS Base AMI Maven Central Cloudbees Jenkins Aminator Bakery Dynaslave AWS Build Slaves Asgard (+ Frigga) Console AWS Baked AMIs Odin Orchestration API AWS Account NetflixOSS Continuous Build and Deployment
  • 66. AWS Account Asgard Console Archaius Config Service Cross region Priam C* Pytheas Dashboards Atlas Monitoring Genie, Lipstick Hadoop Services AWS Usage Cost Monitoring Multiple AWS Regions Eureka Registry Exhibitor ZK Edda History Simian Army 3 AWS Zones Application Clusters Autoscale Groups Instances Priam Cassandra Persistent Storage Evcache Memcached Ephemeral Storage NetflixOSS Services Scope
  • 67. •Baked AMI – Tomcat, Apache, your code •Governator – Guice based dependency injection •Archaius – dynamic configuration properties client •Eureka - service registration client Initialization •Karyon - Base Server for inbound requests •RxJava – Reactive pattern •Hystrix/Turbine – dependencies and real-time status •Ribbon - REST Client for outbound calls Service Requests •Astyanax – Cassandra client and pattern library •Evcache – Zone aware Memcached client •Curator – Zookeeper patterns •Denominator – DNS routing abstraction Data Access •Blitz4j – non-blocking logging •Servo – metrics export for autoscaling •Atlas – high volume instrumentation Logging NetflixOSS Instance Libraries
  • 68. •CassJmeter – Load testing for Cassandra •Circus Monkey – Test account reservation rebalancingTest Tools •Janitor Monkey – Cleans up unused resources •Efficiency Monkey •Doctor Monkey •Howler Monkey – Complains about AWS limits Maintenance •Chaos Monkey – Kills Instances •Chaos Gorilla – Kills Availability Zones •Chaos Kong – Kills Regions •Latency Monkey – Latency and error injection Availability •Security Monkey – security group and S3 bucket permissions •Conformity Monkey – architectural pattern warningsSecurity NetflixOSS Testing and Automation
  • 69. More Use Cases More Features Better portability Higher availability Easier to deploy Contributions from end users Contributions from vendors What’s Coming Next?
  • 70. Vendor Driven Portability Interest in using NetflixOSS for Enterprise Private Clouds “It’s done when it runs Asgard” Functionally complete Demonstrated March Release 3.3 in 2Q13 Some vendor interest Needs AWS compatible Autoscaler Some vendor interest Many missing features “Confused” AWS API strategy
  • 71. AWS 2009 Baseline features needed to support NetflixOSS Eucalyptus 3.3
  • 72. Functionality and scale now, portability coming Moving from parts to a platform in 2013 Netflix is fostering a cloud native ecosystem Rapid Evolution - Low MTBIAMSH (Mean Time Between Idea And Making Stuff Happen)
  • 73. Takeaway NetflixOSS makes it easier for everyone to become Cloud Native @adrianco #netflixcloud @NetflixOSS

Editor's Notes

  1. Hive – thin metadata layer on top of S3Used for ad-hoc analytics (Ursula for merge ETL)HiveQL gets compiled into set of MR jobs (1 -> many)Is a CLI – runs on the gateways, not like a relational DB server, or a service that the query gets shipped toPig – used for ETL (can create DAGs, workflows for Hadoop processes)Pig scripts also get compiled into MR jobsJava – straight up Hadoop, not for the faint of heart. Some recommendation algorithms are in Hadoop.Python/Java – UDFsApplications such as Sting use the tools on some gateway to access all the various componentsNext – focus on two key components: Data & Clusters
  2. When Netflix first moved to cloud it was bleeding edge innovation, we figured stuff out and made stuff up from first principles. Over the last two years more large companies have moved to cloud, and the principles, practices and patterns have become better understood and adopted. At this point there is intense interest in how Netflix runs in the cloud, and several forward looking organizations adopting our architectures and starting to use some of the code we have shared. Over the coming years, we want to make it easier for people to share the patterns we use.
  3. The railroad made it possible for California to be developed quickly, by creating an easy to follow path we can create a much bigger ecosystem around the Netflix platform