SlideShare a Scribd company logo
1 of 108
Download to read offline
Season 2 Episode 1
March 12, 2014
Evening Outline
Lightning Talks:
- S3mper
- PigPen
- STAASH
- Dynomite
- Aegisthus
- Suro
- Zeno
- Lipstick on GCE
- AnsWerS
- IBM
- Coursera
41 projects… Now what?
● Cohesive platform
● Workshops / Training / Documentation
● Participate and contribute :
netflixoss@netflix.com
Lightning talks
Lipstick, Hadoop, and
Big Data on the Google Cloud
Matt Bookman
Solutions Architect
Google Confidential and Proprietary
Google Compute Engine - VMs in Google Datacenters
● Public Preview - May 2013
● General Availability - December 2013
Google Confidential and Proprietary
Demo (Summer 2013): Pig on Compute Engine
Sweet demo!
Google Confidential and Proprietary
Netflix OSS Meetup - July 17, 2013
Google Confidential and Proprietary
Lipstick - Providing insights
Google Confidential and Proprietary
Google Confidential and Proprietary
Hadoop on GCE + Cloud Storage (GCS) Connector
Accenture:
Cloud vs. Bare-Metal
● Cloud-based Hadoop
deployments offer
better price-
performance ratios
than bare-metal
● Cloud’s virtualization
expands performance-
tuning opportunities
● Using remote storage
outperforms local disk
HDFS
Google Confidential and Proprietary
Data in GCS, Lipstick DB in Cloud SQL
Google Cloud Platform
Output Data
Lipstick
Database
Hadoop Master
MapReduce
JobTracker
Hadoop
Worker
MapReduce
TaskTrackerHadoop Worker
MapReduce
TaskTracker
Hadoop Worker
MapReduce
TaskTrackerLipstick Server
Input Data
Google Confidential and Proprietary
● Netflix Lipstick on Google Compute Engine
https://cloud.google.com/developers/articles/netflix-lipstick-on-google-compute-engine
● GCS Connector for Hadoop
https://developers.google.com/hadoop/google-cloud-storage-connector
● Cloud-based Hadoop Deployments: Benefits and Considerations
http://www.accenture.com/SiteCollectionDocuments/PDF/Accenture-Cloud-Based-Hadoop-Deployments-Benefits-and-Considerations.
pdf
● Apache Hadoop, Hive, and Pig on Google Compute Engine
https://cloud.google.com/developers/articles/apache-hadoop-hive-and-pig-on-google-compute-engine
Resources
Google Confidential and Proprietary
Thank you
@Answers4AWS
Cloud Prize and Beyond
Peter Sankauskas
@pas256
@Answers4AWS
March 2013
@Answers4AWS
First idea
• AsgardFormation
• CloudFormation for Asgard
@Answers4AWS
@Answers4AWS
@Answers4AWS
Requirements
• AsgardFormation
• Asgard running
• AWS Credentials
• IAM user
• Policy
• Security Group
• EC2 instance
• Asgard downloaded and
configured
• Tomcat downloaded and
configured
• Java downloaded and
installed
• Linux configured
@Answers4AWS
@Answers4AWS
Asgard playbook
• Base
• Install usual Linux packages
• Basic system hardening and security
packages
• Oracle Java 7
• Tomcat 7
• Asgard
• Latest release from GitHub
@Answers4AWS
Other playbooks
• Eureka
• Edda
• Simian Army
• Ice
• Aminator
• Genie
@Answers4AWS
AMIs
• Initially built using my own scripts based
on Eric Hammond’s (@esh) work
• Then using Aminator
• Created Ubuntu Foundation AMIs
• Added the Ansible Provisioner for
Aminator
• Put a couple of them on the AWS
Marketplace for free
@Answers4AWS
CloudFormation
• One-click deploy
• Well, about 10 going through the AWS
Web Console wizard
• Designed to get you up and running
quickly
• Test it out, see if you like it
• NOT production quality
• No real security
• No HA
@Answers4AWS
@Answers4AWS
What’s next?
@Answers4AWS
Do you do this? (this is not my slide)
@Answers4AWS
@Answers4AWS
Beta users
• From a successful CI build
• To a Fully Baked AMI
• Use in Testing and Production
• Without you doing anything
• ZERO clicks
• Signups are open
@Answers4AWS
Thank you
http://bakery.answersforaws.com/
bakery@answersforaws.com
See me at the demo station
Peter Sankauskas
@pas256
IBM Scalable Services Fabric
for Netflix S2E1 Meetup
Andrew Spyker
@aspyker
History and Future
2012
SPECjEnterprise
2013
AcmeAir Run
On IBM Cloud at
“Web Scale”
2014
Scalable Services
Fabric internally for
IBM Services
Scalable Services
Fabric SaaS and On-Prem?
Sample application
cloud prize work
AcmeAir Cloud/Mobile
Sample/Benchmark born
Codename:
BlueMix
Portability cloud
prize work
Scalable Service Fabric Work
Netflix OSS IBM port/enablement
Netflix “Zen” of Cloud • Worked with initial services to enable cloud native arch
• Worked with initial services to enable NetflixOSS usages
• Created scorecard and tests for “cloud native” readiness
Highly Available IaaS and
Cloud Services
• Deployment across multiple IBM SoftLayer IaaS datacenters and global
and local load balancers
• Complete automation via IBM SoftLayer IaaS API’s
• Ensured facilities for automatic failure recovery
Micro-service Runtimes
(Karyon, Eureka Client, Ribbon,
Hystrix, Archaius)
• Ported to work with IBM SoftLayer IaaS and on the WebSphere Liberty
Profile application server
• Created “eureka-sidecar” for non-Java runtimes and ElasticSearch
discovery
Netflix OSS Servers
(Asgard, Eureka Server,
Turbine)
• Ported to work with IBM SoftLayer IaaS + RightScale
• Operationalized HA and secure deployments for multiple service tenants
Adopted Chaos Testing • Ported Chaos Monkey to IBM SoftLayer IaaS
• Performed manual Chaos Gorilla validation on services
Worked through devops tool
chain
• Worked with initial services to enable continuous delivery with devops
(and imagine baking via Animator like tool)
Come meet the team!
Looks like … Tweets from … Talks about …
Adolfo @adolforod
API Management and Cloud Integration, user of
NetflixOSS platform. Appliances in the cloud.
Brian @bkmartin IBM BlueMix (PaaS), enabling composable apps in PaaS
Darrell
IBM Research focusing on NetflixOSS devops and on-
premise deployments
David @dcurrie
WebSphere Liberty Profile application server NetflixOSS
development and PaaS integration
Jonathan @ma4jpb
NetflixOSS portability across many aspects
Cloud messaging (in relation to Suro)
Matt @matrober
API Management, user of NetflixOSS platform
Converted service to be cloud native
Rachel @rreinitz
IBM Services, interested in helping you get to this cloud
native in SaaS and on-premise
Ricky
@rickymoorhouse API Management, user of NetflixOSS platform
Creator of Imaginator
Will @auwilli98 API Management operations, user of NetflixOSS platform
Priam +
Aegisthus
@Coursera
NetflixOSS Meetup
Introduction
@DanielChiaJH
Software Engineer, Infrastructure Team
Coursera
Overview
• Philosophy
• Priam
• Aegisthus
• Conclusion
Philosophy
• Architecture Patterns
• Use what we can
• Incorporate the spirit of others
Priam – Wins
• Token Management
• S3 Backup + Restore
• Config
Priam – Next Steps?
• SimpleDB -> DynamoDB
• Backups blow out OS disk buffer cache
• Compatibility with newer C* versions
Aegisthus - Wins
• Novel workflow
• Data reduced to one authoritative copy
• Possibility for incremental jobs
Aegisthus – What Next?
• C* 1.2 / 2.0
• CQL3
• Priam <–> Aegisthus
• Better compressed SSTable support
Conclusion
• Come chat with me!
• Especially if you have similar goals to me
Zeno
● In-memory data distribution platform
● Contains tools for:
○ data quality management
○ data serialization
● We use it to distribute and keep up to date
gigabytes of video metadata on tens of
thousands of servers across the globe
Zeno
Why in-memory data?
- Netflix serves billions of requests
per day
- Each request requires metadata
about many movies to answer
Zeno
Netflix Use Case:
● Gigabytes of in-memory data
● Hundreds of thousands of in-memory cache
requests per second, per application
instance
● Tens of thousands of application instances
Distribution
FastBlob:
Binary serialization of a complete
state of data, and/or the changes
in data over time.
Serialization format designed to
propagate, and keep up to date, a
large amount of in-memory data
across many servers.
Optimized for: memory GC effects,
memory footprint, data transfer
size, deserialization CPU usage
Data Quality
Diff Reports - inspect data changes between releases
Data Quality
Diff History - inspect changes in data over time
Zeno Framework
Data Schema (Serializers)
Operation (SerializationFramework)
Input Data (POJOs)
Output
Zeno Framework
Data Schema (Serializers)
Operation (SerializationFramework)
Input Data (POJOs)
Output
JsonSerializationFramework
HashSerializationFramework
DiffSerializationFramework
FastBlobStateEngine
Zeno Benefits
Development Agility:
● Easy to evolve data model, no need to change serialization formats or
operation logic
● Easy to create new functionality, no need to think about data model
structure or semantics
● Included “Diff” tools support high data quality across releases without too
much effort
Resource Efficiency:
● Included “FastBlob” optimized for Netflix scale
● Ask about in-development functionality!
Suro
To Be Processed in Different Ways
A Simple Solution That Supports All These
STAASH
STorage As A Service over Http
STAASH
STAASH
● Storage-Agnostic
● Language-Agnostic
● REST Interface to data
● Pattern Automation / Aware End Points
● Wrapper Around Astyanax Recipes
● Possibilities: Auditing, Cascading CL, Replication
across multiple storages, MapReduce …...many more..
Dynomite!!
Dynomite
● Cross AZ & Region replication to existing Key Value
stores
○ memcached
○ Reddis
● Thin Dynamo implementation provides the
replication
● Keep existing native KV protocol
○ No code refactoring
Dynomite
Dynomite
memcached
Dynomite
memcached
App 1
AZ 1 AZ 2
What do all those events mean?
{“deviceid”: 12345, “action”: “played”, “titleid”: 99999}
{“deviceid”: 12345, “action”: “played”, “titleid”: 99999}
Device C*
12345: “PS3”
{“deviceid”: 12345, “action”: “played”, “titleid”: 99999}
Device C*
12345: “PS3”
Content C*
99999: “HOC”
Don’t hurt production/our customers
Device/Content C*
“My Devices”: {“PS3:HOC”:”12345:99999”}
?!?!?
Sometimes you just want all the data
C*
Priam S3
SSTables
S3
SSTables
Move to HDFS*
Convert to JSON
Compact Rows
S3
JSON
● A splittable input format for SSTables
○ Need less files from the cluster.
○ Faster - just deserializing/serializing the files.
● An input format for the JSON
○ Allow incremental processing of backups
● A reducer that can compact SSTables.
Big Data Platform
Eventual Consistency
Focus on Performance
● Get your job running faster
● Understand why it was slow
● Transition to Hadoop 2
Netflix oss   season 2 episode 1 - meetup Lightning talks

More Related Content

What's hot

Netflix and Containers: Not A Stranger Thing
Netflix and Containers:  Not A Stranger ThingNetflix and Containers:  Not A Stranger Thing
Netflix and Containers: Not A Stranger Thingaspyker
 
CS80A Foothill College Open Source Talk
CS80A Foothill College Open Source TalkCS80A Foothill College Open Source Talk
CS80A Foothill College Open Source Talkaspyker
 
NetflixOSS and ZeroToDocker Talk
NetflixOSS and ZeroToDocker TalkNetflixOSS and ZeroToDocker Talk
NetflixOSS and ZeroToDocker Talkaspyker
 
Netflix Cloud Architecture and Open Source
Netflix Cloud Architecture and Open SourceNetflix Cloud Architecture and Open Source
Netflix Cloud Architecture and Open Sourceaspyker
 
Netflix Container Scheduling and Execution - QCon New York 2016
Netflix Container Scheduling and Execution - QCon New York 2016Netflix Container Scheduling and Execution - QCon New York 2016
Netflix Container Scheduling and Execution - QCon New York 2016aspyker
 
NetflixOSS Meetup S6E1 - Titus & Containers
NetflixOSS Meetup S6E1 - Titus & ContainersNetflixOSS Meetup S6E1 - Titus & Containers
NetflixOSS Meetup S6E1 - Titus & Containersaspyker
 
Herding Kats - Netflix’s Journey to Kubernetes Public
Herding Kats - Netflix’s Journey to Kubernetes PublicHerding Kats - Netflix’s Journey to Kubernetes Public
Herding Kats - Netflix’s Journey to Kubernetes Publicaspyker
 
NetflixOSS Meetup S6E2 - Spinnaker, Kayenta
NetflixOSS Meetup S6E2 - Spinnaker, KayentaNetflixOSS Meetup S6E2 - Spinnaker, Kayenta
NetflixOSS Meetup S6E2 - Spinnaker, Kayentaaspyker
 
20170831 - Greg Palmier: Terraform & AWS at Tempus
20170831 - Greg Palmier: Terraform & AWS at Tempus20170831 - Greg Palmier: Terraform & AWS at Tempus
20170831 - Greg Palmier: Terraform & AWS at TempusDevOps Chicago
 
An approach for migrating enterprise apps into open stack
An approach for migrating enterprise apps into open stackAn approach for migrating enterprise apps into open stack
An approach for migrating enterprise apps into open stackArthur Berezin
 
Netflix: From Zero to Production-Ready in Minutes (QCon 2017)
Netflix: From Zero to Production-Ready in Minutes (QCon 2017)Netflix: From Zero to Production-Ready in Minutes (QCon 2017)
Netflix: From Zero to Production-Ready in Minutes (QCon 2017)Tim Bozarth
 
Container World 2018
Container World 2018Container World 2018
Container World 2018aspyker
 
KURMA - A Containerized Container Platform - KubeCon 2016
KURMA - A Containerized Container Platform - KubeCon 2016KURMA - A Containerized Container Platform - KubeCon 2016
KURMA - A Containerized Container Platform - KubeCon 2016Apcera
 
FOSDEM 2021 - Infrastructure as Code Drift & Driftctl
FOSDEM 2021 - Infrastructure as Code Drift & DriftctlFOSDEM 2021 - Infrastructure as Code Drift & Driftctl
FOSDEM 2021 - Infrastructure as Code Drift & DriftctlStephane Jourdan
 
QCon NYC: Distributed systems in practice, in theory
QCon NYC: Distributed systems in practice, in theoryQCon NYC: Distributed systems in practice, in theory
QCon NYC: Distributed systems in practice, in theoryAysylu Greenberg
 
JavaZone 2017 - Microservices in action at the Dutch National Police
JavaZone 2017 - Microservices in action at the Dutch National PoliceJavaZone 2017 - Microservices in action at the Dutch National Police
JavaZone 2017 - Microservices in action at the Dutch National PoliceBert Jan Schrijver
 
OpenValue meetup October 2017 - Microservices in action at the Dutch National...
OpenValue meetup October 2017 - Microservices in action at the Dutch National...OpenValue meetup October 2017 - Microservices in action at the Dutch National...
OpenValue meetup October 2017 - Microservices in action at the Dutch National...Bert Jan Schrijver
 
Microservices in action at the Dutch National Police
Microservices in action at the Dutch National PoliceMicroservices in action at the Dutch National Police
Microservices in action at the Dutch National PoliceBert Jan Schrijver
 
Scaling tokopedia-past-present-future
Scaling tokopedia-past-present-futureScaling tokopedia-past-present-future
Scaling tokopedia-past-present-futureRein Mahatma
 
Virtualization at Gilt - Rangarajan Radhakrishnan
Virtualization at Gilt - Rangarajan RadhakrishnanVirtualization at Gilt - Rangarajan Radhakrishnan
Virtualization at Gilt - Rangarajan RadhakrishnanDatadog
 

What's hot (20)

Netflix and Containers: Not A Stranger Thing
Netflix and Containers:  Not A Stranger ThingNetflix and Containers:  Not A Stranger Thing
Netflix and Containers: Not A Stranger Thing
 
CS80A Foothill College Open Source Talk
CS80A Foothill College Open Source TalkCS80A Foothill College Open Source Talk
CS80A Foothill College Open Source Talk
 
NetflixOSS and ZeroToDocker Talk
NetflixOSS and ZeroToDocker TalkNetflixOSS and ZeroToDocker Talk
NetflixOSS and ZeroToDocker Talk
 
Netflix Cloud Architecture and Open Source
Netflix Cloud Architecture and Open SourceNetflix Cloud Architecture and Open Source
Netflix Cloud Architecture and Open Source
 
Netflix Container Scheduling and Execution - QCon New York 2016
Netflix Container Scheduling and Execution - QCon New York 2016Netflix Container Scheduling and Execution - QCon New York 2016
Netflix Container Scheduling and Execution - QCon New York 2016
 
NetflixOSS Meetup S6E1 - Titus & Containers
NetflixOSS Meetup S6E1 - Titus & ContainersNetflixOSS Meetup S6E1 - Titus & Containers
NetflixOSS Meetup S6E1 - Titus & Containers
 
Herding Kats - Netflix’s Journey to Kubernetes Public
Herding Kats - Netflix’s Journey to Kubernetes PublicHerding Kats - Netflix’s Journey to Kubernetes Public
Herding Kats - Netflix’s Journey to Kubernetes Public
 
NetflixOSS Meetup S6E2 - Spinnaker, Kayenta
NetflixOSS Meetup S6E2 - Spinnaker, KayentaNetflixOSS Meetup S6E2 - Spinnaker, Kayenta
NetflixOSS Meetup S6E2 - Spinnaker, Kayenta
 
20170831 - Greg Palmier: Terraform & AWS at Tempus
20170831 - Greg Palmier: Terraform & AWS at Tempus20170831 - Greg Palmier: Terraform & AWS at Tempus
20170831 - Greg Palmier: Terraform & AWS at Tempus
 
An approach for migrating enterprise apps into open stack
An approach for migrating enterprise apps into open stackAn approach for migrating enterprise apps into open stack
An approach for migrating enterprise apps into open stack
 
Netflix: From Zero to Production-Ready in Minutes (QCon 2017)
Netflix: From Zero to Production-Ready in Minutes (QCon 2017)Netflix: From Zero to Production-Ready in Minutes (QCon 2017)
Netflix: From Zero to Production-Ready in Minutes (QCon 2017)
 
Container World 2018
Container World 2018Container World 2018
Container World 2018
 
KURMA - A Containerized Container Platform - KubeCon 2016
KURMA - A Containerized Container Platform - KubeCon 2016KURMA - A Containerized Container Platform - KubeCon 2016
KURMA - A Containerized Container Platform - KubeCon 2016
 
FOSDEM 2021 - Infrastructure as Code Drift & Driftctl
FOSDEM 2021 - Infrastructure as Code Drift & DriftctlFOSDEM 2021 - Infrastructure as Code Drift & Driftctl
FOSDEM 2021 - Infrastructure as Code Drift & Driftctl
 
QCon NYC: Distributed systems in practice, in theory
QCon NYC: Distributed systems in practice, in theoryQCon NYC: Distributed systems in practice, in theory
QCon NYC: Distributed systems in practice, in theory
 
JavaZone 2017 - Microservices in action at the Dutch National Police
JavaZone 2017 - Microservices in action at the Dutch National PoliceJavaZone 2017 - Microservices in action at the Dutch National Police
JavaZone 2017 - Microservices in action at the Dutch National Police
 
OpenValue meetup October 2017 - Microservices in action at the Dutch National...
OpenValue meetup October 2017 - Microservices in action at the Dutch National...OpenValue meetup October 2017 - Microservices in action at the Dutch National...
OpenValue meetup October 2017 - Microservices in action at the Dutch National...
 
Microservices in action at the Dutch National Police
Microservices in action at the Dutch National PoliceMicroservices in action at the Dutch National Police
Microservices in action at the Dutch National Police
 
Scaling tokopedia-past-present-future
Scaling tokopedia-past-present-futureScaling tokopedia-past-present-future
Scaling tokopedia-past-present-future
 
Virtualization at Gilt - Rangarajan Radhakrishnan
Virtualization at Gilt - Rangarajan RadhakrishnanVirtualization at Gilt - Rangarajan Radhakrishnan
Virtualization at Gilt - Rangarajan Radhakrishnan
 

Similar to Netflix oss season 2 episode 1 - meetup Lightning talks

Sanger, upcoming Openstack for Bio-informaticians
Sanger, upcoming Openstack for Bio-informaticiansSanger, upcoming Openstack for Bio-informaticians
Sanger, upcoming Openstack for Bio-informaticiansPeter Clapham
 
DevOpsCon 2015 - DevOps in Mobile Games
DevOpsCon 2015 - DevOps in Mobile GamesDevOpsCon 2015 - DevOps in Mobile Games
DevOpsCon 2015 - DevOps in Mobile GamesAndreas Katzig
 
Storage Requirements and Options for Running Spark on Kubernetes
Storage Requirements and Options for Running Spark on KubernetesStorage Requirements and Options for Running Spark on Kubernetes
Storage Requirements and Options for Running Spark on KubernetesDataWorks Summit
 
Next Generation Cloud Computing With Google - RightScale Compute 2013
Next Generation Cloud Computing With Google - RightScale Compute 2013Next Generation Cloud Computing With Google - RightScale Compute 2013
Next Generation Cloud Computing With Google - RightScale Compute 2013RightScale
 
Getting Started with Serverless Architectures
Getting Started with Serverless ArchitecturesGetting Started with Serverless Architectures
Getting Started with Serverless ArchitecturesAmazon Web Services
 
.Net Microservices with Event Sourcing, CQRS, Docker and... Windows Server 20...
.Net Microservices with Event Sourcing, CQRS, Docker and... Windows Server 20....Net Microservices with Event Sourcing, CQRS, Docker and... Windows Server 20...
.Net Microservices with Event Sourcing, CQRS, Docker and... Windows Server 20...Javier García Magna
 
Spark volume requirements 2018
Spark volume requirements 2018Spark volume requirements 2018
Spark volume requirements 2018Rachit Arora
 
Yow Conference Dec 2013 Netflix Workshop Slides with Notes
Yow Conference Dec 2013 Netflix Workshop Slides with NotesYow Conference Dec 2013 Netflix Workshop Slides with Notes
Yow Conference Dec 2013 Netflix Workshop Slides with NotesAdrian Cockcroft
 
Rootconf 2017 - State of the Open Source monitoring landscape
Rootconf 2017 - State of the Open Source monitoring landscape Rootconf 2017 - State of the Open Source monitoring landscape
Rootconf 2017 - State of the Open Source monitoring landscape NETWAYS
 
Developing in the Cloud
Developing in the CloudDeveloping in the Cloud
Developing in the CloudRyan Cuprak
 
Hpc lunch and learn
Hpc lunch and learnHpc lunch and learn
Hpc lunch and learnJohn D Almon
 
Cloud Native Camel Riding
Cloud Native Camel RidingCloud Native Camel Riding
Cloud Native Camel RidingChristian Posta
 
DevOpsGuys - DevOps Automation - The Good, The Bad and The Ugly
DevOpsGuys - DevOps Automation - The Good, The Bad and The UglyDevOpsGuys - DevOps Automation - The Good, The Bad and The Ugly
DevOpsGuys - DevOps Automation - The Good, The Bad and The UglyDevOpsGroup
 
Automation: The Good, The Bad and The Ugly with DevOpsGuys - AppD Summit Europe
Automation: The Good, The Bad and The Ugly with DevOpsGuys - AppD Summit EuropeAutomation: The Good, The Bad and The Ugly with DevOpsGuys - AppD Summit Europe
Automation: The Good, The Bad and The Ugly with DevOpsGuys - AppD Summit EuropeAppDynamics
 
Building a Distributed & Automated Open Source Program at Netflix
Building a Distributed & Automated Open Source Program at NetflixBuilding a Distributed & Automated Open Source Program at Netflix
Building a Distributed & Automated Open Source Program at NetflixAll Things Open
 
Netflix Open Source: Building a Distributed and Automated Open Source Program
Netflix Open Source:  Building a Distributed and Automated Open Source ProgramNetflix Open Source:  Building a Distributed and Automated Open Source Program
Netflix Open Source: Building a Distributed and Automated Open Source Programaspyker
 

Similar to Netflix oss season 2 episode 1 - meetup Lightning talks (20)

Sanger, upcoming Openstack for Bio-informaticians
Sanger, upcoming Openstack for Bio-informaticiansSanger, upcoming Openstack for Bio-informaticians
Sanger, upcoming Openstack for Bio-informaticians
 
Flexible compute
Flexible computeFlexible compute
Flexible compute
 
DevOpsCon 2015 - DevOps in Mobile Games
DevOpsCon 2015 - DevOps in Mobile GamesDevOpsCon 2015 - DevOps in Mobile Games
DevOpsCon 2015 - DevOps in Mobile Games
 
Stackato v2
Stackato v2Stackato v2
Stackato v2
 
Storage Requirements and Options for Running Spark on Kubernetes
Storage Requirements and Options for Running Spark on KubernetesStorage Requirements and Options for Running Spark on Kubernetes
Storage Requirements and Options for Running Spark on Kubernetes
 
Next Generation Cloud Computing With Google - RightScale Compute 2013
Next Generation Cloud Computing With Google - RightScale Compute 2013Next Generation Cloud Computing With Google - RightScale Compute 2013
Next Generation Cloud Computing With Google - RightScale Compute 2013
 
Getting Started with Serverless Architectures
Getting Started with Serverless ArchitecturesGetting Started with Serverless Architectures
Getting Started with Serverless Architectures
 
.Net Microservices with Event Sourcing, CQRS, Docker and... Windows Server 20...
.Net Microservices with Event Sourcing, CQRS, Docker and... Windows Server 20....Net Microservices with Event Sourcing, CQRS, Docker and... Windows Server 20...
.Net Microservices with Event Sourcing, CQRS, Docker and... Windows Server 20...
 
Stackato v5
Stackato v5Stackato v5
Stackato v5
 
Spark volume requirements 2018
Spark volume requirements 2018Spark volume requirements 2018
Spark volume requirements 2018
 
Yow Conference Dec 2013 Netflix Workshop Slides with Notes
Yow Conference Dec 2013 Netflix Workshop Slides with NotesYow Conference Dec 2013 Netflix Workshop Slides with Notes
Yow Conference Dec 2013 Netflix Workshop Slides with Notes
 
Rootconf 2017 - State of the Open Source monitoring landscape
Rootconf 2017 - State of the Open Source monitoring landscape Rootconf 2017 - State of the Open Source monitoring landscape
Rootconf 2017 - State of the Open Source monitoring landscape
 
Developing in the Cloud
Developing in the CloudDeveloping in the Cloud
Developing in the Cloud
 
Hpc lunch and learn
Hpc lunch and learnHpc lunch and learn
Hpc lunch and learn
 
Cloud Native Camel Riding
Cloud Native Camel RidingCloud Native Camel Riding
Cloud Native Camel Riding
 
Head in the clouds
Head in the cloudsHead in the clouds
Head in the clouds
 
DevOpsGuys - DevOps Automation - The Good, The Bad and The Ugly
DevOpsGuys - DevOps Automation - The Good, The Bad and The UglyDevOpsGuys - DevOps Automation - The Good, The Bad and The Ugly
DevOpsGuys - DevOps Automation - The Good, The Bad and The Ugly
 
Automation: The Good, The Bad and The Ugly with DevOpsGuys - AppD Summit Europe
Automation: The Good, The Bad and The Ugly with DevOpsGuys - AppD Summit EuropeAutomation: The Good, The Bad and The Ugly with DevOpsGuys - AppD Summit Europe
Automation: The Good, The Bad and The Ugly with DevOpsGuys - AppD Summit Europe
 
Building a Distributed & Automated Open Source Program at Netflix
Building a Distributed & Automated Open Source Program at NetflixBuilding a Distributed & Automated Open Source Program at Netflix
Building a Distributed & Automated Open Source Program at Netflix
 
Netflix Open Source: Building a Distributed and Automated Open Source Program
Netflix Open Source:  Building a Distributed and Automated Open Source ProgramNetflix Open Source:  Building a Distributed and Automated Open Source Program
Netflix Open Source: Building a Distributed and Automated Open Source Program
 

More from Ruslan Meshenberg

NetflixOSS Meetup season 3 episode 2
NetflixOSS Meetup season 3 episode 2NetflixOSS Meetup season 3 episode 2
NetflixOSS Meetup season 3 episode 2Ruslan Meshenberg
 
Netflix oss past-present-future
Netflix oss   past-present-futureNetflix oss   past-present-future
Netflix oss past-present-futureRuslan Meshenberg
 
Dev309 from asgard to zuul - netflix oss-final
Dev309  from asgard to zuul - netflix oss-finalDev309  from asgard to zuul - netflix oss-final
Dev309 from asgard to zuul - netflix oss-finalRuslan Meshenberg
 
NetflixOSS season 2 episode 2 - Reactive / Async
NetflixOSS   season 2 episode 2 - Reactive / AsyncNetflixOSS   season 2 episode 2 - Reactive / Async
NetflixOSS season 2 episode 2 - Reactive / AsyncRuslan Meshenberg
 
OSS Think Tank - NetflixOSS - OSS as a Competitive Differentiator
OSS Think Tank - NetflixOSS - OSS as a Competitive DifferentiatorOSS Think Tank - NetflixOSS - OSS as a Competitive Differentiator
OSS Think Tank - NetflixOSS - OSS as a Competitive DifferentiatorRuslan Meshenberg
 
Arc305 how netflix leverages multiple regions to increase availability an i...
Arc305 how netflix leverages multiple regions to increase availability   an i...Arc305 how netflix leverages multiple regions to increase availability   an i...
Arc305 how netflix leverages multiple regions to increase availability an i...Ruslan Meshenberg
 
NetflixOSS meetup lightning talks and roadmap
NetflixOSS meetup lightning talks and roadmapNetflixOSS meetup lightning talks and roadmap
NetflixOSS meetup lightning talks and roadmapRuslan Meshenberg
 
NetflixOSS Open House Lightning talks
NetflixOSS Open House Lightning talksNetflixOSS Open House Lightning talks
NetflixOSS Open House Lightning talksRuslan Meshenberg
 
re:Invent 2012 Optimizing Cassandra
re:Invent 2012 Optimizing Cassandrare:Invent 2012 Optimizing Cassandra
re:Invent 2012 Optimizing CassandraRuslan Meshenberg
 
The Netflix Open Source Platform
The Netflix Open Source PlatformThe Netflix Open Source Platform
The Netflix Open Source PlatformRuslan Meshenberg
 

More from Ruslan Meshenberg (11)

NetflixOSS Meetup season 3 episode 2
NetflixOSS Meetup season 3 episode 2NetflixOSS Meetup season 3 episode 2
NetflixOSS Meetup season 3 episode 2
 
Netflix oss past-present-future
Netflix oss   past-present-futureNetflix oss   past-present-future
Netflix oss past-present-future
 
Dev309 from asgard to zuul - netflix oss-final
Dev309  from asgard to zuul - netflix oss-finalDev309  from asgard to zuul - netflix oss-final
Dev309 from asgard to zuul - netflix oss-final
 
NetflixOSS season 2 episode 2 - Reactive / Async
NetflixOSS   season 2 episode 2 - Reactive / AsyncNetflixOSS   season 2 episode 2 - Reactive / Async
NetflixOSS season 2 episode 2 - Reactive / Async
 
OSS Think Tank - NetflixOSS - OSS as a Competitive Differentiator
OSS Think Tank - NetflixOSS - OSS as a Competitive DifferentiatorOSS Think Tank - NetflixOSS - OSS as a Competitive Differentiator
OSS Think Tank - NetflixOSS - OSS as a Competitive Differentiator
 
Arc305 how netflix leverages multiple regions to increase availability an i...
Arc305 how netflix leverages multiple regions to increase availability   an i...Arc305 how netflix leverages multiple regions to increase availability   an i...
Arc305 how netflix leverages multiple regions to increase availability an i...
 
Svc 202-netflix-open-source
Svc 202-netflix-open-sourceSvc 202-netflix-open-source
Svc 202-netflix-open-source
 
NetflixOSS meetup lightning talks and roadmap
NetflixOSS meetup lightning talks and roadmapNetflixOSS meetup lightning talks and roadmap
NetflixOSS meetup lightning talks and roadmap
 
NetflixOSS Open House Lightning talks
NetflixOSS Open House Lightning talksNetflixOSS Open House Lightning talks
NetflixOSS Open House Lightning talks
 
re:Invent 2012 Optimizing Cassandra
re:Invent 2012 Optimizing Cassandrare:Invent 2012 Optimizing Cassandra
re:Invent 2012 Optimizing Cassandra
 
The Netflix Open Source Platform
The Netflix Open Source PlatformThe Netflix Open Source Platform
The Netflix Open Source Platform
 

Recently uploaded

DevoxxFR 2024 Reproducible Builds with Apache Maven
DevoxxFR 2024 Reproducible Builds with Apache MavenDevoxxFR 2024 Reproducible Builds with Apache Maven
DevoxxFR 2024 Reproducible Builds with Apache MavenHervé Boutemy
 
The State of Passkeys with FIDO Alliance.pptx
The State of Passkeys with FIDO Alliance.pptxThe State of Passkeys with FIDO Alliance.pptx
The State of Passkeys with FIDO Alliance.pptxLoriGlavin3
 
Ensuring Technical Readiness For Copilot in Microsoft 365
Ensuring Technical Readiness For Copilot in Microsoft 365Ensuring Technical Readiness For Copilot in Microsoft 365
Ensuring Technical Readiness For Copilot in Microsoft 3652toLead Limited
 
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptxThe Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptxLoriGlavin3
 
Ryan Mahoney - Will Artificial Intelligence Replace Real Estate Agents
Ryan Mahoney - Will Artificial Intelligence Replace Real Estate AgentsRyan Mahoney - Will Artificial Intelligence Replace Real Estate Agents
Ryan Mahoney - Will Artificial Intelligence Replace Real Estate AgentsRyan Mahoney
 
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024BookNet Canada
 
Visualising and forecasting stocks using Dash
Visualising and forecasting stocks using DashVisualising and forecasting stocks using Dash
Visualising and forecasting stocks using Dashnarutouzumaki53779
 
Take control of your SAP testing with UiPath Test Suite
Take control of your SAP testing with UiPath Test SuiteTake control of your SAP testing with UiPath Test Suite
Take control of your SAP testing with UiPath Test SuiteDianaGray10
 
Rise of the Machines: Known As Drones...
Rise of the Machines: Known As Drones...Rise of the Machines: Known As Drones...
Rise of the Machines: Known As Drones...Rick Flair
 
From Family Reminiscence to Scholarly Archive .
From Family Reminiscence to Scholarly Archive .From Family Reminiscence to Scholarly Archive .
From Family Reminiscence to Scholarly Archive .Alan Dix
 
DSPy a system for AI to Write Prompts and Do Fine Tuning
DSPy a system for AI to Write Prompts and Do Fine TuningDSPy a system for AI to Write Prompts and Do Fine Tuning
DSPy a system for AI to Write Prompts and Do Fine TuningLars Bell
 
What's New in Teams Calling, Meetings and Devices March 2024
What's New in Teams Calling, Meetings and Devices March 2024What's New in Teams Calling, Meetings and Devices March 2024
What's New in Teams Calling, Meetings and Devices March 2024Stephanie Beckett
 
TeamStation AI System Report LATAM IT Salaries 2024
TeamStation AI System Report LATAM IT Salaries 2024TeamStation AI System Report LATAM IT Salaries 2024
TeamStation AI System Report LATAM IT Salaries 2024Lonnie McRorey
 
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek SchlawackFwdays
 
Scale your database traffic with Read & Write split using MySQL Router
Scale your database traffic with Read & Write split using MySQL RouterScale your database traffic with Read & Write split using MySQL Router
Scale your database traffic with Read & Write split using MySQL RouterMydbops
 
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024BookNet Canada
 
The Ultimate Guide to Choosing WordPress Pros and Cons
The Ultimate Guide to Choosing WordPress Pros and ConsThe Ultimate Guide to Choosing WordPress Pros and Cons
The Ultimate Guide to Choosing WordPress Pros and ConsPixlogix Infotech
 
SALESFORCE EDUCATION CLOUD | FEXLE SERVICES
SALESFORCE EDUCATION CLOUD | FEXLE SERVICESSALESFORCE EDUCATION CLOUD | FEXLE SERVICES
SALESFORCE EDUCATION CLOUD | FEXLE SERVICESmohitsingh558521
 
Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!Commit University
 
Unraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdfUnraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdfAlex Barbosa Coqueiro
 

Recently uploaded (20)

DevoxxFR 2024 Reproducible Builds with Apache Maven
DevoxxFR 2024 Reproducible Builds with Apache MavenDevoxxFR 2024 Reproducible Builds with Apache Maven
DevoxxFR 2024 Reproducible Builds with Apache Maven
 
The State of Passkeys with FIDO Alliance.pptx
The State of Passkeys with FIDO Alliance.pptxThe State of Passkeys with FIDO Alliance.pptx
The State of Passkeys with FIDO Alliance.pptx
 
Ensuring Technical Readiness For Copilot in Microsoft 365
Ensuring Technical Readiness For Copilot in Microsoft 365Ensuring Technical Readiness For Copilot in Microsoft 365
Ensuring Technical Readiness For Copilot in Microsoft 365
 
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptxThe Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
 
Ryan Mahoney - Will Artificial Intelligence Replace Real Estate Agents
Ryan Mahoney - Will Artificial Intelligence Replace Real Estate AgentsRyan Mahoney - Will Artificial Intelligence Replace Real Estate Agents
Ryan Mahoney - Will Artificial Intelligence Replace Real Estate Agents
 
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
 
Visualising and forecasting stocks using Dash
Visualising and forecasting stocks using DashVisualising and forecasting stocks using Dash
Visualising and forecasting stocks using Dash
 
Take control of your SAP testing with UiPath Test Suite
Take control of your SAP testing with UiPath Test SuiteTake control of your SAP testing with UiPath Test Suite
Take control of your SAP testing with UiPath Test Suite
 
Rise of the Machines: Known As Drones...
Rise of the Machines: Known As Drones...Rise of the Machines: Known As Drones...
Rise of the Machines: Known As Drones...
 
From Family Reminiscence to Scholarly Archive .
From Family Reminiscence to Scholarly Archive .From Family Reminiscence to Scholarly Archive .
From Family Reminiscence to Scholarly Archive .
 
DSPy a system for AI to Write Prompts and Do Fine Tuning
DSPy a system for AI to Write Prompts and Do Fine TuningDSPy a system for AI to Write Prompts and Do Fine Tuning
DSPy a system for AI to Write Prompts and Do Fine Tuning
 
What's New in Teams Calling, Meetings and Devices March 2024
What's New in Teams Calling, Meetings and Devices March 2024What's New in Teams Calling, Meetings and Devices March 2024
What's New in Teams Calling, Meetings and Devices March 2024
 
TeamStation AI System Report LATAM IT Salaries 2024
TeamStation AI System Report LATAM IT Salaries 2024TeamStation AI System Report LATAM IT Salaries 2024
TeamStation AI System Report LATAM IT Salaries 2024
 
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
"Subclassing and Composition – A Pythonic Tour of Trade-Offs", Hynek Schlawack
 
Scale your database traffic with Read & Write split using MySQL Router
Scale your database traffic with Read & Write split using MySQL RouterScale your database traffic with Read & Write split using MySQL Router
Scale your database traffic with Read & Write split using MySQL Router
 
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
 
The Ultimate Guide to Choosing WordPress Pros and Cons
The Ultimate Guide to Choosing WordPress Pros and ConsThe Ultimate Guide to Choosing WordPress Pros and Cons
The Ultimate Guide to Choosing WordPress Pros and Cons
 
SALESFORCE EDUCATION CLOUD | FEXLE SERVICES
SALESFORCE EDUCATION CLOUD | FEXLE SERVICESSALESFORCE EDUCATION CLOUD | FEXLE SERVICES
SALESFORCE EDUCATION CLOUD | FEXLE SERVICES
 
Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!Nell’iperspazio con Rocket: il Framework Web di Rust!
Nell’iperspazio con Rocket: il Framework Web di Rust!
 
Unraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdfUnraveling Multimodality with Large Language Models.pdf
Unraveling Multimodality with Large Language Models.pdf
 

Netflix oss season 2 episode 1 - meetup Lightning talks

  • 1. Season 2 Episode 1 March 12, 2014
  • 2. Evening Outline Lightning Talks: - S3mper - PigPen - STAASH - Dynomite - Aegisthus - Suro - Zeno - Lipstick on GCE - AnsWerS - IBM - Coursera
  • 3. 41 projects… Now what? ● Cohesive platform ● Workshops / Training / Documentation ● Participate and contribute : netflixoss@netflix.com
  • 5. Lipstick, Hadoop, and Big Data on the Google Cloud Matt Bookman Solutions Architect
  • 6. Google Confidential and Proprietary Google Compute Engine - VMs in Google Datacenters ● Public Preview - May 2013 ● General Availability - December 2013
  • 7. Google Confidential and Proprietary Demo (Summer 2013): Pig on Compute Engine Sweet demo!
  • 8. Google Confidential and Proprietary Netflix OSS Meetup - July 17, 2013
  • 9. Google Confidential and Proprietary Lipstick - Providing insights
  • 10. Google Confidential and Proprietary
  • 11. Google Confidential and Proprietary Hadoop on GCE + Cloud Storage (GCS) Connector Accenture: Cloud vs. Bare-Metal ● Cloud-based Hadoop deployments offer better price- performance ratios than bare-metal ● Cloud’s virtualization expands performance- tuning opportunities ● Using remote storage outperforms local disk HDFS
  • 12. Google Confidential and Proprietary Data in GCS, Lipstick DB in Cloud SQL Google Cloud Platform Output Data Lipstick Database Hadoop Master MapReduce JobTracker Hadoop Worker MapReduce TaskTrackerHadoop Worker MapReduce TaskTracker Hadoop Worker MapReduce TaskTrackerLipstick Server Input Data
  • 13. Google Confidential and Proprietary ● Netflix Lipstick on Google Compute Engine https://cloud.google.com/developers/articles/netflix-lipstick-on-google-compute-engine ● GCS Connector for Hadoop https://developers.google.com/hadoop/google-cloud-storage-connector ● Cloud-based Hadoop Deployments: Benefits and Considerations http://www.accenture.com/SiteCollectionDocuments/PDF/Accenture-Cloud-Based-Hadoop-Deployments-Benefits-and-Considerations. pdf ● Apache Hadoop, Hive, and Pig on Google Compute Engine https://cloud.google.com/developers/articles/apache-hadoop-hive-and-pig-on-google-compute-engine Resources
  • 14. Google Confidential and Proprietary Thank you
  • 15. @Answers4AWS Cloud Prize and Beyond Peter Sankauskas @pas256
  • 20. @Answers4AWS Requirements • AsgardFormation • Asgard running • AWS Credentials • IAM user • Policy • Security Group • EC2 instance • Asgard downloaded and configured • Tomcat downloaded and configured • Java downloaded and installed • Linux configured
  • 22. @Answers4AWS Asgard playbook • Base • Install usual Linux packages • Basic system hardening and security packages • Oracle Java 7 • Tomcat 7 • Asgard • Latest release from GitHub
  • 23. @Answers4AWS Other playbooks • Eureka • Edda • Simian Army • Ice • Aminator • Genie
  • 24. @Answers4AWS AMIs • Initially built using my own scripts based on Eric Hammond’s (@esh) work • Then using Aminator • Created Ubuntu Foundation AMIs • Added the Ansible Provisioner for Aminator • Put a couple of them on the AWS Marketplace for free
  • 25. @Answers4AWS CloudFormation • One-click deploy • Well, about 10 going through the AWS Web Console wizard • Designed to get you up and running quickly • Test it out, see if you like it • NOT production quality • No real security • No HA
  • 28. @Answers4AWS Do you do this? (this is not my slide)
  • 30. @Answers4AWS Beta users • From a successful CI build • To a Fully Baked AMI • Use in Testing and Production • Without you doing anything • ZERO clicks • Signups are open
  • 32. IBM Scalable Services Fabric for Netflix S2E1 Meetup Andrew Spyker @aspyker
  • 33. History and Future 2012 SPECjEnterprise 2013 AcmeAir Run On IBM Cloud at “Web Scale” 2014 Scalable Services Fabric internally for IBM Services Scalable Services Fabric SaaS and On-Prem? Sample application cloud prize work AcmeAir Cloud/Mobile Sample/Benchmark born Codename: BlueMix Portability cloud prize work
  • 34. Scalable Service Fabric Work Netflix OSS IBM port/enablement Netflix “Zen” of Cloud • Worked with initial services to enable cloud native arch • Worked with initial services to enable NetflixOSS usages • Created scorecard and tests for “cloud native” readiness Highly Available IaaS and Cloud Services • Deployment across multiple IBM SoftLayer IaaS datacenters and global and local load balancers • Complete automation via IBM SoftLayer IaaS API’s • Ensured facilities for automatic failure recovery Micro-service Runtimes (Karyon, Eureka Client, Ribbon, Hystrix, Archaius) • Ported to work with IBM SoftLayer IaaS and on the WebSphere Liberty Profile application server • Created “eureka-sidecar” for non-Java runtimes and ElasticSearch discovery Netflix OSS Servers (Asgard, Eureka Server, Turbine) • Ported to work with IBM SoftLayer IaaS + RightScale • Operationalized HA and secure deployments for multiple service tenants Adopted Chaos Testing • Ported Chaos Monkey to IBM SoftLayer IaaS • Performed manual Chaos Gorilla validation on services Worked through devops tool chain • Worked with initial services to enable continuous delivery with devops (and imagine baking via Animator like tool)
  • 35. Come meet the team! Looks like … Tweets from … Talks about … Adolfo @adolforod API Management and Cloud Integration, user of NetflixOSS platform. Appliances in the cloud. Brian @bkmartin IBM BlueMix (PaaS), enabling composable apps in PaaS Darrell IBM Research focusing on NetflixOSS devops and on- premise deployments David @dcurrie WebSphere Liberty Profile application server NetflixOSS development and PaaS integration Jonathan @ma4jpb NetflixOSS portability across many aspects Cloud messaging (in relation to Suro) Matt @matrober API Management, user of NetflixOSS platform Converted service to be cloud native Rachel @rreinitz IBM Services, interested in helping you get to this cloud native in SaaS and on-premise Ricky @rickymoorhouse API Management, user of NetflixOSS platform Creator of Imaginator Will @auwilli98 API Management operations, user of NetflixOSS platform
  • 38. Overview • Philosophy • Priam • Aegisthus • Conclusion
  • 39. Philosophy • Architecture Patterns • Use what we can • Incorporate the spirit of others
  • 40. Priam – Wins • Token Management • S3 Backup + Restore • Config
  • 41. Priam – Next Steps? • SimpleDB -> DynamoDB • Backups blow out OS disk buffer cache • Compatibility with newer C* versions
  • 42. Aegisthus - Wins • Novel workflow • Data reduced to one authoritative copy • Possibility for incremental jobs
  • 43. Aegisthus – What Next? • C* 1.2 / 2.0 • CQL3 • Priam <–> Aegisthus • Better compressed SSTable support
  • 44. Conclusion • Come chat with me! • Especially if you have similar goals to me
  • 45. Zeno ● In-memory data distribution platform ● Contains tools for: ○ data quality management ○ data serialization ● We use it to distribute and keep up to date gigabytes of video metadata on tens of thousands of servers across the globe
  • 46. Zeno Why in-memory data? - Netflix serves billions of requests per day - Each request requires metadata about many movies to answer
  • 47. Zeno Netflix Use Case: ● Gigabytes of in-memory data ● Hundreds of thousands of in-memory cache requests per second, per application instance ● Tens of thousands of application instances
  • 48. Distribution FastBlob: Binary serialization of a complete state of data, and/or the changes in data over time. Serialization format designed to propagate, and keep up to date, a large amount of in-memory data across many servers. Optimized for: memory GC effects, memory footprint, data transfer size, deserialization CPU usage
  • 49. Data Quality Diff Reports - inspect data changes between releases
  • 50. Data Quality Diff History - inspect changes in data over time
  • 51. Zeno Framework Data Schema (Serializers) Operation (SerializationFramework) Input Data (POJOs) Output
  • 52. Zeno Framework Data Schema (Serializers) Operation (SerializationFramework) Input Data (POJOs) Output JsonSerializationFramework HashSerializationFramework DiffSerializationFramework FastBlobStateEngine
  • 53. Zeno Benefits Development Agility: ● Easy to evolve data model, no need to change serialization formats or operation logic ● Easy to create new functionality, no need to think about data model structure or semantics ● Included “Diff” tools support high data quality across releases without too much effort Resource Efficiency: ● Included “FastBlob” optimized for Netflix scale ● Ask about in-development functionality!
  • 54. Suro
  • 55.
  • 56.
  • 57.
  • 58.
  • 59.
  • 60. To Be Processed in Different Ways
  • 61.
  • 62.
  • 63. A Simple Solution That Supports All These
  • 64.
  • 65. STAASH STorage As A Service over Http
  • 67. STAASH ● Storage-Agnostic ● Language-Agnostic ● REST Interface to data ● Pattern Automation / Aware End Points ● Wrapper Around Astyanax Recipes ● Possibilities: Auditing, Cascading CL, Replication across multiple storages, MapReduce …...many more..
  • 69. Dynomite ● Cross AZ & Region replication to existing Key Value stores ○ memcached ○ Reddis ● Thin Dynamo implementation provides the replication ● Keep existing native KV protocol ○ No code refactoring
  • 71.
  • 72. What do all those events mean?
  • 73. {“deviceid”: 12345, “action”: “played”, “titleid”: 99999}
  • 74. {“deviceid”: 12345, “action”: “played”, “titleid”: 99999} Device C* 12345: “PS3”
  • 75. {“deviceid”: 12345, “action”: “played”, “titleid”: 99999} Device C* 12345: “PS3” Content C* 99999: “HOC”
  • 77. Device/Content C* “My Devices”: {“PS3:HOC”:”12345:99999”} ?!?!?
  • 78. Sometimes you just want all the data
  • 80. S3 SSTables Move to HDFS* Convert to JSON Compact Rows S3 JSON
  • 81. ● A splittable input format for SSTables ○ Need less files from the cluster. ○ Faster - just deserializing/serializing the files. ● An input format for the JSON ○ Allow incremental processing of backups ● A reducer that can compact SSTables.
  • 82.
  • 83.
  • 84.
  • 85.
  • 86.
  • 87.
  • 88.
  • 89.
  • 90.
  • 91.
  • 92.
  • 93.
  • 94.
  • 95.
  • 96.
  • 97.
  • 98.
  • 99.
  • 100.
  • 103.
  • 104.
  • 105.
  • 106.
  • 107. Focus on Performance ● Get your job running faster ● Understand why it was slow ● Transition to Hadoop 2