This document provides an overview and agenda for a presentation on architecting for the cloud using the Netflix approach. Some key points:
- Netflix has over 40 million members streaming over 1 billion hours per month of content across over 40 countries.
- Netflix runs on AWS and handles billions of requests per day across thousands of instances globally.
- The presentation will discuss how to build your own platform as a service (PaaS) based on Netflix's open source libraries, including platform services, libraries, and tools.
- The Netflix approach focuses on microservices, automation, and resilience to support rapid iteration on cloud infrastructure.
This is a talk on how you can monitor your microservices architecture using Prometheus and Grafana. This has easy to execute steps to get a local monitoring stack running on your local machine using docker.
Installation of Grafana on linux ; connectivity with Prometheus database , installation of Prometheus ; Installation of node_exporter ,Tomcat-exporter ; installation and configuration of alert manager .. Detailed step by step installation and working
Kubernetes Concepts And Architecture Powerpoint Presentation SlidesSlideTeam
Get these visually appealing Kubernetes Concepts And Architecture PowerPoint Presentation Slides to discuss the process of operating containerized applications. You can display the need for containers by the company with the help of an open-source architecture PPT slideshow. The architecture of containers can be demonstrated with the help of a visually appealing PPT slideshow. The reasons for opting for Kubernetes by an organization can be explained to your teammates with the help of containers PowerPoint infographics. Highlight the roadmap for installing Kubernetes in the organization by using content-ready PPT slides. Take the assistance of visually appealing PPT templates to depict the major advantages of Kubernetes such as improving productivity, the stability of application run, and many more. After that, display 30 60 90 days plan to implement Kubernetes in the organization. Display the key components of Kubernetes with the help of a diagram using this professionally designed cluster architecture PPT layouts. Describe the functionality of each components of Kubernetes. Hence, download Kubernetes architecture PPT slides to easily and efficiently manage the clusters. https://bit.ly/34DWa7x
Watch this talk here: https://www.confluent.io/online-talks/apache-kafka-architecture-and-fundamentals-explained-on-demand
This session explains Apache Kafka’s internal design and architecture. Companies like LinkedIn are now sending more than 1 trillion messages per day to Apache Kafka. Learn about the underlying design in Kafka that leads to such high throughput.
This talk provides a comprehensive overview of Kafka architecture and internal functions, including:
-Topics, partitions and segments
-The commit log and streams
-Brokers and broker replication
-Producer basics
-Consumers, consumer groups and offsets
This session is part 2 of 4 in our Fundamentals for Apache Kafka series.
This is a talk on how you can monitor your microservices architecture using Prometheus and Grafana. This has easy to execute steps to get a local monitoring stack running on your local machine using docker.
Installation of Grafana on linux ; connectivity with Prometheus database , installation of Prometheus ; Installation of node_exporter ,Tomcat-exporter ; installation and configuration of alert manager .. Detailed step by step installation and working
Kubernetes Concepts And Architecture Powerpoint Presentation SlidesSlideTeam
Get these visually appealing Kubernetes Concepts And Architecture PowerPoint Presentation Slides to discuss the process of operating containerized applications. You can display the need for containers by the company with the help of an open-source architecture PPT slideshow. The architecture of containers can be demonstrated with the help of a visually appealing PPT slideshow. The reasons for opting for Kubernetes by an organization can be explained to your teammates with the help of containers PowerPoint infographics. Highlight the roadmap for installing Kubernetes in the organization by using content-ready PPT slides. Take the assistance of visually appealing PPT templates to depict the major advantages of Kubernetes such as improving productivity, the stability of application run, and many more. After that, display 30 60 90 days plan to implement Kubernetes in the organization. Display the key components of Kubernetes with the help of a diagram using this professionally designed cluster architecture PPT layouts. Describe the functionality of each components of Kubernetes. Hence, download Kubernetes architecture PPT slides to easily and efficiently manage the clusters. https://bit.ly/34DWa7x
Watch this talk here: https://www.confluent.io/online-talks/apache-kafka-architecture-and-fundamentals-explained-on-demand
This session explains Apache Kafka’s internal design and architecture. Companies like LinkedIn are now sending more than 1 trillion messages per day to Apache Kafka. Learn about the underlying design in Kafka that leads to such high throughput.
This talk provides a comprehensive overview of Kafka architecture and internal functions, including:
-Topics, partitions and segments
-The commit log and streams
-Brokers and broker replication
-Producer basics
-Consumers, consumer groups and offsets
This session is part 2 of 4 in our Fundamentals for Apache Kafka series.
How Netflix Tunes EC2 Instances for PerformanceBrendan Gregg
CMP325 talk for AWS re:Invent 2017, by Brendan Gregg. "
At Netflix we make the best use of AWS EC2 instance types and features to create a high performance cloud, achieving near bare metal speed for our workloads. This session will summarize the configuration, tuning, and activities for delivering the fastest possible EC2 instances, and will help other EC2 users improve performance, reduce latency outliers, and make better use of EC2 features. We'll show how we choose EC2 instance types, how we choose between EC2 Xen modes: HVM, PV, and PVHVM, and the importance of EC2 features such SR-IOV for bare-metal performance. SR-IOV is used by EC2 enhanced networking, and recently for the new i3 instance type for enhanced disk performance as well. We'll also cover kernel tuning and observability tools, from basic to advanced. Advanced performance analysis includes the use of Java and Node.js flame graphs, and the new EC2 Performance Monitoring Counter (PMC) feature released this year."
An inroduction to Terraform, a tool that helps you deploy and change your infrastructure as code. Given at Rencontres Mondiales du Logiciel libre (RMLL) 2017
How to test infrastructure code: automated testing for Terraform, Kubernetes,...Yevgeniy Brikman
This talk is a step-by-step, live-coding class on how to write automated tests for infrastructure code, including the code you write for use with tools such as Terraform, Kubernetes, Docker, and Packer. Topics covered include unit tests, integration tests, end-to-end tests, test parallelism, retries, error handling, static analysis, and more.
Cloud Native Night August 2016, Munich: Talk by Julius Volz (@juliusvolz, Co-founder at Prometheus).
Join our Meetup: www.meetup.com/cloud-native-muc
Abstract: This talk is on monitoring dynamic cloud environments with Prometheus.
In this session, we will start with the importance of monitoring of services and infrastructure. We will discuss about Prometheus an opensource monitoring tool. We will discuss the architecture of Prometheus. We will also discuss some visualization tools which can be used over Prometheus. Then we will have a quick demo for Prometheus and Grafana.
Modern cloud-native applications are incredibly complex systems. Keeping the systems healthy and meeting SLAs for our customers is crucial for long-term success. In this session, we will dive into the three pillars of observability - metrics, logs, tracing - the foundation of successful troubleshooting in distributed systems. You'll learn the gotchas and pitfalls of rolling out the OpenTelemetry stack on Kubernetes to effectively collect all your signals without worrying about a vendor lock in. Additionally we will replace parts of the Prometheus stack to scrape metrics with OpenTelemetry collector and operator.
OpenStack is an open source cloud project and community with broad commercial and developer support. OpenStack is currently developing two interrelated technologies: OpenStack Compute and OpenStack Object Storage. OpenStack Compute is the internal fabric of the cloud creating and managing large groups of virtual private servers and OpenStack Object Storage is software for creating redundant, scalable object storage using clusters of commodity servers to store terabytes or even petabytes of data. In this tutorial, Bret Piatt will explain how to deploy OpenStack Compute and Object Storage, including an overview of the architecture and technology requirements.
This beginning terraform workshop will teach you how to safely create and provision Infrastructure as Code (IAC) using Hashicorp Terraform in an AWS environment. In this class you will learn how to setup and install terraform. You will also be given a walkthrough of Terraform fundamentals. You will be lead through the process of deploying a single server, deploying a cluster and setting up a load balancer. You will also learn how to author Terraform Modules, work with Route53 and how to manage DNS.
Requirements. You will need to have an AWS account set up already with Terraform v0.9.3 installed. You will also need to have git install to download the workshop material.
You can find more informaiton on how to install terraform here: https://www.terraform.io/intro/getting-started/install.html. You can sign up for an AWS account here: https://aws.amazon.com/account/
https://github.com/jasonvance/terraform-introduction
MeetUp Monitoring with Prometheus and Grafana (September 2018)Lucas Jellema
This presentation introduces the concept of monitoring - focusing on why and how and finally on the tools to use. It introduces Prometheus (metrics gathering, processing, alerting), application instrumentation and Prometheus exporters and finally it introduces Grafana as a common companion for dashboarding, alerting and notifications. This presentations also introduces the handson workshop - for which materials are available from https://github.com/lucasjellema/monitoring-workshop-prometheus-grafana
How to Choose the Right Technology, Framework or Tool to Build MicroservicesKai Wähner
Microservices are the next step after SOA: Services implement a limited set of functions. Services are developed, deployed and scaled independently. This way you get shorter time to results and increased flexibility.
Microservices have to be independent regarding build, deployment, data management and business domains. A solid Microservices design requires single responsibility, loose coupling and a decentralized architecture. A Microservice can to be closed or open to partners and public via APIs.
This session discusses technologies such as REST, WebSockets, OSGi, Puppet, Docker, Cloud Foundry, and many more, which can be used to build and deploy Microservices. The main part shows different open service frameworks and proprietary tools to build Microservices on top of these technologies. Live demos illustrate the differences. The audience will learn how to choose the right alternative for building Microservices.
How Netflix Tunes EC2 Instances for PerformanceBrendan Gregg
CMP325 talk for AWS re:Invent 2017, by Brendan Gregg. "
At Netflix we make the best use of AWS EC2 instance types and features to create a high performance cloud, achieving near bare metal speed for our workloads. This session will summarize the configuration, tuning, and activities for delivering the fastest possible EC2 instances, and will help other EC2 users improve performance, reduce latency outliers, and make better use of EC2 features. We'll show how we choose EC2 instance types, how we choose between EC2 Xen modes: HVM, PV, and PVHVM, and the importance of EC2 features such SR-IOV for bare-metal performance. SR-IOV is used by EC2 enhanced networking, and recently for the new i3 instance type for enhanced disk performance as well. We'll also cover kernel tuning and observability tools, from basic to advanced. Advanced performance analysis includes the use of Java and Node.js flame graphs, and the new EC2 Performance Monitoring Counter (PMC) feature released this year."
An inroduction to Terraform, a tool that helps you deploy and change your infrastructure as code. Given at Rencontres Mondiales du Logiciel libre (RMLL) 2017
How to test infrastructure code: automated testing for Terraform, Kubernetes,...Yevgeniy Brikman
This talk is a step-by-step, live-coding class on how to write automated tests for infrastructure code, including the code you write for use with tools such as Terraform, Kubernetes, Docker, and Packer. Topics covered include unit tests, integration tests, end-to-end tests, test parallelism, retries, error handling, static analysis, and more.
Cloud Native Night August 2016, Munich: Talk by Julius Volz (@juliusvolz, Co-founder at Prometheus).
Join our Meetup: www.meetup.com/cloud-native-muc
Abstract: This talk is on monitoring dynamic cloud environments with Prometheus.
In this session, we will start with the importance of monitoring of services and infrastructure. We will discuss about Prometheus an opensource monitoring tool. We will discuss the architecture of Prometheus. We will also discuss some visualization tools which can be used over Prometheus. Then we will have a quick demo for Prometheus and Grafana.
Modern cloud-native applications are incredibly complex systems. Keeping the systems healthy and meeting SLAs for our customers is crucial for long-term success. In this session, we will dive into the three pillars of observability - metrics, logs, tracing - the foundation of successful troubleshooting in distributed systems. You'll learn the gotchas and pitfalls of rolling out the OpenTelemetry stack on Kubernetes to effectively collect all your signals without worrying about a vendor lock in. Additionally we will replace parts of the Prometheus stack to scrape metrics with OpenTelemetry collector and operator.
OpenStack is an open source cloud project and community with broad commercial and developer support. OpenStack is currently developing two interrelated technologies: OpenStack Compute and OpenStack Object Storage. OpenStack Compute is the internal fabric of the cloud creating and managing large groups of virtual private servers and OpenStack Object Storage is software for creating redundant, scalable object storage using clusters of commodity servers to store terabytes or even petabytes of data. In this tutorial, Bret Piatt will explain how to deploy OpenStack Compute and Object Storage, including an overview of the architecture and technology requirements.
This beginning terraform workshop will teach you how to safely create and provision Infrastructure as Code (IAC) using Hashicorp Terraform in an AWS environment. In this class you will learn how to setup and install terraform. You will also be given a walkthrough of Terraform fundamentals. You will be lead through the process of deploying a single server, deploying a cluster and setting up a load balancer. You will also learn how to author Terraform Modules, work with Route53 and how to manage DNS.
Requirements. You will need to have an AWS account set up already with Terraform v0.9.3 installed. You will also need to have git install to download the workshop material.
You can find more informaiton on how to install terraform here: https://www.terraform.io/intro/getting-started/install.html. You can sign up for an AWS account here: https://aws.amazon.com/account/
https://github.com/jasonvance/terraform-introduction
MeetUp Monitoring with Prometheus and Grafana (September 2018)Lucas Jellema
This presentation introduces the concept of monitoring - focusing on why and how and finally on the tools to use. It introduces Prometheus (metrics gathering, processing, alerting), application instrumentation and Prometheus exporters and finally it introduces Grafana as a common companion for dashboarding, alerting and notifications. This presentations also introduces the handson workshop - for which materials are available from https://github.com/lucasjellema/monitoring-workshop-prometheus-grafana
How to Choose the Right Technology, Framework or Tool to Build MicroservicesKai Wähner
Microservices are the next step after SOA: Services implement a limited set of functions. Services are developed, deployed and scaled independently. This way you get shorter time to results and increased flexibility.
Microservices have to be independent regarding build, deployment, data management and business domains. A solid Microservices design requires single responsibility, loose coupling and a decentralized architecture. A Microservice can to be closed or open to partners and public via APIs.
This session discusses technologies such as REST, WebSockets, OSGi, Puppet, Docker, Cloud Foundry, and many more, which can be used to build and deploy Microservices. The main part shows different open service frameworks and proprietary tools to build Microservices on top of these technologies. Live demos illustrate the differences. The audience will learn how to choose the right alternative for building Microservices.
A brief description of a software architecture based on golang micro services.
The keynote was shown during the Golangit meetup @ Codemotion Rome 2015 event on March 27th 2015.
This talk covered the OpenStack basics that VMware Administrators need to be aware of to be successful in their deployments. We also had the Tesora team join us on stage to discuss the importance of Database-as-a-Service with the Trove project!
[Presented at All Things Open 2015 in Raleigh, NC, USA]
OpenStack is one of the fastest-growing and exciting open source projects of our time. OpenStack has drawn together technologists from all over the world to create a cloud operating system and a huge, diverse community behind it. This talk will provide an introduction to OpenStack for newcomers to the project of those who just want to know more. We’ll take a brief look at OpenStack’s history, get a technical overview of the project, learn how to contribute, and check out a few emerging trends and hot topics in the OpenStack world.
Web Scale Applications using NeflixOSS Cloud PlatformSudhir Tonse
Web Scale Applications using NeflixOSS Cloud Platform. Infographics on IaaS, PaaS, SaaS. Commandments of developing a cloud based distributed application.
Building and scaling your containerized microservices on Amazon ECSAmazon Web Services
Microservices is a software architectural method where you decompose complex applications into smaller, independent services. Containers are great for running small decoupled services, but how do you coordinate running microservices in production at scale and what AWS services do you use?
Openstack - An introduction/Installation - Presented at Dr Dobb's conference...Rahul Krishna Upadhyaya
Slide was presented at Dr. Dobb's Conference in Bangalore.
Talks about Openstack Introduction in general
Projects under Openstack.
Contributing to Openstack.
This was presented jointly by CB Ananth and Rahul at Dr. Dobb's Conference Bangalore on 12th Apr 2014.
Slides from the OpenStack 101 presentation with the SA OpenStackers October Meetup in San Antonio TX.
Link to video: https://www.youtube.com/watch?v=DzrSs0n6nSM
According to Gartner, big data will drive $232 billion in IT spending through 2016. The benefits to organizations for adding big data to their information management and analytics infrastructure will force a more rapid cycle of replacing existing solutions.
Learn more about:
• Provisioning a Data-intensive Application Cluster (Hadoop or Spark) on top of OpenStack.
• Building an Architecture combining the Hadoop and OpenStack Ecosystems.
• Build OpenStack Cloud and implement Big Data Architectures with comparative benefits of other Architectures.
Cloud computing gives you a number of advantages, such as being able to scale your application on demand. As a new business looking to use the cloud, you inevitably ask yourself, "Where do I start?" Join us in this session to understand best practices for scaling your resources from zero to millions of users. We will show you how to best combine different AWS services, make smarter decisions for architecting your application, and best practices for scaling your infrastructure in the cloud.
DEVNET-1106 Upcoming Services in OpenStackCisco DevNet
There are several new upcoming OpenStack projects/services that are build upon the core OpenStack infrastructure services. This session will first briefly discuss the new changes introduced for the project governance structure in OpenStack. Subsequently, the focus of the presentation will be to provide feature and architecture details on few of the new projects and services in OpenStack. These will include Trove-Database Service, Sahara-Dataprocessing Service, Congress - Policy Service and Magnum – Container Service. A summary of other OpenStack related services will also be provided.
Slides of QCon London 2016 talk. How stream processing is used within the Uber's Marketplace system to solve a wide range problems, including but not limited to realtime indexing and querying of geospatial time series, aggregation and computing of streaming data, and extracting patterns from data streams. In addition, it will touch upon various TimeSeries analysis and predictions. The underlying systems utilize many open source technologies such as Apache Kafka, Samza and Spark streaming.
Pros and Cons of a MicroServices Architecture talk at AWS ReInventSudhir Tonse
Netflix morphed from a private datacenter based monolithic application into a cloud based Microservices architecture. This talk highlights the pros and cons of building software applications as suites of independently deployable services, as well as practical approaches for overcoming challenges - especially in the context of an elastic but ephemeral cloud ecosystem. What were the lessons learned while building and managing these services? What are the best practices and anti-patterns?
MicroServices at Netflix - challenges of scaleSudhir Tonse
MicroServices has caught on as the design pattern of choice for many companies at scale. While MicroServices and SOA in general have many positives compared to Monolithic apps, it does come with its own challenges - especially when running at scale. These slides were for a 15 min Meetup talk hosted at Cisco
Big Data Pipeline and Analytics Platform Using NetflixOSS and Other Open Sour...Sudhir Tonse
Slides on the OSCON talk about the data platform used at Netflix for event collection, aggregation, and analysis. The platform helps Netflix process and analyze billions of events every day. Attendees will learn how to assemble their own large-scale data pipeline/analytics platform using open source software from NetflixOSS and others, such as Kafka, ElasticSearch, Druid from Metamarkets, and Hive.
Netflix Cloud Platform Building BlocksSudhir Tonse
Architectural Building Blocks of the Netflix Cloud Platform and lessons learned while implementing the same.
Commandments of Web Scale Cloud Deployments
Neuro-symbolic is not enough, we need neuro-*semantic*Frank van Harmelen
Neuro-symbolic (NeSy) AI is on the rise. However, simply machine learning on just any symbolic structure is not sufficient to really harvest the gains of NeSy. These will only be gained when the symbolic structures have an actual semantics. I give an operational definition of semantics as “predictable inference”.
All of this illustrated with link prediction over knowledge graphs, but the argument is general.
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...Jeffrey Haguewood
Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows.
We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases.
This video focuses on the notifications, alerts, and approval requests using Slack for Bonterra Impact Management. The solutions covered in this webinar can also be deployed for Microsoft Teams.
Interested in deploying notification automations for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered QualityInflectra
In this insightful webinar, Inflectra explores how artificial intelligence (AI) is transforming software development and testing. Discover how AI-powered tools are revolutionizing every stage of the software development lifecycle (SDLC), from design and prototyping to testing, deployment, and monitoring.
Learn about:
• The Future of Testing: How AI is shifting testing towards verification, analysis, and higher-level skills, while reducing repetitive tasks.
• Test Automation: How AI-powered test case generation, optimization, and self-healing tests are making testing more efficient and effective.
• Visual Testing: Explore the emerging capabilities of AI in visual testing and how it's set to revolutionize UI verification.
• Inflectra's AI Solutions: See demonstrations of Inflectra's cutting-edge AI tools like the ChatGPT plugin and Azure Open AI platform, designed to streamline your testing process.
Whether you're a developer, tester, or QA professional, this webinar will give you valuable insights into how AI is shaping the future of software delivery.
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...UiPathCommunity
💥 Speed, accuracy, and scaling – discover the superpowers of GenAI in action with UiPath Document Understanding and Communications Mining™:
See how to accelerate model training and optimize model performance with active learning
Learn about the latest enhancements to out-of-the-box document processing – with little to no training required
Get an exclusive demo of the new family of UiPath LLMs – GenAI models specialized for processing different types of documents and messages
This is a hands-on session specifically designed for automation developers and AI enthusiasts seeking to enhance their knowledge in leveraging the latest intelligent document processing capabilities offered by UiPath.
Speakers:
👨🏫 Andras Palfi, Senior Product Manager, UiPath
👩🏫 Lenka Dulovicova, Product Program Manager, UiPath
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
Securing your Kubernetes cluster_ a step-by-step guide to success !KatiaHIMEUR1
Today, after several years of existence, an extremely active community and an ultra-dynamic ecosystem, Kubernetes has established itself as the de facto standard in container orchestration. Thanks to a wide range of managed services, it has never been so easy to set up a ready-to-use Kubernetes cluster.
However, this ease of use means that the subject of security in Kubernetes is often left for later, or even neglected. This exposes companies to significant risks.
In this talk, I'll show you step-by-step how to secure your Kubernetes cluster for greater peace of mind and reliability.
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
GenAISummit 2024 May 28 Sri Ambati Keynote: AGI Belongs to The Community in O...
Architecting for the Cloud using NetflixOSS - Codemash Workshop
1. Architecting for the Cloud
- The Netflix Way!
Codemash PreCompiler Workshop
- Jan 8th 2014
Sudhir Tonse
Manager, Cloud Platform Infrastructure
@stonse
2. NETFLIX
Netflix is the world’s leading Internet television
network with more than 40 million members in >40
countries enjoying more than one billion hours of TV
shows and movies per month, including original
series.
3. NETFLIX SCALE
• 100s of Mid-tier services and applications
• Billions of Requests per day
• ~70 Billion Events per day
• 10,000s of Ec2 Instances in use in multiple AWS Regions/
Zones
• Cassandra NoSQL database in a Global Ring spanning
regions: Terabytes of data
• At peak consumes ~1/3 of US Internet Bandwidth
4. I am PaaS (and so can You!)
Your Cloud PaaS
@stonse
6. Agenda
•
PaaS? What is it good for?
•
NetflixOSS -> Build your own PaaS the Netflix Way!
• What you need to build, deploy and operate a Cloud Native set of applications
• Cloud Native Architecture
• Platform Services -> Application Services on top of IaaS
• Platform Libraries -> Application Framework Libraries
• Platform Tools/Operational Utilities -> Helps deploy and manage your
Cloud Apps
•
@stonse
Build your own Web Service using NetflixOSS libraries
7. PaaS? What is it?
Libraries, Tools and Frameworks that enable
one to Build, Deploy and Manage Software
Services on Ephemeral, On Demand Virtual
Machines (IaaS)
@stonse
18. Think …
Cloud Native Architecture
@stonse
More at http://www.siia.net/presentations/software/
AATC2013/OpenSourceAtNetflix.pdf
- Adrian Cockroft (@adrianco)
25. • RPS
• Load Average
Slow Growth
Predictable Bursts
Compute
• …
Time
Time
Compute
• Use CloudWatch
Time
Periodic Jobs
Compute
Scale UP & Down based on
Demand
Compute
•
Compute
ON DEMAND AUTO SCALING!
Time
Unpredictable Bursts
Time
Steady State
40. STEP 1
Choice A: Bare bones Web Service Template
$git clone https://github.com/Netflix/karyon.git
$cd karyon
$./gradlew clean build
$./gradlew :karyon-examples:hello-netflixoss:jettyRun
Choice B: Full blown Recipe (RSS Reader
Recipe)
$git clone https://github.com/Netflix/recipes-rss.git
$cd recipes-rss
$./gradlew clean build
@stonse
41. STEP 2
•
Customize your App using .war of previous step as a template
• TIPS (optional)
• Use Ribbon for RPC calls
• Use Hystrix/RxJava for wrapping concurrent calls/RPC calls (Latency/Fault
Tolerance)
@stonse
51. RIBBON (RPC) CALL
try {
RestClient client = (RestClient)
ClientFactory.getNamedClient("FlixmashClient");
HttpRequest request = HttpRequest.newBuilder().uri(new URI("/
flixmash/rest/v1/movies/popularMovies")).build();
response = client.executeWithLoadBalancer(request);
String movies = response.getEntity(String.class);
return movies;
} catch (Exception exc) {
throw new RuntimeException("Exception", exc);
} finally {
if (response!=null){
//hand the connection back to the pool
response.close();
}
}
52. Archaius: Configuration (properties) as a Service
•
Property Driven Development
for (int i=0; I < numRowsProperty.get(); i++){
fetchMovieForRow(i, numMoviePerRowProperty.get());
}
@stonse
69. Simian Army: Availability/Resiliency as a Service
•
Chaos Monkey
• Kills instances
•
Janitor Monkey
• Clean up unused Resources
•
Conformity Monkey
• Confirms adherence to Best
Practices
•
Others (yet to be open sourced)
• Latency Monkey
• Chaos Kong
• …
@stonse
75. Workshop …
Hands On: Building Hello Netflix OSS application
•
Concepts
•
•
Configuration
•
Build Script (gradle)
•
Application Container/Bootstrap
•
JSR-311 Web Resource
•
Properties via Archaius
•
Metrics via Servo
•
Dependency Injection via Governator/Guice
•
RPC calls via Ribbon/Eureka
•
Resiliency via Hystrix (wrappers on top of Ribbon)
•
Invoking a Mid-tier Web Service
•
@stonse
Project Layout
AdminConsole (port 8077)
76. Workshop …
Hands On: Building Hello Netflix OSS application
•
@stonse
Instructions at http://bit.ly/1dlDyVC
80. JSR-311 Web Resource
1. Hello World Resource -> https://netflix.box.com/s/
nyspqi1an095d4p5xd7k
2. Flixmash Resource -> https://netflix.box.com/s/
nb1nd7bqk7qcmxnwuol0
@Path("/flixmash")
public class FlixmashResource {
@GET
@Path("popularMovies")
@Produces({MediaType.APPLICATION_JSON})
public Response movies() {
// Start timer
Stopwatch stopwatch = getMoviesTimer.start();
JSONObject response = new JSONObject();
try {
getMoviesCounter.increment();
response.put("Movies",
flixmashManager.getPopularMovies()); // obtain movies from our
flixMovies component
return Response.ok(response.toString()).build();
} catch (JSONException e) {
getMoviesErrorCounter.increment();
logger.error("Error creating json response.", e);
return
Response.status(Response.Status.INTERNAL_SERVER_ERROR).build
();
} finally {
stopwatch.stop();
}
}
81. ARCHAIUS: PROPERTIES
flixmash.popularMovieList=Lord of the Ring,Matrix,Harry Potter,Gravity
private static final DynamicStringListProperty popularMovieList
= new DynamicStringListProperty("flixmash.popularMovieList",
(List<String>) null);
List<String> movies = popularMovieList.get();
85. Takeaways …
•
Cloud Computing and Cloud based Services requires embracing Cloud Native
- Adrian Cockroft (@adrianco)
•
NetflixOSS is here to help!
• You can build your own custom PaaS the Netflix Way!
http://netflix.github.io
@stonse
Yes, we are Hiring!