This document provides a summary of a final report analyzing Netflix's content distribution network architecture. It describes Netflix's growth from 1999 to becoming a leading video streaming service. The report analyzes Netflix's deployment of Amazon Web Services cloud computing and its own content delivery network called Open Connect. It also examines Netflix's traffic flow, use of content distribution networks, and strategies for efficiently streaming video at large scale. The document aims to understand Netflix's current architecture and discuss possibilities for further optimizing its CDN and video delivery.
Infrastructure & System Monitoring using PrometheusMarco Pas
The document introduces infrastructure and system monitoring using Prometheus. It discusses the importance of monitoring, common things to monitor like services, applications, and OS metrics. It provides an overview of Prometheus including its main components and data format. The document demonstrates setting up Prometheus, adding host metrics using Node Exporter, configuring Grafana, monitoring Docker containers using cAdvisor, configuring alerting in Prometheus and Alertmanager, instrumenting application code, and integrating Consul for service discovery. Live code demos are provided for key concepts.
This document provides an overview of setting up monitoring for MySQL and MongoDB servers using Prometheus and Grafana. It discusses installing and configuring Prometheus, Grafana, exporters for collecting metrics from MySQL, MongoDB and systems, and dashboards for visualizing the metrics in Grafana. The tutorial hands-on sets up Prometheus and Grafana in two virtual machines to monitor a MySQL master-slave replication setup and MongoDB cluster.
Prometheus is an open-source monitoring system that collects metrics from instrumented systems and applications and allows for querying and alerting on metrics over time. It is designed to be simple to operate, scalable, and provides a powerful query language and multidimensional data model. Key features include no external dependencies, metrics collection by scraping endpoints, time-series storage, and alerting handled by the AlertManager with support for various integrations.
Architecting for the Cloud using NetflixOSS - Codemash WorkshopSudhir Tonse
This document provides an overview and agenda for a presentation on architecting for the cloud using the Netflix approach. Some key points:
- Netflix has over 40 million members streaming over 1 billion hours per month of content across over 40 countries.
- Netflix runs on AWS and handles billions of requests per day across thousands of instances globally.
- The presentation will discuss how to build your own platform as a service (PaaS) based on Netflix's open source libraries, including platform services, libraries, and tools.
- The Netflix approach focuses on microservices, automation, and resilience to support rapid iteration on cloud infrastructure.
This document provides an overview of microservices and various methodologies for deploying microservices. It begins with an introduction comparing monolithic and microservice architectures. Next, it discusses considerations for microservice design like scalability and complexity. It then covers challenges of deploying microservices like monitoring and scaling. The document proceeds to introduce Kubernetes as a container orchestration tool for deploying microservices. It includes descriptions of various Kubernetes components like pods, deployments, services, and ingress. Finally, it demonstrates deploying microservices locally using Minikube to simulate a Kubernetes cluster on a single node.
Presented at GDG Devfest Ukraine 2018.
Prometheus has become the defacto monitoring system for cloud native applications, with systems like Kubernetes and Etcd natively exposing Prometheus metrics. In this talk Tom will explore all the moving part for a working Prometheus-on-Kubernetes monitoring system, including kube-state-metrics, node-exporter, cAdvisor and Grafana. You will learn about the various methods for getting to a working setup: the manual approach, using CoreOS’s Prometheus Operator, or using Prometheus Ksonnet Mixin. Tom will also share some little tips and tricks for getting the most out of your Prometheus monitoring, including the common pitfalls and what you should be alerting on.
Red Hat OpenShift 4 allows for automated and customized deployments. The Full Stack Automation method fully automates installation and updates of both the OpenShift platform and Red Hat Enterprise Linux CoreOS host operating system. The Pre-existing Infrastructure method allows OpenShift to be deployed on user-managed infrastructure, where the customer provisions resources like load balancers and DNS. Both methods use the openshift-install tool to generate ignition configs and monitor the cluster deployment.
We are on the cusp of a new era of application development software: instead of bolting on operations as an after-thought to the software development process, Kubernetes promises to bring development and operations together by design.
Infrastructure & System Monitoring using PrometheusMarco Pas
The document introduces infrastructure and system monitoring using Prometheus. It discusses the importance of monitoring, common things to monitor like services, applications, and OS metrics. It provides an overview of Prometheus including its main components and data format. The document demonstrates setting up Prometheus, adding host metrics using Node Exporter, configuring Grafana, monitoring Docker containers using cAdvisor, configuring alerting in Prometheus and Alertmanager, instrumenting application code, and integrating Consul for service discovery. Live code demos are provided for key concepts.
This document provides an overview of setting up monitoring for MySQL and MongoDB servers using Prometheus and Grafana. It discusses installing and configuring Prometheus, Grafana, exporters for collecting metrics from MySQL, MongoDB and systems, and dashboards for visualizing the metrics in Grafana. The tutorial hands-on sets up Prometheus and Grafana in two virtual machines to monitor a MySQL master-slave replication setup and MongoDB cluster.
Prometheus is an open-source monitoring system that collects metrics from instrumented systems and applications and allows for querying and alerting on metrics over time. It is designed to be simple to operate, scalable, and provides a powerful query language and multidimensional data model. Key features include no external dependencies, metrics collection by scraping endpoints, time-series storage, and alerting handled by the AlertManager with support for various integrations.
Architecting for the Cloud using NetflixOSS - Codemash WorkshopSudhir Tonse
This document provides an overview and agenda for a presentation on architecting for the cloud using the Netflix approach. Some key points:
- Netflix has over 40 million members streaming over 1 billion hours per month of content across over 40 countries.
- Netflix runs on AWS and handles billions of requests per day across thousands of instances globally.
- The presentation will discuss how to build your own platform as a service (PaaS) based on Netflix's open source libraries, including platform services, libraries, and tools.
- The Netflix approach focuses on microservices, automation, and resilience to support rapid iteration on cloud infrastructure.
This document provides an overview of microservices and various methodologies for deploying microservices. It begins with an introduction comparing monolithic and microservice architectures. Next, it discusses considerations for microservice design like scalability and complexity. It then covers challenges of deploying microservices like monitoring and scaling. The document proceeds to introduce Kubernetes as a container orchestration tool for deploying microservices. It includes descriptions of various Kubernetes components like pods, deployments, services, and ingress. Finally, it demonstrates deploying microservices locally using Minikube to simulate a Kubernetes cluster on a single node.
Presented at GDG Devfest Ukraine 2018.
Prometheus has become the defacto monitoring system for cloud native applications, with systems like Kubernetes and Etcd natively exposing Prometheus metrics. In this talk Tom will explore all the moving part for a working Prometheus-on-Kubernetes monitoring system, including kube-state-metrics, node-exporter, cAdvisor and Grafana. You will learn about the various methods for getting to a working setup: the manual approach, using CoreOS’s Prometheus Operator, or using Prometheus Ksonnet Mixin. Tom will also share some little tips and tricks for getting the most out of your Prometheus monitoring, including the common pitfalls and what you should be alerting on.
Red Hat OpenShift 4 allows for automated and customized deployments. The Full Stack Automation method fully automates installation and updates of both the OpenShift platform and Red Hat Enterprise Linux CoreOS host operating system. The Pre-existing Infrastructure method allows OpenShift to be deployed on user-managed infrastructure, where the customer provisions resources like load balancers and DNS. Both methods use the openshift-install tool to generate ignition configs and monitor the cluster deployment.
We are on the cusp of a new era of application development software: instead of bolting on operations as an after-thought to the software development process, Kubernetes promises to bring development and operations together by design.
Serverless with Google Cloud FunctionsJerry Jalava
This document discusses Google Cloud Functions, a serverless platform for running code in response to events. It provides an overview of Google Cloud Functions' features such as triggers from Cloud Pub/Sub and Storage, integration with other Google Cloud services, and use cases including building mobile backends, APIs, data processing, and IoT. The document also discusses using Google Cloud Functions with Firebase and pricing.
Observability, Distributed Tracing, and Open Source: The Missing PrimerVMware Tanzu
Open source tools like OpenTelemetry, OpenTracing, and W3C Trace Context are helping to standardize distributed tracing and observability. This allows developers to understand problems in microservices architectures by propagating unique trace IDs and collecting metrics and traces across services. While open source tools are useful for development and pre-production, commercial solutions are needed to handle production workloads at scale with additional features like access control and automated instrumentation. Standardization through open source is key to managing today's complexity in distributed systems.
While many organizations have started to automate their software develop processes, many still engineer their infrastructure largely by hand. Treating your infrastructure just like any other piece of code creates a “programmable infrastructure” that allows you to take full advantage of the scalability and reliability of the AWS cloud. This session will walk through practical examples of how AWS customers have merged infrastructure configuration with application code to create application-specific infrastructure and a truly unified development lifecycle. You will learn how AWS customers have leveraged tools like CloudFormation, orchestration engines, and source control systems to enable their applications to take full advantage of the scalability and reliability of the AWS cloud, create self-reliant applications, and easily recover when things go seriously wrong with their infrastructure.
Infrastructure-as-Code (IaC) using TerraformAdin Ermie
Learn the benefits of Infrastructure-as-Code (IaC), what Terraform is and why people love it, along with a breakdown of the basics (including live demo deployments). Then wrap up with a comparison of Azure Resource Manager (ARM) templates versus Terraform, consider some best practices, and walk away with some key resources in your Terraform learning adventure.
This document provides an overview of a workshop on using Terraform to manage AWS infrastructure as code. The workshop objective is to get started using Terraform to build, change, and version AWS resources in an easy and efficient manner. Attendees will learn about Terraform concepts like providers, resources, execution plans, and state management. The workshop will demonstrate how to install Terraform and AWS CLI, configure credentials, and build sample infrastructure on AWS including a "Hello World" instance. It will also cover additional Terraform topics like variables, outputs, modules, and data sources.
DevSecOps, An Organizational Primer - AWS Security Week at the SF LoftAmazon Web Services
DevSecOps, An Organizational Primer - AWS Security Week at the San Francisco Loft
We examine building DevSecOps culture for you or your customers, which includes foundational practices and scaling functions to instantiate and resiliently operate a DevSecOps model. To achieve this shift, we analyze common success patterns, such as how to use a secure CI/CD pipeline. You’ll learn key points such as building security owners, integrating continuous compliance and security, and removing people from the data to vastly improve your security posture over traditional operating models. Takeaways include a blueprint for building a DevSecOps operating model in your organization; an understanding the security practitioners' point of view and embracing it to drive innovation; and ways to identify operating characteristics in your organization and use them to drive a strategy for DevSecOps.
Level: 100
Speaker: Tim Anderson - Tech Industry Specialist, AWS Security
Quarta puntata del MuleSoft Meetup di Milano - 22 Luglio 2021
Approfondiremo insieme a Giacomo che opzioni abbiamo per esternalizzare i log di Mule e con Gonzalo vedremo in dettaglio il modulo di Advanced Monitoring e le differenze fra le sottoscrizioni Platinum e Titanium.
End to-end monitoring with the prometheus operator - Max IndenParis Container Day
The document discusses end-to-end monitoring of containers using the Prometheus monitoring system and the Prometheus Operator. It describes what Prometheus and the Prometheus Operator are, how they can be used to monitor applications running in Kubernetes clusters, and how the Prometheus Operator automates management of Prometheus deployments. It also provides a demo of using the Kube-Prometheus project which bundles Prometheus, Alertmanager and dashboards into a single installation command.
Infrastructure-as-Code with Pulumi- Better than all the others (like Ansible)?Jonas Hecht
There's a new Infrastructure-as-Code (IaC) kid on the block: Pulumi is there to frighten the established: Chef, Puppet, Terraform, Cloudformation, Ansible... But is it really the "better" tool and how could they be compared? Is it only hype-driven? We'll find out, incl. lot's of example code. (ContainerConf / Continuous Lifecycle 2019 Talk in Mannheim)
Example GitHub code: https://github.com/jonashackt/pulumi-python-aws-ansible
https://github.com/jonashackt/pulumi-typescript-aws-fargate
Manchester MuleSoft Meetup #6 - Runtime Fabric with Mulesoft Akshata Sawant
Come join us at the Online Meetup to learn more about ServiceNow and Gmail Integration with MuleSoft with detailed Demo. Help us spread the knowledge of Mule!
A brief agenda:
> Networking and Knowledge sharing.
> MuleSoft Latest Product Release Updates.
> Runtime Fabric in depth architecture.
> Finally, we will wrap-up this event with the agenda for the next meetup.
Stay connected to get updates on what's new in MuleSoft.
Terraform Best Practices - DevOps Unicorns 2019Anton Babenko
Terraform best practices include using modules to break infrastructure into reusable components, structuring configurations in a one-in-one approach with directories for each module, and avoiding workspaces in favor of additional modules. Terraform 0.12 benefits developers most through features like loops and conditionals that enable more flexible modules, while users appreciate minor syntax improvements. The presentation emphasizes reusability, separation of concerns, and standardization through open-source modules.
This document introduces infrastructure as code (IaC) using Terraform and provides examples of deploying infrastructure on AWS including:
- A single EC2 instance
- A single web server
- A cluster of web servers using an Auto Scaling Group
- Adding a load balancer using an Elastic Load Balancer
It also discusses Terraform concepts and syntax like variables, resources, outputs, and interpolation. The target audience is people who deploy infrastructure on AWS or other clouds.
This document discusses using Grafana to optimize visualization of metrics from Prometheus in a dynamic environment. It describes deploying multiple Prometheus instances to monitor over 100 instances per service across various services running on EC2. Key Grafana features discussed include templating to dynamically filter dashboards, panel repetition to show multiple graphs, and scripted dashboards to generate dashboards from JSON definitions. The document provides examples of using these features to create service trend dashboards, dynamically refresh dashboards based on time range changes, switch data sources, and generate alert dashboards from Prometheus alert views.
Prometheus has become the defacto monitoring system for cloud native applications, with systems like Kubernetes and Etcd natively exposing Prometheus metrics. In this talk Tom will explore all the moving part for a working Prometheus-on-Kubernetes monitoring system, including kube-state-metrics, node-exporter, cAdvisor and Grafana. You will learn about the various methods for getting to a working setup: the manual approach, using CoreOSs Prometheus Operator, or using Prometheus Ksonnet Mixin. Tom will also share some little tips and tricks for getting the most out of your Prometheus monitoring, including the common pitfalls and what you should be alerting on.
MeetUp Monitoring with Prometheus and Grafana (September 2018)Lucas Jellema
This presentation introduces the concept of monitoring - focusing on why and how and finally on the tools to use. It introduces Prometheus (metrics gathering, processing, alerting), application instrumentation and Prometheus exporters and finally it introduces Grafana as a common companion for dashboarding, alerting and notifications. This presentations also introduces the handson workshop - for which materials are available from https://github.com/lucasjellema/monitoring-workshop-prometheus-grafana
This document summarizes internet infrastructure and opportunities in Africa. It notes that over 1 billion people in Africa are reachable by fiber, and mobile users are expected to double to 60% penetration in the next 5 years. Key hubs like Johannesburg, Cape Town, and Nairobi have improved metro connectivity averaging 100 Mbps. There are over 50 data centers across Africa concentrated in South Africa, Kenya, and Nigeria, which represent 95% of the continent's data center capacity. Growth opportunities exist in emerging markets in countries like Tanzania, Cote d'Ivoire, and Ghana, which are seeing rising GDP and lack dedicated data center facilities currently.
Netflix Edge Engineering Open House Presentations - June 9, 2016Daniel Jacobson
Netflix's Edge Engineering team is responsible for handling all device traffic for to support the user experience, including sign-up, discovery and the triggering of the playback experience. Developing and maintaining this set of massive scale services is no small task and its success is the difference between millions of happy streamers or millions of missed opportunities.
This video captures the presentations delivered at the first ever Edge Engineering Open House at Netflix. This video covers the primary aspects of our charter, including the evolution of our API and Playback services as well as building a robust developer experience for the internal consumers of our APIs.
Serverless with Google Cloud FunctionsJerry Jalava
This document discusses Google Cloud Functions, a serverless platform for running code in response to events. It provides an overview of Google Cloud Functions' features such as triggers from Cloud Pub/Sub and Storage, integration with other Google Cloud services, and use cases including building mobile backends, APIs, data processing, and IoT. The document also discusses using Google Cloud Functions with Firebase and pricing.
Observability, Distributed Tracing, and Open Source: The Missing PrimerVMware Tanzu
Open source tools like OpenTelemetry, OpenTracing, and W3C Trace Context are helping to standardize distributed tracing and observability. This allows developers to understand problems in microservices architectures by propagating unique trace IDs and collecting metrics and traces across services. While open source tools are useful for development and pre-production, commercial solutions are needed to handle production workloads at scale with additional features like access control and automated instrumentation. Standardization through open source is key to managing today's complexity in distributed systems.
While many organizations have started to automate their software develop processes, many still engineer their infrastructure largely by hand. Treating your infrastructure just like any other piece of code creates a “programmable infrastructure” that allows you to take full advantage of the scalability and reliability of the AWS cloud. This session will walk through practical examples of how AWS customers have merged infrastructure configuration with application code to create application-specific infrastructure and a truly unified development lifecycle. You will learn how AWS customers have leveraged tools like CloudFormation, orchestration engines, and source control systems to enable their applications to take full advantage of the scalability and reliability of the AWS cloud, create self-reliant applications, and easily recover when things go seriously wrong with their infrastructure.
Infrastructure-as-Code (IaC) using TerraformAdin Ermie
Learn the benefits of Infrastructure-as-Code (IaC), what Terraform is and why people love it, along with a breakdown of the basics (including live demo deployments). Then wrap up with a comparison of Azure Resource Manager (ARM) templates versus Terraform, consider some best practices, and walk away with some key resources in your Terraform learning adventure.
This document provides an overview of a workshop on using Terraform to manage AWS infrastructure as code. The workshop objective is to get started using Terraform to build, change, and version AWS resources in an easy and efficient manner. Attendees will learn about Terraform concepts like providers, resources, execution plans, and state management. The workshop will demonstrate how to install Terraform and AWS CLI, configure credentials, and build sample infrastructure on AWS including a "Hello World" instance. It will also cover additional Terraform topics like variables, outputs, modules, and data sources.
DevSecOps, An Organizational Primer - AWS Security Week at the SF LoftAmazon Web Services
DevSecOps, An Organizational Primer - AWS Security Week at the San Francisco Loft
We examine building DevSecOps culture for you or your customers, which includes foundational practices and scaling functions to instantiate and resiliently operate a DevSecOps model. To achieve this shift, we analyze common success patterns, such as how to use a secure CI/CD pipeline. You’ll learn key points such as building security owners, integrating continuous compliance and security, and removing people from the data to vastly improve your security posture over traditional operating models. Takeaways include a blueprint for building a DevSecOps operating model in your organization; an understanding the security practitioners' point of view and embracing it to drive innovation; and ways to identify operating characteristics in your organization and use them to drive a strategy for DevSecOps.
Level: 100
Speaker: Tim Anderson - Tech Industry Specialist, AWS Security
Quarta puntata del MuleSoft Meetup di Milano - 22 Luglio 2021
Approfondiremo insieme a Giacomo che opzioni abbiamo per esternalizzare i log di Mule e con Gonzalo vedremo in dettaglio il modulo di Advanced Monitoring e le differenze fra le sottoscrizioni Platinum e Titanium.
End to-end monitoring with the prometheus operator - Max IndenParis Container Day
The document discusses end-to-end monitoring of containers using the Prometheus monitoring system and the Prometheus Operator. It describes what Prometheus and the Prometheus Operator are, how they can be used to monitor applications running in Kubernetes clusters, and how the Prometheus Operator automates management of Prometheus deployments. It also provides a demo of using the Kube-Prometheus project which bundles Prometheus, Alertmanager and dashboards into a single installation command.
Infrastructure-as-Code with Pulumi- Better than all the others (like Ansible)?Jonas Hecht
There's a new Infrastructure-as-Code (IaC) kid on the block: Pulumi is there to frighten the established: Chef, Puppet, Terraform, Cloudformation, Ansible... But is it really the "better" tool and how could they be compared? Is it only hype-driven? We'll find out, incl. lot's of example code. (ContainerConf / Continuous Lifecycle 2019 Talk in Mannheim)
Example GitHub code: https://github.com/jonashackt/pulumi-python-aws-ansible
https://github.com/jonashackt/pulumi-typescript-aws-fargate
Manchester MuleSoft Meetup #6 - Runtime Fabric with Mulesoft Akshata Sawant
Come join us at the Online Meetup to learn more about ServiceNow and Gmail Integration with MuleSoft with detailed Demo. Help us spread the knowledge of Mule!
A brief agenda:
> Networking and Knowledge sharing.
> MuleSoft Latest Product Release Updates.
> Runtime Fabric in depth architecture.
> Finally, we will wrap-up this event with the agenda for the next meetup.
Stay connected to get updates on what's new in MuleSoft.
Terraform Best Practices - DevOps Unicorns 2019Anton Babenko
Terraform best practices include using modules to break infrastructure into reusable components, structuring configurations in a one-in-one approach with directories for each module, and avoiding workspaces in favor of additional modules. Terraform 0.12 benefits developers most through features like loops and conditionals that enable more flexible modules, while users appreciate minor syntax improvements. The presentation emphasizes reusability, separation of concerns, and standardization through open-source modules.
This document introduces infrastructure as code (IaC) using Terraform and provides examples of deploying infrastructure on AWS including:
- A single EC2 instance
- A single web server
- A cluster of web servers using an Auto Scaling Group
- Adding a load balancer using an Elastic Load Balancer
It also discusses Terraform concepts and syntax like variables, resources, outputs, and interpolation. The target audience is people who deploy infrastructure on AWS or other clouds.
This document discusses using Grafana to optimize visualization of metrics from Prometheus in a dynamic environment. It describes deploying multiple Prometheus instances to monitor over 100 instances per service across various services running on EC2. Key Grafana features discussed include templating to dynamically filter dashboards, panel repetition to show multiple graphs, and scripted dashboards to generate dashboards from JSON definitions. The document provides examples of using these features to create service trend dashboards, dynamically refresh dashboards based on time range changes, switch data sources, and generate alert dashboards from Prometheus alert views.
Prometheus has become the defacto monitoring system for cloud native applications, with systems like Kubernetes and Etcd natively exposing Prometheus metrics. In this talk Tom will explore all the moving part for a working Prometheus-on-Kubernetes monitoring system, including kube-state-metrics, node-exporter, cAdvisor and Grafana. You will learn about the various methods for getting to a working setup: the manual approach, using CoreOSs Prometheus Operator, or using Prometheus Ksonnet Mixin. Tom will also share some little tips and tricks for getting the most out of your Prometheus monitoring, including the common pitfalls and what you should be alerting on.
MeetUp Monitoring with Prometheus and Grafana (September 2018)Lucas Jellema
This presentation introduces the concept of monitoring - focusing on why and how and finally on the tools to use. It introduces Prometheus (metrics gathering, processing, alerting), application instrumentation and Prometheus exporters and finally it introduces Grafana as a common companion for dashboarding, alerting and notifications. This presentations also introduces the handson workshop - for which materials are available from https://github.com/lucasjellema/monitoring-workshop-prometheus-grafana
This document summarizes internet infrastructure and opportunities in Africa. It notes that over 1 billion people in Africa are reachable by fiber, and mobile users are expected to double to 60% penetration in the next 5 years. Key hubs like Johannesburg, Cape Town, and Nairobi have improved metro connectivity averaging 100 Mbps. There are over 50 data centers across Africa concentrated in South Africa, Kenya, and Nigeria, which represent 95% of the continent's data center capacity. Growth opportunities exist in emerging markets in countries like Tanzania, Cote d'Ivoire, and Ghana, which are seeing rising GDP and lack dedicated data center facilities currently.
Netflix Edge Engineering Open House Presentations - June 9, 2016Daniel Jacobson
Netflix's Edge Engineering team is responsible for handling all device traffic for to support the user experience, including sign-up, discovery and the triggering of the playback experience. Developing and maintaining this set of massive scale services is no small task and its success is the difference between millions of happy streamers or millions of missed opportunities.
This video captures the presentations delivered at the first ever Edge Engineering Open House at Netflix. This video covers the primary aspects of our charter, including the evolution of our API and Playback services as well as building a robust developer experience for the internal consumers of our APIs.
Netflix uses its Open Connect content delivery network to stream video content directly to internet service providers. The Open Connect network consists of appliances located within ISP networks that cache Netflix's content. This allows 95% of Netflix's streaming traffic to be served directly from the Open Connect caches, reducing load on upstream networks. The control plane determines the optimal stream for a user, while the data plane serves the cached content through Open Connect during playback. Netflix fills the caches overnight using predictive algorithms to pre-position popular content at locations close to users.
Netflix Open Connect: Delivering Internet TV to the worldInternet Society
This document discusses Netflix's global streaming services and partnerships with internet service providers (ISPs). Some key points:
- Netflix now serves over 190 countries with over 1 billion hours streamed per month and 81.5 million members globally.
- Netflix partners with ISPs by allowing them to embed Netflix Open Connect appliances (OCAs) within their networks at no cost, in order to directly deliver Netflix content to users and reduce upstream internet traffic.
- There are over 50 global points of presence for OCAs. Requirements for ISPs to participate include having at least 5Gbps of peak traffic and hosting a 1U or 2U OCA appliance within their network.
A presentation on the Netflix Cloud Architecture and NetflixOSS open source. For the All Things Open 2015 conference in Raleigh 2015/10/19. #ATO2015 #NetflixOSS
The document contains peering information for various Malaysian and Southeast Asian networks. It lists their autonomous system numbers (ASNs), traffic profiles (e.g. eyeball, content, balanced), traffic volumes, peering policies and locations, PeeringDB entries, and contacts. The networks range in size from regional eyeball networks to large global content providers.
ABC News implemented a new digital network called BureauNET to streamline its news gathering workflow. BureauNET replaced expensive satellite uplinks with internet-based technology, allowing ABC to transmit live video and files over a single managed network connection. This reduced operating costs by over $1.5 million per year. Partners Level(3) Communications and Pathfire provided the network and video transmission software. The new system significantly increased content sharing between bureaus and eased the process of delivering video from remote locations.
CDNs are surrogate web servers distributed across different data centers in different geographical regions to deliver content to end-users, based on proximity of this user.
In this article we show the historical development of this technology.
Netflix is the world's leading internet television network. In this presentation, I talk about the aspects of Netflix culture that have contributed to its success. The freedom and responsibility culture, with highly aligned, loosely coupled teams of amazing, enthusiastic, helpful, high performance people with excellent judgement has helped build Netflix to adapt and change quickly, and iterate to make the best service possible. By eliminating rules and processes, and ensuring that employees have the contexts of the business in all aspects, Netflix has enabled employees to use their judgement to get things done rather than relying on control or process. This has resulted in Netflix becoming a leader in not only internet streaming, but also cloud computing, media, and culture. Because of the Netflix culture, we have been able to attract and retain great employees.
I give a few examples from my team, Edge Engineering of how the culture enables us to build the high scale, resilient and dynamic services that are the front door to the Netflix streaming application, and how the freedom we have has enabled us to open source core technologies that are needed for large scale, service-oriented architectures in a cloud environment.
This slideshow also gives a high level overview of how the streaming service works, and how Netflix's Open Connect Appliance can help ISPs.
Arcus Advisors Report_Quality of ServiceScott Landman
This document discusses four basic networking services - connectivity, bandwidth, latency, and advanced services - that network operators provide. It focuses on latency and how some operators are evaluating models to monetize latency through various content delivery strategies like caching content close to users or deploying fiber in straight lines to minimize latency. While guaranteed low-latency delivery faces challenges, future opportunities to monetize latency include prioritizing traffic on mobile networks through small cells and caches, delivering work content with lower latency for BYOD users, and caching content closer to homes or devices.
An SDN Based Approach To Measuring And Optimizing ABR Video Quality Of Experi...Cisco Service Provider
Reprinted with permission of NCTA, from the 2014 Cable Connection Spring Technical Forum Conference Proceedings. For more information on Cisco video solutions, visit: http://www.cisco.com/c/en/us/products/video/index.html
A content delivery network (CDN) distributes content across multiple servers located in different geographic locations. This improves performance by serving content from the server closest to the user. CDNs provide advantages like decreasing server load, faster content delivery, 100% availability even during outages, increased concurrent users, and more control over asset delivery. While CDNs improve performance, they also have disadvantages to consider like increased complexity and upfront costs.
Knowledge Network is a public broadcaster in British Columbia that relies on an automated, file-based workflow using Telestream products like Vantage and Pipeline to ingest, transcode, and deliver programming. Vantage transcodes content into various formats for broadcast and online distribution. It has replaced their tape-based system, improving efficiency. Vantage allows them to process multiple files simultaneously and handle content in various formats from international distributors. The automated workflow is vital to their operations.
As consumers demand ever-greater amounts of high-quality content over the Internet,
service providers (SPs) are finding it difficult to increase revenues while containing costs.
This is due mainly to two trends: (1) over-the-top (OTT) content providers have outsourced
the delivery of content to pure-play content delivery network (CDN) companies, and (2) traffic
growth (with no resulting revenue benefit) is increasing network build-out and maintenance
costs for SPs.
The document discusses video streaming services at the University of Glasgow. It recommends maintaining the current Windows Media streaming service and connecting the media server to a storage network to enable remote file uploads. While there are costs, a centralized streaming service provides benefits like a scalable and reliable way to deliver multimedia teaching materials live or on-demand.
Scaling to millions of users with Amazon CloudFront - April 2017 AWS Online T...Amazon Web Services
Learning Objectives:
• Learn how to use CloudFront dynamic delivery features • See a live demo and learn how to take advantage of Cloud Front newest features
Traditionally, content delivery networks (CDNs) were designed to accelerate static content. Amazon CloudFront supports delivery of an entire website, including dynamic, static, streaming and interactive content using a global network of edge locations. CloudFront integrates with other AWS services that are built to scale massively. Together, the solution can automatically scale to millions of users by leveraging the global reach of CloudFront and the auto scaling capability of AWS platform. In this talk, we introduce you to various design patterns and best practices to build a massively scalable solution using CloudFront. We discuss how this scale can be achieved without compromising on availability, security or cost.
PLNOG 17 - Stefan Meinders - Slow is the new DownPROIDEA
In this presentation, we will review trends in Over-the-Top (OTT) traffic usage, user behavior and rapidly growing volumes of video streaming from services such as Netflix. The amount of video traffic carried by networks and its interconnections is significant. Delivery methods and encryption are challenges for today’s methods of network analysis—as are unexpected network changes that can drastically impact subscribers, transit costs, and network overload. Recent research unveiled that existing monitoring tools cannot cope with these variations in traffic delivery, caching strategies, and encryption. Finally, we describe how the use of “big data” techniques and multi-dimensional databases combined with global internet service mapping provides new insights into traffic delivery, consumption, and quality.
Netflix over Qos Enabled LTE Research Paper FinalAjit Kahaduwe
1) The document discusses how using Quality of Service (QoS) mechanisms can enhance the experience of streaming video applications like Netflix over cellular networks.
2) Tests were conducted on an LTE network to analyze how giving Netflix higher priority through QoS impacted video quality and buffering during congestion compared to default "best effort" treatment.
3) The results showed that higher QoS priority for Netflix improved video quality and reduced buffering events during congestion compared to best effort, demonstrating how QoS can provide a better user experience for video streaming.
Implementing a Caching Scheme for Media Streaming in a Proxy ServerAbdelrahman Hosny
In the past few years, websites have moved from being
static web pages into rich media applications that use audio,
images and videos heavily in their interaction with users. This
change has made a dramatic change in network traffics
nowadays. Organizations spend a lot of effort, time and money
to improve response time and design intermediary systems that
enhance overall user experience. Media traffic represents
about 69.9-88.8% of all traffic. Therefore, enhancing networks
to accommodate this large traffic is a major trend. Content
Distribution Networks (CDNs) are now largely deployed for a
faster delivery of media. Redundancy and caching are also
implemented to decrease response time.
In this project, we are implementing a caching scheme for
media streaming in a proxy server. Unlike CDNs, which
require huge infrastructure, our caching proxy server will be
as simple as a piece of software that is portable and can be
installed in small as well as large scales. It may be deployed in
a university network, company’s private network or on ISPs
servers. This caching scheme, specially tailored for media
streaming, will reduce traffic and enhance network efficiency
in general.
Index Terms – Proxy servers, Caching, Media streaming
Hire some ii towards privacy-aware cross-cloud service composition for big da...ieeepondy
Hire some ii towards privacy-aware cross-cloud service composition for big data applications
+91-9994232214,8144199666, ieeeprojectchennai@gmail.com,
www.projectsieee.com, www.ieee-projects-chennai.com
IEEE PROJECTS 2015-2016
-----------------------------------
Contact:+91-9994232214,+91-8144199666
Email:ieeeprojectchennai@gmail.com
ieee projects chennai, ieee projects bangalore
GCX Cloud X Launch Presentation (October 14th, 2014)Ahmed Abdel-Latif
Global Cloud Xchange Launches Transformational New Cloud Ecosystem: Delivering applications on a virtualized infrastructure, operating across the world’s largest private sub-sea cable system, the new enterprise-class Cloud switching structure enables automated deployment of Cloud services –a market first.
This document discusses how broadcasters can start shifting their operations to the cloud. It outlines typical broadcast workflows and how aspects like storage, processing, and delivery can be moved using AWS and partner solutions. The challenges of bridging physical requirements to the virtual cloud are addressed through services like AWS Storage Gateway and AWS Direct Connect. Key trends highlighted include the need to manage increasing storage demands, scale media processing, and deliver content globally to meet rising consumer expectations across multiple devices. Case studies demonstrate how media companies are leveraging AWS to save costs while improving reliability and the ability to innovate.
Media and Entertainment Network Exchange ConceptJason Banks
Abstract— The Media and Entertainment industry is in the midst of rapid evolution as movie studios and broadcast television productions move away from traditional analog formats to digital. Rapid advancements in display resolution and file formats such as 4k, Ultra High Definition, and High Frame Rate filming are placing additional stress on I.T. systems and increasing the demand for storage and network bandwidth. While many media organizations are already leveraging Internet technologies in their day -to- day operations, this paper will present concepts to improve the media creation process, creative collaboration, efficiency, and security via traditional network technologies, as well as newer advancements in cloud computing and software defined networking.
Introducing Events and Stream Processing into Nationwide Building Societyconfluent
Watch this talk here: https://www.confluent.io/online-talks/introducing-events-and-stream-processing-nationwide-building-society
Open Banking regulations compel the UK’s largest banks, and building societies to enable their customers to share personal information with other regulated companies securely. As a result companies such as Nationwide Building Society are re-architecting their processes and infrastructure around customer needs to reduce the risk of losing relevance and the ability to innovate.
In this online talk, you will learn why, when facing Open Banking regulation and rapidly increasing transaction volumes, Nationwide decided to take load off their back-end systems through real-time streaming of data changes into Apache Kafka®. You will hear how Nationwide started their journey with Apache Kafka®, beginning with the initial use case of creating a real-time data cache using Change Data Capture, Confluent Platform and Microservices. Rob Jackson, Head of Application Architecture, will also cover how Confluent enabled Nationwide to build the stream processing backbone that is being used to re-engineer the entire banking experience including online banking, payment processing and mortgage applications.
View now to:
-Explore the technologies used by Nationwide to meet the challenges of Open Banking
-Understand how Nationwide is using KSQL and Kafka Streams Framework to join topics and process data.
-Learn how Confluent Platform can enable enterprises such as Nationwide to embrace the event streaming paradigm
-See a working demo of the Nationwide system and what happens when the underlying infrastructure breaks.
Cloud Aggregation: Smart Access to a Smarter CloudExponential_e
The noise in the market is all about placing data in the cloud – security,
compliance and risk. However, a good cloud strategy is underpinned with the network providing the access to cloud services and providers. How secure is the route to the cloud, is the infrastructure fit for purpose and is bandwidth scalable?
Cloud Aggregation: Smart Access to a Smarter Cloud
CDN_Netflix_analysis
1. Columbia University - Content Distribution Networks, Final Report, Fall 2015
Netflix Architechture Analysis
Saad Ahmed, Ralston Goes, Sanket Jain
ELEN E6776 Content Distribution Networks, Electrical Engineering
Columbia University
sa3205, sj2659, rrg2148 @columbia.edu
Abstract—Netflix, a subscription based video-streaming
service provider was founded in 1999. By 2007 it already had
outperformed all its competitors and it further started to
optimize and expand its services. Today Netflix alone accounts
for around one-third of the peak download traffic in North
America. Because of propriety issues, there are not many
analyses performed for Netflix’s traffic and CDN optimization.
In this paper, we perform an analysis of the deployed
architecture of Netflix as well as observe the traffic flow of
packets. Additionally, we observe how throttling the
Bandwidth changes the video quality and check if the CDN
redirection takes place. We then discuss the scalability issues
and further possibility for optimization of CDN selection and
video delivery strategy that Netflix might be looking forward
to deploying to deal with its ever-growing video traffic
problem.
I. INTRODUCTION
Netflix is one of the smartest companies of this decade. As
the author of book titled Netflixed says, "It has more drama
than most of the movies Netflix rents", the same stands for
the data distribution from content origin to the end user of
Netflix. One of the most commonly heard public case being
bottleneck of Netflix by various ISP like Comcast and
Verizon. In past decade Netflix demand increased almost
exponentially resulting in bigger issues like limited
bandwidth and fast lane competition. Reed Hastings the co-
founder and CEO of Netflix is investing heavily to improve
not only the contents but also on how they are distributed
through CDN. One subscriber of Netflix uses about 3 GB
per second for online streaming High Definition video and
about 1GB per hour for Standard Definition and much more
for Ultra High Definition. The leading subscription-video
streaming of Netflix accounted for 36% of the total
downstream Internet bandwidth during the peak traffic
hours last March in US. Just in past six months, Netflix
boosted its traffic share from 34% to 36% which is 2%
increase in just 6 months. As of March 2015, Netflix’s video
streaming service has gathered over 50 million subscribers
in 40 different countries especially the developed ones like:
America, Australia, Japan and many more in Europe. Such a
popular service has to solve newer engineering problems
like efficient streaming, on-demand content and a huge
number of requests causing heavy traffic. It becomes crucial
that Netflix takes these factors into account to maintain its
successful business model.
Two major factors contribute to Distribution of Content:
The diameter of fiber optic cable, which has huge capacity
and the infrastructure, provided by ISP. Obviously the
infrastructure of distributing content needs to be improved
and CDN is the main thing that needs that entire
infrastructure. As Reed Hastings says, “A few more shelves
of equipment might be needed in the buildings that house
interconnection points...”, I think it is time to rethink about
the current CDN which is not only used by Netflix but also
by various ISP. Hence we are presenting an analysis of
current CDN of Netflix so that further improvements in
infrastructure can be made to provide better Quality of
Experience to user.
Netflix has taken many steps like providing service up to
but not including last mile through better implementation of
CDN and cloud. CDN are content distribution networks
which use variety of techniques to send data from one center
to end or from end to center to another end. Netflix initially
outsourced streaming video delivery to three main vendors
namely Level 3, Akamai and Limelight. Now Netflix has its
own CDN, which accounts for more than 90% of data flow
in USA and 100% outside America. It has also resorted to
the use of cloud service, specifically Amazon Web Service
(AWS) adding another layer to its CDN providing an IT
infrastructure. Amazon Simple-DB, S3 and Cassandra are
used for file storage, video streaming is served out of
multiple CDNs, and a public DNS service called Ultra-DNS
is used as its authoritative DNS servers. Microsoft
Silverlight is the video playback platform for Netflix
desktop users. As of June 2012 Netflix invested in Open
Connect which is being used to cache Netflix content within
its ISP’s data center, bringing it close to subscriber except at
the last mile hence providing better Quality of Service.
Additionally, the company invested in NGINX, which is the
streaming media server that runs on every Open Connect
delivery appliance. Recently it made a collaborative project
with Google named Spinnaker, which is used for cloud
management (elastic cloud) and as a delivery engine. Netflix
is a very smart and growing company that uses advanced
2. Columbia University - Content Distribution Networks, Final Report, Fall 2015
techniques and basics like downloading video based on
recommendation algorithms in mid-night when users are
asleep.
In this paper we:
Study and analyze the classic architecture model used
by Netflix before 2012 that is from 2008-2012.
Discuss current architecture involving Open Connect
that was implemented in 2014.
Perform graphic analysis of packet flow from client to
host. When and whom does the client contact - CDN,
Amazon cloud or other options?
Packet analysis over TCP and how encryption key is
handled. Which cloud is responsible for which user
and/or content?
DASH protocol, how does low bandwidth high
congestion is handled by DASH for Netflix.
Variation in packet loss during download and of the
ratio of packets lost to packets received.
II. NETFLIX ARCHITECHTURE
A. Overview
As the Netflix traffic increased at such a rapid pace, it had to
combine multiple third party services in addition to its own
platform and come up with a cloud based video streaming
architecture, the one which allowed Netflix to acknowledge
itself as a scalable, infrastructure-less content provider.
Introduction to cloud based service was necessary since the
local servers were eventually running out of space and also
with the increase in traffic, bandwidth available per user
was becoming the bottleneck in the video streaming
performance.
Netflix moved to Software as Service (SaaS), a distribution
model in which applications are hosted on a remote server
to be accessed over from anywhere. It also established and
built its own (PaaS) to develop Big Data Tools, HA Patterns
and deploys its application and content over virtualized
servers and operating systems. As far as the storage issues
are concerned, it moved incremental capacity to (IaaS).
Therefore, since 2008 they did not require to build any data
center space and all the content and streaming app was
moved to cloud. For the storage, AWS cloud services were
adopted.
B. Amazon Cloud services
The following goals were achieved with Netflix going 100%
on cloud: There was lower latency than the equivalent
datacenter web pages and API calls. This was true for both,
the first hit to home page as well as in-session hits.
Scalability was revived as there were no more datacenters
needed to accommodate the incoming new subscribers. It
not only enabled substantially higher robustness and
availability of content than datacenter services but also
optimized agility of a large development team with
automation and tools leaving behind the complex tangled
datacenter code base. In order to modify the website, there
is no scheduled down time since there is no central schema
to change. Clean-layered interfaces and re-usable
components were enforced. Figure 1 shows the constituents
and distribution of deployed Netflix services on AWS
Cloud.
Figure 1: The constituents on distribution of deployed Netflix Services as
on Amazon Web Services Cloud.
C. Key Components and Protocols
This lead to 4 key components in the architecture: 1) Netflix
center, 2) Amazon AWS cloud, 3) Third party CDNs and 4)
players. Figure 2 displays the architecture representation.
To observe the basic service behavior, we create an account
and login into the Netflix main website. We monitor the
traffic while performing all this activity. Once we get the IP
addresses of the destination servers we then perform DNS
resolutions to collect the canonical names of the servers that
browser contacted. Figure 2 shows basic architecture of
video streaming platform for Netflix.
As observed from traffic analysis, it is seen that Netflix uses
its own IP address space for the host name www.netflix.com
[3]. The registration of new user accounts and all billing
related matters are either redirected to its own host name or
Amazon. When we start streaming the movie the Netflix
server does not interact again with the client.
Once we are on Netflix video selection page, or if we are
streaming a movie, we are served by Amazon AWS cloud.
It uses many AWS zones for high availability and
scalability. For our analysis, all the website traffic including
the video streaming was directed to host [ec2-184-72-246-
221.compute-1.amazonaws.com].
3. Columbia University - Content Distribution Networks, Final Report, Fall 2015
Figure 2: Architecture representation of Netflix
D. Content Distribution Networks:
Netflix employs multiple third party CDNs to deliver the
video content to end-users. All the encoded and protected
video content is sourced out to Amazon cloud, which then
sends multiple copies to different CDNs. As shown in fig. 2,
initially, just three CDNs were employed: Akamai,
Limelight and Level-3. However, Netflix is further
optimizing its architecture by deploying its own specialized
content delivery and caching system ‘Open Connect
Appliances’. We will discuss this in detail in next section.
Netflix uses Silverlight to download and decode chunks on
Desktop [2]. Players are also available for mobile/Ipad etc.
For streaming in volatile Bandwidth environment, Netflix
uses DASH (Dynamic Streaming over HTTP) protocol.
DASH is a video streaming protocol that allows player to
freely switch between video quality levels. It is totally
different from UDP-based streaming and involves the
adaption of both by the application and by the TCP.
Netflix’s adaption seems to default to TCP control during
heavy network congestions or even during periods of
volatile network conditions [3]. For every packet download,
the protocol measures the received bandwidth and runs a
rate-resolving algorithm to control the quality of next chunk.
E. Architecture Improvement: Open Connect
In June 2012, Netflix articulated its own specialized content
delivery system called Open Connect. Open Connect
Appliances (OCAs) are provided at no charge to Netflix’s
partner ISPs. Since this content delivery system is created
by Netflix for its own use, it is very efficient in content
delivery and performs much better than relying on proxy
caches. It uses the data of its known set of users and its
known set of content to result in a simplified content
delivery. It uses the ISPs ability to gather routing
information via BGP protocol and combines it with Netflix
content routing to create a robust and extensible layer 3-7
routing engine. Layer 7 load balancing operates at the high-
level application layer and deals with the actual contents of
the message. It terminates the network traffic and reads the
content of the message to make better load balancing
decisions. The benefits of layer 7 includes the ability to
apply optimizations and changes to content as well as use
buffering to offload slow connections from upstream servers
thus improving the overall system’s performance.
Therefore, the owning server and client logic allows for
optimization of the streaming performance. Open Connect
also uses proactive caching. Proactive caching allows for
users to be able to see the most recent data while the source
data and aggregations are stored in a multidimensional
structure to improve performance. The old cache can be
invalidated and a new one can be built with the most recent
data from the source system at any set schedule. In Netflix,
this is taken as an advantage by pre-populating content
within ISP networks at off-peak times. This dramatically
reduces upstream network utilization at peak times by 75-
100%. This also reduces the load on ISPs and thus
eliminates the necessity to scale, transport or set up internal
links for Netflix traffic. Another advantage of using
proactive caching is that it allows for central popularity
calculation, which is more reliable and accurate than a
cache, or proxy that determines popularity based on the
requests it receives. Also, the OCAs are simple web servers
that are 100% utilized for throughput and they know how to
sort files and report on health.
Figure 3: Open Connect Appliance
The content flow in OCA can be described with reference to
the above diagram. The main components we consider is the
OCA, with manifest service, the OCA Control Service, the
Client Control Service and a Popularity service. The OCA
registers itself with the OCA control service. The OCA
control service delivers the content manifest to the OCA.
The manifest service on OCA then compares the received
manifest from the OCA control service with the manifest
that’s on the disk. If it finds new content that it requires, the
manifest service creates a script to fetch the new content
from the origin. The manifest service also deletes content
that is no longer in use.
4. Columbia University - Content Distribution Networks, Final Report, Fall 2015
Figure 4: Netflix OCA-CDN Implementation
The OCA then submits a new manifest to the OCA control
service (OCA CS). The OCA CS in turn communicates with
a Client Control Service, providing the definition of the
OCA as well as the content to cache mapping. The OCA CS
is also fed by a Popularity service, which tells it about the
ordered list of popular downloadable URLs and the
downloadable file sizes. As any control service, the OCA
CS plays a critical role here as well in maintaining and
updating the OCA.
To obtain the best service, the client needs to know its
closest OCA for video stream delivery. It first contacts the
Netflix control servers to obtain the address of the local
Netflix cache.
III. TRAFFIC ANALYSIS
A. DNS and Packet Routing
Once we go to the Netflix.com and select any movie title to
play, the DNS resolves which CDN to contact by first
contacting the Netflix server based on ec2. As we can see
below in figure 5, standard query is first made to host
Netflix.com. Once we receive the response back with the
information about what CDN to connect to, standard query
for connection is sent to that particular CDN base. Once we
get the response, the connection is established.
Figure 5: DNS Resolution
Also, Netflix assigns a fixed CDN to a user the moment he
subscribes for the service. This CDN remains the same for
that particular user regardless of its location, IP address,
computer, traffic condition etc. For our account, our traffic
was all redirected to Akamai CDN from Amazon ec2 cloud,
no matter from where and when we accessed the site.
Figure 6: IP Addresses showing traversed traffic
B. Traffic Analysis on Wireshark
As we go on Netflix.com with our account logged in, Hello
message is sent to host (54.243.242.133 – Amazon ec2) for
handshake over TLSv1.2. Server sends back the HTTP text
indicating that it accepted the connection and this
confirmation is received via TLSv1.2. Then server and
client exchange the keys and change cypher specifications
of the file as seen in figure 7. Once the encryption
handshake messages are exchanged, the host is ready to
send the requested application data over TCP. All the
packets are requested and sent over a TCP protocol. All
frames sent to and received from host are of length 66 bytes
and 1454 bytes respectively
Figure 7: Wireshark Analysis
As displayed, the encryption keys are exchanged over
TLSv1.2.
Figure 8: Encryption handshake message header
C. TCP Packet Flow
Once we choose to stream a video content, it is observed
that the destination IP changes from 54.243.242.133 to
5. Columbia University - Content Distribution Networks, Final Report, Fall 2015
208.54.37.30. We can say that the requests for the content
have now been directed to Akamai CDN closest to our ISP.
Figure 9: TCP Packets Flow
CDN host then sends the video content in packets with
varying frame window size, depending on the bandwidth
limitation. Frame length of 1454 bytes is consistent for all
the received video chunks. Client requests the packets
through sequence numbers and acknowledgements for the
received packets are also piggybacked along with the next
request. The figure below shows the transmission flow of
few segments of TCP packets:
Figure 10: Transmission of Packets
We also analyzed the packet flow per second for one of the
streamed movies for several minutes. There were several
packet losses and retransmission observed. Figure 11 is the
graph obtained. Bars in blue represent the packet losses. As
we can see the packets received per second is the highest as
soon as we started streaming the video. It is mainly because
the buffer is empty and extra packets are received to fill it.
Figure 11: Packets received at user end per second.
Next, we throttled the incoming Bandwidth to observe the
following:
1) Does the Netflix allow switching of CDN to keep up
with the streaming video quality
2) Observe the DASH protocol by noticing the video
quality adaption with variation in Bandwidth
3) Notice the change in window size and variation in
packet losses
The analysis performed by [2] shows that once the available
bandwidth for the linked CDN decreases, the Netflix
architecture does not switch the client to some other CDN.
Instead, it uses DASH protocol to lower the available
quality to adjust it to the available bit rate. Only when it is
no longer able to support even the lowest quality (usually at
100kb/s), it directs client to some other CDN that is
providing better performance and lower latency.
Since we cannot throttle the bandwidth at the CDN end, in
order to notice the switching of clients connection to
different CDN under traffic congestion, we will virtually
create such an environment by varying limiting the
bandwidth at our end to have a basic idea. We have used
‘Network Link Conditioner’ to throttle the inbound
bandwidth to the client.
We have investigated and analyzed the instantaneous
bandwidth variation during several minutes of video
playing. This is important as DASH player constantly
monitors the available bandwidth to determine the level of
video quality to stream. The small time scale bandwidth can
significantly affect the viewer’s experience. Figure 12, 13
and 14 shows the packets received per second at different
Bandwidths: 0.1MB/s, 0.2 MB/s, 1.5 MB/s and 10 MB/s.
When the Bandwidth was limited to 100kbps, Netflix video
failed to load and our result is consistent with the analysis
performed in [2]. Once we increase the bandwidth to
200kbps, the video content takes a while to load/buffer and
streams with a poor quality. As we can see from graph of
figure 12, the number of packets received per second is at
6. Columbia University - Content Distribution Networks, Final Report, Fall 2015
maximum, 150, in the start. Also, the number of packets lost
is much higher.
Figure 12: Packets received and lost per second at bandwidth of 200 kbps
As we increase the Bandwidth to 1.5 MB/s, we observe a
little better video quality (medium quality) and as seen from
the graph the number of video chunks received per second
increases to ~280 packets per second. The packet loss is also
decreased, hence resulting in fewer pauses due to buffering.
Figure 13: Packets received and lost per second at bandwidth of 1.5 Mbps
Finally we increase the Bandwidth to 10 MB/s and observe
a tremendous improvement in the performance of video
streaming. Not only does the video quality become HD, but
also the ratio of packets received to packet lost increases
tremendously. The maximum number of packets received in
the start of playback is 1500 per seconds. The latency time
is also negligible.
Figure 14: Packets received and lost per second at bandwidth of 10 Mbps
IV. CONCLUSION
For this project, we analyzed basic as well as improvised
Netflix’s video-streaming services and CDN modeling to
uncover the overall architecture of Netflix, which is
currently by far, the leading video-streaming subscription
based platform. We have observed the routing of TCP video
chunks and overall traffic flow with in its architecture. We
have also evaluated the performance of Netflix’s current
architecture by performing traffic measurements under
bandwidth variation. We have also observed that CDN for
each user is fixed for each subscriber and does not change
regardless of location, time or traffic congestions in CDN’s
networks.
Netflix is already planning to introduce Ultra High
definition movies. Due to the size of UHD files, the traffic
at the CDNs and the network load would increase
significantly resulting in throttling the available Bandwidth
and video bit rate for each user. In order for Netflix’s to
keep up with this new feature while further improving on
scalability and better user appearance, it needs to make its
CDN assignment strategies more elastic and dynamic.
Netflix also has the capacity to improve recommendation
algorithm to ensure that the recommendation titles for a
particular user are being cached pro-actively at user’s
closest CDN or caching platform.
ACKNOWLEDGMENT
WE WOULD LIKE TO EXPRESS OUR GRATITUDE TO AND
THANK PROFESSOR ANWAR WALID FOR SHARING HIS VITAL
KNOWLEDGE TO HELP US GRASP AND UNDERSTAND THE
ESSENTIALS OF CDNS AND RELATED TOPICS
REFERENCES
[1] http://people.cs.clemson.edu/~jmarty/projects/DASH/C
CNC2013SUBMISSION.pdf
[2] Unreeling Netflix: Understanding and Improving Multi-
CDN Movie Delivery. By Adhikari, Vijay Kumar
[3] http://people.cs.clemson.edu/~jmarty/projects/DASH/C
CNC2013SUBMISSION.pdf
[4] Slides on Netflix CDN:
http://www.slideshare.net/adrianco/netflix-global-cloud
[5] Peers of Netflix in New York:
https://openconnect.netflix.com/peeringLocations/
[6] Spinnaker
http://googlecloudplatform.blogspot.com/2015/11/Netfl
ixs-Spinnaker-available-now-on-Google-Cloud-
Platform.html
[7] http://techblog.netflix.com/2015/11/global-continuous-
delivery-with.html
[8] New optimization Netflix has come up with to reduce
bit rate (they started testing this only last week or so):