Hyperscale? Don't Try This at Home - Jesse Proudman - OpenStack Day Israel 2016

•

0 likes•448 views

The document discusses the concept of hyperscale computing. It describes hyperscale as unthinkably large data centers that are entirely automated and use commodity hardware and cloud native technologies. It contrasts the standardized, white box hardware approach of public clouds with the more proprietary approach of traditional enterprise data centers. It asks how enterprises can blend hyperscale lessons and discusses focusing on consistency, automation, and network effects to achieve hyperscale capabilities.

Hyperscale?
Don’t try this at home…
@blueboxjesse
jesse proudman
CTO & IBM Distinguished Engineer
j.proudman@us.ibm.com

HYPERSCALE
Unthinkably Large
Entirely Automated Commodity Everything
Cloud Native
@blueboxjesse

@blueboxjesse
https://ibm.biz/BdHsa9
https://ibm.biz/BdHsa9
https://ibm.biz/BdHsa9
https://ibm.biz/BdHsa9

The Future
is Software
Defined
@blueboxjesse

That SOFTWARE
runs on
HARDWARE
@blueboxjesse

@blueboxjesse https://www.stickermule.com/marketplace/3442-there-is-no-cloud

Reality of Public Clouds
§ Standardized Hardware
§ White Box / ODM
§ “Sick Servers” left for dead
§ High Availability via Software
@blueboxjesse

Reality of Enterprise Data Centers
@blueboxjesse
§ “One of everything”
§ Proprietary and unique hardware
§ “Sick Server? Call a medic!”
§ High Availability via Hardware

How do we
Blend Hyperscale
Lessons?
@blueboxjesse

“It’s hard to out innovate
the world…”
@blueboxjesse

Where do you want to
invest your time?
@blueboxjesse

Pick your poison
DistroDIY PCaaS
@blueboxjesse

Consistency
Consistency
Consistency
@blueboxjesse

YOU CAN DO IT!
@blueboxjesse
jesse proudman
CTO & IBM Distinguished Engineer
j.proudman@us.ibm.com

This document discusses virtual customer premises equipment (vCPE) as a top network functions virtualization (NFV) and software-defined networking (SDN) use case for network operators. It analyzes different deployment models for vCPE, including distributed, on the aggregation switch/provider edge router, and centralized. Centralized vCPE places virtualized customer equipment functions on network infrastructure virtualization in the edge point of presence. This allows simplified management and flexibility through SDN control. While each model has advantages, a hybrid approach combining centralized and distributed vCPE may be a new trend that operators are pursuing.

SDN Network virtualization, NFV & MPLS synergies

Hector.Avalos

Running a vCPE using OpenStack, OpenDaylight and SFC

Ettore Simone

Mellanox Approach to NFV & SDN

Mellanox Technologies

Presented by Eran Bello at the "NFV & SDN Summit" held March 2014 in Paris, France Ideal for Cloud DataCenter, Data Processing Platforms and Network Functions Virtualization Leading SerDes Technology: High Bandwidth – Advanced Process 10/40/56Gb VPI with PCIe 3.0 Interface 10/40/56Gb High Bandwidth Switch: 36 ports of 10/40/56Gb or 64 ports of 10Gb RDMA/RoCE technology: Ultra Low Latency Data Transfer Software Defined Networking: SDN Switch and Control End to End Solution Cloud Management: OpenStack integration Paving the way to 100Gb/s Interconnect End to End Network Interconnect for Compute/Processing and Switching Software Defined Networking High Bandwidth, Low Latency and Lower TCO: $/Port/Gb

A Tale of Two OpenStack Contributors: A Newbie Developer and a Frustrated Ope...

Cloud Native Day Tel Aviv

OpenStack is a community. It takes a community to develop such a massive ecosystem, but the sheer size of it also presents a barrier. This session will present two aspects of OpenStack Contributions. From a Developer's perspective you will learn how to make your first OpenStack contribution, from your creating your account to identifying low hanging fruits, or even trying to push a new project to the ecosystem. From the Operator's perspective, we have not yet reached the pot of gold at the end of the rainbow. Operators are still not yet accepted as equal members of the OpenStack community. We will go over some of the history, current issues, what is considered a contribution to OpenStack, and how this should, and can change.

OpenStack & OVS: From Love-Hate Relationship to Match Made in Heaven - Erez C...

Cloud Native Day Tel Aviv

"Many developers building OpenStack clouds have “love-hate” relationship with OVS. They love flexibility and elasticity offered by OVS, but hate the network performance and scalability. As emerging technologies such as NFV keep pushing for higher network performance, it becomes critical to improve OVS performance without compromising flexibility, network programmability, and cost. In this session, we will present an approach that Mellanox has devised with input from key partners and customers to accelerate Virtual Switch dataplane, using the embedded switch implemented in the server Network Interface Card (NIC)’s hardware. This approach supports both ParaVirt vNIC interfaces and SRIOV based vNICs interfaces"

vCPE 2.0 – the business case for an open vCPE framework

Cloudify Community

Software developers are screwing up the digital world. Security is often an afterthought, or worse, the job of I.T., who is expected to sprinkle magic fairy dust on an app that magically makes it secure. That's an impossible ask and forces a perimeter-based security model that cannot succeed alone in a world of cloud apps, mobile devices, and distributed data. Developers must embrace security by design principles and fundamentally shift their attitude about who is responsible for security.

Codemotion Milan 2018 - AI with a devops mindset: experimentation, sharing an...

Thiago de Faria

AI is the buzzword while ML is the underlying component... but when do we use ML? To solve problems that machines can find patterns without explicitly programming them to do so. But do you have a team building an ML model? How far are they from the IT team? Do they know how to deploy and serve that? Testing? And sharing what they have done? That's where a devops mindset comes in: reduce the batch size, continuous-everything and a culture of failure/experimentation are vital for your data team! In the end, I will show how the workflow of a data scientist can be in real life with a live demo!

Thiago de Faria - AI with a devops mindset - experimentation, sharing and eas...

Codemotion

Hybrid is the Road to a Fast & Furious OpenStack Public Cloud Future

Angel Diaz

Going Cloud Native

David Schmitz

Codemotion Berlin 2018 - AI with a devops mindset: experimentation, sharing a...

Thiago de Faria

Thiago de Faria - AI with a devops mindset - experimentation, sharing and eas...

Codemotion

The document discusses applying a DevOps mindset to machine learning. It notes traditional problems in ML like data scientists not being engineers, lack of versioning and scaling issues. Applying continuous integration/deployment practices from DevOps can help address these by facilitating collaboration, versioning of code and data, continuous evaluation, packaging, deployment and model serving. This requires culture change and bridging the gap between data scientists and engineers. DevOps practices may help achieve the collaboration, automation and monitoring needed between ML development and operations.

OpenStack Summit 2015: Tokyo - IBM Sponsor Keynote

Jesse Proudman

Introducing Backblaze B2, the lowest cost cloud storage on the planet.

Backblaze

OSMC 2021 | Pipeline your Dashboards as Code

NETWAYS

Pipeline all the Dashboards as Code

Kris Buytaert

Kris Buytaert discusses the problems with dashboard sprawl in Grafana and proposes managing dashboards as code. He introduces Dashboard Manager, a tool for fetching dashboards from a development Grafana instance, comparing them, and uploading them to a production instance. This allows end users to create dashboards in development and then promote them to production in a version controlled and reproducible manner.

Belgium jenkins-meetup-job-jungle-0.1

Damien Coraboeuf

This document discusses how to scale Jenkins jobs as the number of jobs grows very large. It describes moving from individual Jenkins jobs to defining pipelines as code. The key steps are: 1) defining pipelines using a Job DSL plugin; 2) refactoring duplicated DSL code into reusable libraries; and 3) configuring pipelines through property files instead of code. This allows generating and securing pipelines for multiple projects through a seed job that is triggered by source code management hooks. The end result is "automation of automation" through self-service, secure, simple and extensible pipeline management.

Testing in Production: From DevTestOops to DevTestOps

Alex Soto

DevOps has grown in popularity in recent years, particularly in software companies that want to reduce lead time to be measured in days/weeks instead of months/years. If you want to make sure your software does the right things and does things right, you need to test it implacably. The big problem is that companies see the testing phase as the bottleneck of the process slowing down product release. To change that, we need a new way of testing our applications, making the release process of an application a testing process as well and involving the QA from the beginning within the team. QAs are not a separate team anymore: DevTestOps. How is the testing pyramid affected? In this session, we will not only describe but actively demonstrate several techniques that you can use immediately following the session for starting testing in production and speeding up your release cycle to unimaginable limits. Demo1: https://www.youtube.com/watch?v=3mJX38m-D3A Demo2: https://www.youtube.com/watch?v=IcRsb-boWA4 Demo3: https://youtu.be/F15ZeZuqo54

Getting out of the Job Jungle with Jenkins

Sonatype

Damien Corabouef, Multipharma, Clear2Pay Implementing a CI/CD solution based on Jenkins has become very easy. Dealing with multiple feature, staging and release branches? Not so much. Having to handle that for multiple teams and multiple projects becomes a real challenge. This presentation shows a solution to scale to several thousands of jobs, used by dozens of different development and test teams, 24 hours a day, 7 days a week, on a worldwide schedule. I will talk about the challenges that we’ve met, and how we’ve put in place a scalable and on-demand solution, secure and simple to use. This is a real-life, real-scale story of making CI/CD a day-to-day reality by allowing development and test teams to consider automation as a simple and customisable service.

Brujug Jenkins pipeline scalability

Damien Coraboeuf

Where Do We Go From Here?

Matthew Broberg

Our industry as at a saturation point of buzzwords and it's mucky for those on the job hunt going forward. I'd like to take some time to preach how Open Source's history might be an opportunity to connect into a better future for us all and then map that to technical and career trends happening around us. Sysadmin skills might just be more valuable in the future than we realize. This talk also takes the theme from my favorite talk by John Rauser with his permission.

Happy Browser, Happy User! NY Web Performance Meetup 9/20/19

Katie Sylor-Miller

xPerformance is fundamentally, a UX concern. Sites that are slow to render or janky to interact with are a bad user experience. We strive to write performant code for our users, but users don’t directly interact with our code - it all happens through the medium of the browser. The browser is the middleman between us and our users; therefore to make our users happy, we first have to make the browser happy. But how exactly do we do that? In this talk, we’ll learn how browsers work under the hood: how they request, construct, and render a website. At each step along the way, we’ll cover what we can do as developers to make the browser’s job easier, and why those best practices work. You’ll leave with a solid understanding of how to write code that works *with* the browser, not against it, and ultimately improves your users’ experience.

Cloud Native is a Cultural Decision. By Reshef Mann

Cloud Native Day Tel Aviv

This document discusses how adopting a cloud native approach is a cultural decision for companies. It emphasizes that engineering organizations have their own cultures defined by a small set of principles and questions. These include values like quality, rapid deployment, accountability, and stability. While there are no silver bullets, identifying core invariants and defining cultural values can help guide a company towards being customer-obsessed and proud of their work.

Container Runtime Security with Falco, by Néstor Salceda

Cloud Native Day Tel Aviv

In any Cloud Native architecture there’s a seemingly endless stream of events that happen at each layer. These events can be used to detect abnormal activity and possible security incidents, as well as providing an audit trail of activity. In this talk we’ll cover how we extended Falco to ingest events beyond just host system calls, such as Kubernetes audit events or even application level events. We will also show how to create Falco rules to detect behaviors in these new event streams. We show how we implemented Kubernetes audit events in Falco, and how to configure the event stream.

Similar to Hyperscale? Don't Try This at Home - Jesse Proudman - OpenStack Day Israel 2016

Hyper Scale, You Are Not - OpenStack Tokyo 2015

Jesse Proudman

Distros are Dead, The Future of OpenStack, Jesse Proudman, Blue Box

Sriram Subramanian

Distros are Dead, The Future of OpenStack

Jesse Proudman

Stop expecting magic fairy dust: Make apps secure by design

Patrick Walsh

Codemotion Milan 2018 - AI with a devops mindset: experimentation, sharing an...

Thiago de Faria

Thiago de Faria - AI with a devops mindset - experimentation, sharing and eas...

Codemotion

Hybrid is the Road to a Fast & Furious OpenStack Public Cloud Future

Angel Diaz

Going Cloud Native

David Schmitz

Codemotion Berlin 2018 - AI with a devops mindset: experimentation, sharing a...

Thiago de Faria

Thiago de Faria - AI with a devops mindset - experimentation, sharing and eas...

Codemotion

OpenStack Summit 2015: Tokyo - IBM Sponsor Keynote

Jesse Proudman

Introducing Backblaze B2, the lowest cost cloud storage on the planet.

Backblaze

OSMC 2021 | Pipeline your Dashboards as Code

NETWAYS

Pipeline all the Dashboards as Code

Kris Buytaert

Belgium jenkins-meetup-job-jungle-0.1

Damien Coraboeuf

Testing in Production: From DevTestOops to DevTestOps

Alex Soto

Getting out of the Job Jungle with Jenkins

Sonatype

Brujug Jenkins pipeline scalability

Damien Coraboeuf

Where Do We Go From Here?

Matthew Broberg

Happy Browser, Happy User! NY Web Performance Meetup 9/20/19

Katie Sylor-Miller

Similar to Hyperscale? Don't Try This at Home - Jesse Proudman - OpenStack Day Israel 2016 (20)

Hyper Scale, You Are Not - OpenStack Tokyo 2015

Distros are Dead, The Future of OpenStack, Jesse Proudman, Blue Box

Distros are Dead, The Future of OpenStack

Stop expecting magic fairy dust: Make apps secure by design

Codemotion Milan 2018 - AI with a devops mindset: experimentation, sharing an...

Thiago de Faria - AI with a devops mindset - experimentation, sharing and eas...

Hybrid is the Road to a Fast & Furious OpenStack Public Cloud Future

Going Cloud Native

Codemotion Berlin 2018 - AI with a devops mindset: experimentation, sharing a...

Thiago de Faria - AI with a devops mindset - experimentation, sharing and eas...

OpenStack Summit 2015: Tokyo - IBM Sponsor Keynote

Introducing Backblaze B2, the lowest cost cloud storage on the planet.

OSMC 2021 | Pipeline your Dashboards as Code

Pipeline all the Dashboards as Code

Belgium jenkins-meetup-job-jungle-0.1

Testing in Production: From DevTestOops to DevTestOps

Getting out of the Job Jungle with Jenkins

Brujug Jenkins pipeline scalability

Where Do We Go From Here?

Happy Browser, Happy User! NY Web Performance Meetup 9/20/19

More from Cloud Native Day Tel Aviv

Cloud Native is a Cultural Decision. By Reshef Mann

Cloud Native Day Tel Aviv

Container Runtime Security with Falco, by Néstor Salceda

Cloud Native Day Tel Aviv

Kafka Mirror Tester: Go and Kubernetes Powered Test Suite for Kafka Replicati...

Cloud Native Day Tel Aviv

Running I/O intensive workloads on Kubernetes, by Nati Shalom

Cloud Native Day Tel Aviv

WTF Do We Need a Service Mesh? By Anton Weiss.

Cloud Native Day Tel Aviv

Service meshes are all the buzz in cloud-native world. How come only yesterday we didn't know such a thing existed and now everybody seems to want one? If you're already running a microservice-based system or only starting out with one, you may be asking yourself : "Do I also need a mesh?" In this session we'll try to answer what the mesh is good for, what problem it solves, what new questions it poses.

Update Strategies for the Edge, by Kat Cosgrove

Cloud Native Day Tel Aviv

Building a Cloud-Native SaaS Product The Hard Way. By Arthur Berezin

Cloud Native Day Tel Aviv

The Four Questions (Every Monitoring Engineer gets asked), by Leon Adato

Cloud Native Day Tel Aviv

If you are interested in monitoring, and successfully set up a system (whether home-grown or custom-off-the-shelf) for your own use, there comes a moment when you go from monitoring only the systems you care about, to monitoring systems that other people care about. Monitoring for yourself is all about having the best data for the least effort. Monitoring for others? That's when your job becomes a game of "what just happened" whack-a-mole.

K8s Pod Scheduling - Deep Dive. By Tsahi Duek.

Cloud Native Day Tel Aviv

Cloud Native: The Cattle, the Pets, and the Germs, by Avishai Ish-Shalom

Cloud Native Day Tel Aviv

MySQL Shell: the daily tool for devs and admins. By Vittorio Cioe.

Cloud Native Day Tel Aviv

MySQL shell is the MySQL client of the future. It will help you in your daily operations, whatever they are. It doesn't matter if you are a developer or an administrator, if you want to work with relational or non relational data, if you want to setup or monitor your cluster, if you want to work with SQL language or javascript or python. Discover how MySQL shell will help you, no matter what you want to do with MySQL!

Cloud native transformation patterns, by Pini Reznik

Cloud Native Day Tel Aviv

Cloud and Edge: price, performance and privacy considerations in IOT, by Tsvi...

Cloud Native Day Tel Aviv

Two Years, Zero servers: Lessons learned from running a startup 100% on Serve...

Cloud Native Day Tel Aviv

Running Highly Available Large Scale Systems is a lot of work. For the past 2.5 years, we've been running 100% serverless on a full production environment, serving customers worldwide. No VMs, no containers, no Kubernetes. Just code. In this session I will present why we decided to go fully serverless at Torii, how it helped us move faster than our competitors, where did serverless computing worked best and where there's more work to be done.

12 Factor Serverless Applications - Mike Morain, AWS - Cloud Native Day Tel A...

Cloud Native Day Tel Aviv

The “Twelve-Factor” application model has come to represent twelve best practices for building modern, cloud-native applications. With guidance on things like configuration, deployment, runtime, and multiple service communication, the Twelve-Factor model prescribes best practices that apply to everything from web applications to APIs to data processing applications. Although Serverless computing and AWS Lambda have changed how application development is done, the “Twelve-Factor” best practices remain relevant and applicable in a Serverless world. In this talk, we’ll apply the “Twelve-Factor” model to Serverless application development with AWS Lambda and Amazon API Gateway and show you how these services enable you to build scalable, low cost, and low administration applications.

Not my problem! Delegating responsibilities to the infrastructure - Yshay Yaa...

Cloud Native Day Tel Aviv

When creating a new Microservice you typically need to add a lot of boilerplate to the code, such as logging, metrics, authentication, SSL, secrets/credentials, etc... All this ends up overshadowing the actual logic of the service itself and results in a lot of dependencies and code. Because of this, we at Soluto created a template for the boilerplate, which worked great... Until we had to upgrade one of the dependencies which resulted in huge effort of code upgrades and deployment across each and every service that uses the template. And that’s besides the fact we needed to make a separate template for each language we used (and over the years, the number of programming languages we use at production increased dramatically). We needed a way to simplify all this. We wanted to get rid of the boilerplate while maintaining the functions listed above. So we decided to delegate the entire responsibility to the Kubernetes infrastructure. In this lecture, I will do a live coding session and show you how to remove the boilerplate from your code and move it to the infrastructure, and discuss the benefits and limitations of this approach.

Brain in the Cloud: Machine Learning on OpenStack & Kubernetes Done Right - E...

Cloud Native Day Tel Aviv

Machine Learning is no doubt the hottest trend in IT nowadays. Deep Neural Network (DNN), a subfield of Machine Learning with mode of operation loosely inspired by the brain, allows us to solve complex problems such as image recognition that has been very difficult to solve using standard programming paradigms. DNN concepts are not new. However, and until recently, applying them in practice could not be realized due to their high computational demands. With the recent development in parallel computing, especially around GPU acceleration and high speed and efficient networking, DNN has become a reality in modern data centers. In this talk we will describe the system requirements to effectively run a machine learning cluster with popular frameworks such as TensorFlow. We will discuss how such a system can be deployed in an OpenStack-based cloud without compromises, enjoying high-performance DNN programming paradigm as well as the benefits of cloud and software-defined data centers.

A stateful application walks into a Kubernetes bar - Arthur Berezin, JovianX ...

Cloud Native Day Tel Aviv

Cloud native applications are commonly thought as stateless, horizontally scalable workloads that you can scale-up and down on-demand. Kubernetes, as the commodity cloud native orchestrator, was originally designed for such workloads. A lot has evolved since Kubernetes’ inception, and nowadays many of the stateful applications are migrating to Kubernetes. While not everything is perfect, more and more features are added to support complex stateful use-cases. In this session Arthur will cover the following topics: - Breakdown of a stateful application - Planning a stateful application on Kubernetes - The state of Kubernetes StatefulSets, Persistent Volumes, DNS, Networking, operators and High Availability - A practical use-case and DEMO of a stateful application with Kubernetes

The story of how KubeMQ was born - Oz Golan, KubeMQ - Cloud Native Day Tel Av...

Cloud Native Day Tel Aviv

KubeMQ started as an internal project at a FinTech company to upgrade their infrastructure from monolithic to distributed services with zero downtime using containers. They developed KubeMQ as an open source message broker for containers since no existing MQ could meet their requirements. KubeMQ integrates with Kubernetes and other CNCF projects to provide a small footprint broker with high throughput, observability, security, and support for multiple languages and messaging patterns to make it easy to deploy and scale distributed services.

I want it all: go hybrid - Orit Yaron, Outbrain - Cloud Native Day Tel Aviv 2018

Cloud Native Day Tel Aviv

More from Cloud Native Day Tel Aviv (20)