DevOps Days Boston 2017
Microservices is an increasingly popular approach to building cloud-native applications. Dozens of new technologies that streamline adopting microservices development such as Docker, Kubernetes, and Envoy have been released over the past few years. But how do you actually use these technologies together to develop, deploy, and run microservices?
In this presentation, we’ll cover the nuances of deploying containerized applications on Kubernetes, including creating a Kubernetes manifest, debugging and logging, and how to build an automated continuous deployment pipeline. Then, we’ll do a brief tour of some of the advanced concepts related to microservices, including service mesh, canary deployments, resilience, and security.
KubeCon NA 2017: Ambassador and Envoy (Envoy Salon)Ambassador Labs
Ambassador is an open source Kubernetes-native API Gateway built on the Envoy proxy. We talked about why and how we built Ambassador during the Envoy salon at KubeCon.
DevOps Days Boston 2017: Developer first workflows for KubernetesAmbassador Labs
Kubernetes is a powerful, operational platform for containerized applications. However, the developer workflow on Kubernetes – how you code, deploy, update, and monitor your services – is much less mature.
How should you lay out your Git repo? How do you create loosely coupled services? How do you support deploying your service at any time?
In this talk, we’ll talk about these questions and more. We’ll discuss the journey towards a rapid development workflow, discuss best practices, and, talk about the process we followed to get to a rapid development workflow.
DevOps Days Boston 2017
Back to the Future: Containerize Legacy ApplicationsDocker, Inc.
People typically think of Docker for microservices and try to make the smallest container they can. There are tremendous benefits to a microservices model but those are not the only apps that qualify for containers. Traditional, homegrown, monolithic apps are also great candidates for Docker - why? By containerizing these apps, many of the same agility, portability, security and cost savings benefits can be applied to the hundreds (if not thousands) of apps in your datacenters. But where to begin? Attend this session to learn how to approach modernizing traditional apps (MTA), considerations, the available tools and possibilities.
Docker provides PODA (Package Once Deploy Anywhere) and complements WORA (Write Once Run Anywhere) provided by Java. It also helps you reduce the impedance mismatch between dev, test, and production environment and simplifies Java application deployment.
This session will explain how to:
* Run your first Java application with Docker
* Package your Java application with Docker
* Share your Java application using Docker Hub
* Deploy your Java application using Maven
* Deploy your application using Docker for AWS
* Scale Java services with Docker Engine swarm mode
* Package your multi-container application and use service discovery
* Monitor your Docker + Java applications
* Build a deployment pipeline using common tools
What’s the key to successfully adopting microservices on Kubernetes?
Building a development workflow that helps developers code faster.
In this webinar, we introduce the principles of a cloud-native development workflow where individual teams build and ship software independently from each other.
Docker Enterprise Edition (EE) is a secure, scalable, and supported container platform for building and orchestrating applications across multi-tenant Linux and Windows environments. Join Docker product managers as they dive into how Docker EE addresses challenges faced by enterprise customers, as well as the technical architecture of the solution. They will also walk through demos for the latest and upcoming features around application runtime and image management.
KubeCon NA 2017: Ambassador and Envoy (Envoy Salon)Ambassador Labs
Ambassador is an open source Kubernetes-native API Gateway built on the Envoy proxy. We talked about why and how we built Ambassador during the Envoy salon at KubeCon.
DevOps Days Boston 2017: Developer first workflows for KubernetesAmbassador Labs
Kubernetes is a powerful, operational platform for containerized applications. However, the developer workflow on Kubernetes – how you code, deploy, update, and monitor your services – is much less mature.
How should you lay out your Git repo? How do you create loosely coupled services? How do you support deploying your service at any time?
In this talk, we’ll talk about these questions and more. We’ll discuss the journey towards a rapid development workflow, discuss best practices, and, talk about the process we followed to get to a rapid development workflow.
DevOps Days Boston 2017
Back to the Future: Containerize Legacy ApplicationsDocker, Inc.
People typically think of Docker for microservices and try to make the smallest container they can. There are tremendous benefits to a microservices model but those are not the only apps that qualify for containers. Traditional, homegrown, monolithic apps are also great candidates for Docker - why? By containerizing these apps, many of the same agility, portability, security and cost savings benefits can be applied to the hundreds (if not thousands) of apps in your datacenters. But where to begin? Attend this session to learn how to approach modernizing traditional apps (MTA), considerations, the available tools and possibilities.
Docker provides PODA (Package Once Deploy Anywhere) and complements WORA (Write Once Run Anywhere) provided by Java. It also helps you reduce the impedance mismatch between dev, test, and production environment and simplifies Java application deployment.
This session will explain how to:
* Run your first Java application with Docker
* Package your Java application with Docker
* Share your Java application using Docker Hub
* Deploy your Java application using Maven
* Deploy your application using Docker for AWS
* Scale Java services with Docker Engine swarm mode
* Package your multi-container application and use service discovery
* Monitor your Docker + Java applications
* Build a deployment pipeline using common tools
What’s the key to successfully adopting microservices on Kubernetes?
Building a development workflow that helps developers code faster.
In this webinar, we introduce the principles of a cloud-native development workflow where individual teams build and ship software independently from each other.
Docker Enterprise Edition (EE) is a secure, scalable, and supported container platform for building and orchestrating applications across multi-tenant Linux and Windows environments. Join Docker product managers as they dive into how Docker EE addresses challenges faced by enterprise customers, as well as the technical architecture of the solution. They will also walk through demos for the latest and upcoming features around application runtime and image management.
Docker Enterprise Edition: Building a Secure Supply Chain for the Enterprise ...Docker, Inc.
Learn from the development team as we dive into some of the latest and upcoming features in Docker EE, our enterprise container management solution. We will focus on the architecture and configuration of the features and how they can be used with both modern apps and containerized legacy apps. Stay for some tips on monitoring and troubleshooting to help you prevent your production environment from going sideways.
Troubleshooting tips from docker support engineersDocker, Inc.
Docker makes everything easier. But even with the easiest platforms, sometimes you run into problems. In this session, you'll learn first-hand from someone whose job is helping customers fix these problems. In this session, you'll learn how to:
Ask better questions & identify the real problems you want to solve
Use command line tools to inspect internal Swarm and Kubernetes components
Debug and troubleshoot interactions between Enterprise Edition components
Troubleshoot Docker Enterprise Edition Windows nodes
Apply these skills to troubleshooting Docker Enterprise Edition
DockerCon EU 2015: Cultural Revolution - How to Mange the Change Docker BringsDocker, Inc.
Presentation by Ian Miell, author of 'Docker in Practice'
The adoption of Docker brings with it many challenges, not all of themtechnical. There is a clear vision of Docker regularly articulated thatencompasses microservices, decoupled applications, agile and continuous deliveryand integration, and DevOps. But the application of this vision often flounders when confronted by the legacy structures of working and successful businesses.
This talk draws on the experienced gained from several attempts to lead changewithin technical organisations. The speaker experienced both failure to effectchange (through productization of a bespoke software stack, and the introduction of Erlang), and success (moving to a Continuous Improvement modelof complex software maintenance). These experiences informed a successful drive to Docker as the build and delivery system of a 700-strong software company,with significant efficiency improvements.
As with most historical revolutions, the integration of the vision with realityresulted in corners cut and principles compromised in order to effect change.This talk is a discussion of the lessons learned from these experiences in an effort to help the listener clarify and overcome the unique challenges their own organisation brings to making change happen.
DCEU 18: App-in-a-Box with Docker Application PackagesDocker, Inc.
Michael Irwin - Application Architect, Virginia Tech
Docker Application Packages is an experimental tool that makes it easy to share multi-service applications. Create a Compose file, package it in an image, and voilà! You now have an "app-in-a-box"! Not convinced yet? No worries! It took a while for me to be convinced too! In this session, we'll start off by diving into how Docker Application Packages actually works, which will help us understand the use cases. We'll see how dev environments can hook in to this app-in-a-box by replacing the service being worked on with a dev container. Then we'll move on to see how end-to-end functional tests are much easier to run. And, finally, we'll see how to maintain an "app-in-a-box" with the latest versions of each component in a CI/CD pipeline, allowing for a unique app-in-a-box for each feature branch under development. Lots of good material! And lots of live demos!
QCon SF 2017 - Microservices: Service-Oriented DevelopmentAmbassador Labs
Conventional wisdom is that microservices is an architecture that is the spiritual successor to service-oriented architecture. While true, this myopic view of microservices ignores some of the profound workflow shifts in today’s microservices organizations.
The reality is that microservices is an architecture _and_ workflow. In this talk, we’ll introduce the workflow of service-oriented development. Rafael will talk about how the real goal of microservices is to break up a monolithic development workflow. We’ll show you how, by breaking up your workflow, you can build software that lets you move fast and make things.
Tales of Training: Scaling CodeLabs with Swarm Mode and Docker-ComposeDocker, Inc.
Why does any "code lab workshop" or live demo are always such a challenge?
A wise sysadmin once told me: “Get your hand dirty with the production to learn”.
So I want to tell you a story of getting hand dirties, by creating a code lab environment considered as production.
This story will show that we can build a reproducible environment for code-labs workshops, by using the Docker “tools”: the Engine, Swarm Mode, Docker-Compose, Moby, LinuxKit.
Following the spirit of “Play With Docker”, but generalized at any service collection, this Codelab toolkit has been used on code-labs workshops of 120+ people.
That path was not a free lunch, but the lessons learned will give you an idea on how a training environment can be efficiently done with Compose and Swarm Mode, by treating it as a “production” platform, tackling the plumbing “youth” limitations for the better of your use case.
As a trainer, I never learned so much than building something to teach people someone else: this the story I want to tell you, the tale of using Docker as a tool of MASSIVE KNOWLEDGE SHARING, which is the root of growing our industry together.
Using Docker Containers to Improve Reproducibility in Software and Web Engine...Vincenzo Ferme
The ability to replicate and reproduce scientific results has become an increasingly important topic for many academic disciplines. In computer science and, more specifically, software and Web engineering, contributions of scientific work rely on developed algorithms, tools and prototypes, quantitative evaluations, and other computational analyses. Published code and data come with many undocumented assumptions, dependencies, and configurations that are internal knowledge and make reproducibility hard to achieve. This tutorial presents how Docker containers can overcome these issues and aid the reproducibility of research artefacts in software engineering and discusses their applications in the field.
Cite us: http://link.springer.com/chapter/10.1007/978-3-319-38791-8_58
Talking TUF: Securing Software DistributionDocker, Inc.
The Update Framework (TUF) secures new or existing software update systems by providing a specification and library that can be flexibly and universally integrated or natively implemented. The update procedure is notoriously susceptible to malicious attacks and TUF is designed to prevent these and other updater weaknesses.
Docker's Notary project integrates the Go implementation of TUF with Docker Content Trust to verify the publisher of Docker images.
https://github.com/theupdateframework/tuf
DockerCon SF 2015: Docker in the New York Times NewsroomDocker, Inc.
Eric Buth's Presentation at DockerCon SF 2015:
Talk Description: In the New York Times newsroom you’ll find a deeply heterogeneous technology environment that exists – by necessity – outside of the larger, more well-defined development pipelines of the rest of the organization. On the Interactive News team, part of our mission is providing a reliable path to production for designers/developers/reporters who need to be able to make their own technology choices on what can be extremely tight deadlines.
Containerization is becoming increasingly important to these efforts, and we’re ready to share our experience working with Docker and crafting complementary orchestration, communication, and organization solutions.
Learn all there is to know about Docker and containers from the bottom up. In this course, you'll learn what containers are and why they're great, as well as how to utilize Docker to create custom containers and images.
Introduction to Kubernetes - Docker Global Mentor Week 2016Opsta
Kubernetes is an open-source system for automating
deployment, scaling, and management of containerized
applications. This presentation will show you overview of Kubernetes concept.
Docker Global Mentor Week 2016 #DockerInThai at Kaidee on November 18, 2016
Continuous Packaging is also Mandatory for DevOpsDocker, Inc.
While DevOps are comfortable with continuous integration and automatic tests, the area of continuous packaging has not been given the attention it deserves.
Even with containers, delivering an application using software packages provides multiple advantages with regards to file-based installation: it allows to manage dependencies more easily, to provide metadata, checksum, and signature mechanisms, to deal with packages repositories.
But doing that in a continuous packaging approach means that the generation of these packages is fully automated and part of the build process of the software. As a consequence, it eases the various steps of a solution lifecycle (controlled impact of installation/uninstallation,
identical deliveries up to the customer, avoidance of code or metadata duplication)
This presentation will detail the methodological approach around continuous packaging and demonstrate how this can be put in place using an Open Source tool such as project-builder.org and how this allows the MondoRescue project to deliver packages at will for lots of distribution tuples through the same number of Docker containers.
Becoming the Docker Champion: Bringing Docker Back to WorkDocker, Inc.
You’re at DockerCon and have spent the last two days deep in sessions, the Hallway Track, and networking. You’ve heard the stories, learnings and benefits from large and small organizations that are on their devops and app modernization journey with Docker. You may have even begun to identify multiple use cases for Docker at your work and how it could benefit your business and other teams.
In this session, Jim Armstrong of Docker will share how other Docker users have built their cases for broader use of Docker in their organizations. He will share real experiences of developers convincing their ops teams, ops teams introducing Docker to their developers, and passionate Docker users convincing IT executives to adopt Docker.
Continuous Delivery the Hard Way with Kubernetes Weaveworks
How do you version control your deployment config + automate the delivery of your software through a CI/CD pipeline?
We will cover open source tools that facilitate:
• Simultaneously updating configuration and performing releases to Kubernetes
• Rolling back and pinning releases
• Having different release policies to different environments
Read the blog: https://www.weave.works/blog/continuous-delivery-the-hard-way
Visit Weave Cloud: https://www.weave.works/product/cloud/
For more free talks, join our Weave Online User Group: https://www.meetup.com/Weave-User-Group/
Yuvraj Mehta - Group Product Manager, Docker
Steve Richards - Solutions Architect, Docker
Creating a Secure Supply Chain for your applications is vitally important for a compliant and smooth-running application development organization. Every organization needs to understand where their container images come from, who has access to them, understand the security risks to weigh ALL options available before deploying . In this session, we will take a closer look at how Docker Enterprise helps developers, DevOps and DevSecOps teams securely Build and Ship applications through the software pipeline. We’ll dive into security features of the platform’s private registry Image Signing which provides authenticity for image sources and Image Scanning which provides insight into any vulnerabilities. We’ll also look at how this can be automated by policy and seamlessly integrated with your software pipeline to provide a succinct audit trail.
Docker moves very fast, with an edge channel released every month and a stable release every 3 months. Patrick will talk about how Docker introduced Docker EE and a certification program for containers and plugins with Docker CE and EE 17.03 (from March), the announcements from DockerCon (April), and the many new features planned for Docker CE 17.05 in May.
This talk will be about what's new in Docker and what's next on the roadmap
Free GitOps Workshop + Intro to Kubernetes & GitOpsWeaveworks
Follow along in this free workshop and experience GitOps!
AGENDA:
Welcome - Tamao Nakahara, Head of DX (Weaveworks)
Introduction to Kubernetes & GitOps - Mark Emeis, Principal Engineer (Weaveworks)
Weave Gitops Overview - Tamao Nakahara
Free Gitops Workshop - David Harris, Product Manager (Weaveworks)
If you're new to Kubernetes and GitOps, we'll give you a brief introduction to both and how GitOps is the natural evolution of Kubernetes.
Weave GitOps Core is a continuous delivery product to run apps in any Kubernetes. It is free and open source, and you can get started today!
https://www.weave.works/product/gitops-core
If you’re stuck, also come talk to us at our Slack channel! #weave-gitops http://bit.ly/WeaveGitOpsSlack (If you need to invite yourself to the Slack, visit https://slack.weave.works/)
Docker Enterprise Edition: Building a Secure Supply Chain for the Enterprise ...Docker, Inc.
Learn from the development team as we dive into some of the latest and upcoming features in Docker EE, our enterprise container management solution. We will focus on the architecture and configuration of the features and how they can be used with both modern apps and containerized legacy apps. Stay for some tips on monitoring and troubleshooting to help you prevent your production environment from going sideways.
Troubleshooting tips from docker support engineersDocker, Inc.
Docker makes everything easier. But even with the easiest platforms, sometimes you run into problems. In this session, you'll learn first-hand from someone whose job is helping customers fix these problems. In this session, you'll learn how to:
Ask better questions & identify the real problems you want to solve
Use command line tools to inspect internal Swarm and Kubernetes components
Debug and troubleshoot interactions between Enterprise Edition components
Troubleshoot Docker Enterprise Edition Windows nodes
Apply these skills to troubleshooting Docker Enterprise Edition
DockerCon EU 2015: Cultural Revolution - How to Mange the Change Docker BringsDocker, Inc.
Presentation by Ian Miell, author of 'Docker in Practice'
The adoption of Docker brings with it many challenges, not all of themtechnical. There is a clear vision of Docker regularly articulated thatencompasses microservices, decoupled applications, agile and continuous deliveryand integration, and DevOps. But the application of this vision often flounders when confronted by the legacy structures of working and successful businesses.
This talk draws on the experienced gained from several attempts to lead changewithin technical organisations. The speaker experienced both failure to effectchange (through productization of a bespoke software stack, and the introduction of Erlang), and success (moving to a Continuous Improvement modelof complex software maintenance). These experiences informed a successful drive to Docker as the build and delivery system of a 700-strong software company,with significant efficiency improvements.
As with most historical revolutions, the integration of the vision with realityresulted in corners cut and principles compromised in order to effect change.This talk is a discussion of the lessons learned from these experiences in an effort to help the listener clarify and overcome the unique challenges their own organisation brings to making change happen.
DCEU 18: App-in-a-Box with Docker Application PackagesDocker, Inc.
Michael Irwin - Application Architect, Virginia Tech
Docker Application Packages is an experimental tool that makes it easy to share multi-service applications. Create a Compose file, package it in an image, and voilà! You now have an "app-in-a-box"! Not convinced yet? No worries! It took a while for me to be convinced too! In this session, we'll start off by diving into how Docker Application Packages actually works, which will help us understand the use cases. We'll see how dev environments can hook in to this app-in-a-box by replacing the service being worked on with a dev container. Then we'll move on to see how end-to-end functional tests are much easier to run. And, finally, we'll see how to maintain an "app-in-a-box" with the latest versions of each component in a CI/CD pipeline, allowing for a unique app-in-a-box for each feature branch under development. Lots of good material! And lots of live demos!
QCon SF 2017 - Microservices: Service-Oriented DevelopmentAmbassador Labs
Conventional wisdom is that microservices is an architecture that is the spiritual successor to service-oriented architecture. While true, this myopic view of microservices ignores some of the profound workflow shifts in today’s microservices organizations.
The reality is that microservices is an architecture _and_ workflow. In this talk, we’ll introduce the workflow of service-oriented development. Rafael will talk about how the real goal of microservices is to break up a monolithic development workflow. We’ll show you how, by breaking up your workflow, you can build software that lets you move fast and make things.
Tales of Training: Scaling CodeLabs with Swarm Mode and Docker-ComposeDocker, Inc.
Why does any "code lab workshop" or live demo are always such a challenge?
A wise sysadmin once told me: “Get your hand dirty with the production to learn”.
So I want to tell you a story of getting hand dirties, by creating a code lab environment considered as production.
This story will show that we can build a reproducible environment for code-labs workshops, by using the Docker “tools”: the Engine, Swarm Mode, Docker-Compose, Moby, LinuxKit.
Following the spirit of “Play With Docker”, but generalized at any service collection, this Codelab toolkit has been used on code-labs workshops of 120+ people.
That path was not a free lunch, but the lessons learned will give you an idea on how a training environment can be efficiently done with Compose and Swarm Mode, by treating it as a “production” platform, tackling the plumbing “youth” limitations for the better of your use case.
As a trainer, I never learned so much than building something to teach people someone else: this the story I want to tell you, the tale of using Docker as a tool of MASSIVE KNOWLEDGE SHARING, which is the root of growing our industry together.
Using Docker Containers to Improve Reproducibility in Software and Web Engine...Vincenzo Ferme
The ability to replicate and reproduce scientific results has become an increasingly important topic for many academic disciplines. In computer science and, more specifically, software and Web engineering, contributions of scientific work rely on developed algorithms, tools and prototypes, quantitative evaluations, and other computational analyses. Published code and data come with many undocumented assumptions, dependencies, and configurations that are internal knowledge and make reproducibility hard to achieve. This tutorial presents how Docker containers can overcome these issues and aid the reproducibility of research artefacts in software engineering and discusses their applications in the field.
Cite us: http://link.springer.com/chapter/10.1007/978-3-319-38791-8_58
Talking TUF: Securing Software DistributionDocker, Inc.
The Update Framework (TUF) secures new or existing software update systems by providing a specification and library that can be flexibly and universally integrated or natively implemented. The update procedure is notoriously susceptible to malicious attacks and TUF is designed to prevent these and other updater weaknesses.
Docker's Notary project integrates the Go implementation of TUF with Docker Content Trust to verify the publisher of Docker images.
https://github.com/theupdateframework/tuf
DockerCon SF 2015: Docker in the New York Times NewsroomDocker, Inc.
Eric Buth's Presentation at DockerCon SF 2015:
Talk Description: In the New York Times newsroom you’ll find a deeply heterogeneous technology environment that exists – by necessity – outside of the larger, more well-defined development pipelines of the rest of the organization. On the Interactive News team, part of our mission is providing a reliable path to production for designers/developers/reporters who need to be able to make their own technology choices on what can be extremely tight deadlines.
Containerization is becoming increasingly important to these efforts, and we’re ready to share our experience working with Docker and crafting complementary orchestration, communication, and organization solutions.
Learn all there is to know about Docker and containers from the bottom up. In this course, you'll learn what containers are and why they're great, as well as how to utilize Docker to create custom containers and images.
Introduction to Kubernetes - Docker Global Mentor Week 2016Opsta
Kubernetes is an open-source system for automating
deployment, scaling, and management of containerized
applications. This presentation will show you overview of Kubernetes concept.
Docker Global Mentor Week 2016 #DockerInThai at Kaidee on November 18, 2016
Continuous Packaging is also Mandatory for DevOpsDocker, Inc.
While DevOps are comfortable with continuous integration and automatic tests, the area of continuous packaging has not been given the attention it deserves.
Even with containers, delivering an application using software packages provides multiple advantages with regards to file-based installation: it allows to manage dependencies more easily, to provide metadata, checksum, and signature mechanisms, to deal with packages repositories.
But doing that in a continuous packaging approach means that the generation of these packages is fully automated and part of the build process of the software. As a consequence, it eases the various steps of a solution lifecycle (controlled impact of installation/uninstallation,
identical deliveries up to the customer, avoidance of code or metadata duplication)
This presentation will detail the methodological approach around continuous packaging and demonstrate how this can be put in place using an Open Source tool such as project-builder.org and how this allows the MondoRescue project to deliver packages at will for lots of distribution tuples through the same number of Docker containers.
Becoming the Docker Champion: Bringing Docker Back to WorkDocker, Inc.
You’re at DockerCon and have spent the last two days deep in sessions, the Hallway Track, and networking. You’ve heard the stories, learnings and benefits from large and small organizations that are on their devops and app modernization journey with Docker. You may have even begun to identify multiple use cases for Docker at your work and how it could benefit your business and other teams.
In this session, Jim Armstrong of Docker will share how other Docker users have built their cases for broader use of Docker in their organizations. He will share real experiences of developers convincing their ops teams, ops teams introducing Docker to their developers, and passionate Docker users convincing IT executives to adopt Docker.
Continuous Delivery the Hard Way with Kubernetes Weaveworks
How do you version control your deployment config + automate the delivery of your software through a CI/CD pipeline?
We will cover open source tools that facilitate:
• Simultaneously updating configuration and performing releases to Kubernetes
• Rolling back and pinning releases
• Having different release policies to different environments
Read the blog: https://www.weave.works/blog/continuous-delivery-the-hard-way
Visit Weave Cloud: https://www.weave.works/product/cloud/
For more free talks, join our Weave Online User Group: https://www.meetup.com/Weave-User-Group/
Yuvraj Mehta - Group Product Manager, Docker
Steve Richards - Solutions Architect, Docker
Creating a Secure Supply Chain for your applications is vitally important for a compliant and smooth-running application development organization. Every organization needs to understand where their container images come from, who has access to them, understand the security risks to weigh ALL options available before deploying . In this session, we will take a closer look at how Docker Enterprise helps developers, DevOps and DevSecOps teams securely Build and Ship applications through the software pipeline. We’ll dive into security features of the platform’s private registry Image Signing which provides authenticity for image sources and Image Scanning which provides insight into any vulnerabilities. We’ll also look at how this can be automated by policy and seamlessly integrated with your software pipeline to provide a succinct audit trail.
Docker moves very fast, with an edge channel released every month and a stable release every 3 months. Patrick will talk about how Docker introduced Docker EE and a certification program for containers and plugins with Docker CE and EE 17.03 (from March), the announcements from DockerCon (April), and the many new features planned for Docker CE 17.05 in May.
This talk will be about what's new in Docker and what's next on the roadmap
Free GitOps Workshop + Intro to Kubernetes & GitOpsWeaveworks
Follow along in this free workshop and experience GitOps!
AGENDA:
Welcome - Tamao Nakahara, Head of DX (Weaveworks)
Introduction to Kubernetes & GitOps - Mark Emeis, Principal Engineer (Weaveworks)
Weave Gitops Overview - Tamao Nakahara
Free Gitops Workshop - David Harris, Product Manager (Weaveworks)
If you're new to Kubernetes and GitOps, we'll give you a brief introduction to both and how GitOps is the natural evolution of Kubernetes.
Weave GitOps Core is a continuous delivery product to run apps in any Kubernetes. It is free and open source, and you can get started today!
https://www.weave.works/product/gitops-core
If you’re stuck, also come talk to us at our Slack channel! #weave-gitops http://bit.ly/WeaveGitOpsSlack (If you need to invite yourself to the Slack, visit https://slack.weave.works/)
For this info-packed and hands-on workshop we cover:
📍 Introduction to Kubernetes & GitOps talk:
We cover the most popular path that has brought success to many users already - GitOps as a natural evolution of Kubernetes. We'll give an overview of how you can benefit from Kubernetes and GitOps: greater security, reliability, velocity and more. Importantly, we cover definitions and principles standardized by the CNCF's OpenGitOps group and what it means for you.
📍 Get Started with GitOps:
You'll have GitOps up and running in about 30 mins using our free and open source tools! We'll give a brief vision of where you want to be with those security, reliability, and velocity benefits, and then we'll support you while go through the getting started steps. During the workshop, you'll also experience in action and see demos for:
- an opinionated repo structure to minimize decision fatigue
- disaster recovery using GitOps
- Helm charts example
- Multi-cluster example
- all with free and open source tools mostly in the CNCF (eg. Flux and Helm).
If you have questions before or after the workshop, talk to us at #weave-gitops http://bit.ly/WeaveGitOpsSlack (If you need to invite yourself to the Slack, visit https://slack.weave.works/)
Join this info-packed and hands-on workshop where we will cover:
Introduction to Kubernetes & GitOps talk:
We'll cover the most popular path that has brought success to many users already - GitOps as a natural evolution of Kubernetes. We'll give an overview of how you can benefit from Kubernetes and GitOps: greater security, reliability, velocity and more. Importantly, we cover definitions and principles standardized by the CNCF's OpenGitOps group and what it means for you.
Get Started with GitOps:
You'll have GitOps up and running in about 30 mins using our free and open source tools! We'll give a brief vision of where you want to be with those security, reliability, and velocity benefits, and then we'll support you while go through the getting started steps. During the workshop, you'll also experience in action and see demos for:
* an opinionated repo structure to minimize decision fatigue
* disaster recovery using GitOps
* Helm charts example
* Multi-cluster example
* all with free and open source tools mostly in the CNCF (eg. Flux and Helm).
If you have questions before or after the workshop, talk to us at #weave-gitops http://bit.ly/WeaveGitOpsSlack (If you need to invite yourself to the Slack, visit https://slack.weave.works/)
Free GitOps Workshop (with Intro to Kubernetes & GitOps)Weaveworks
View this video on Youtube here: https://youtu.be/tK4S8y3j5TA
In this info-packed and hands-on workshop we covered:
Introduction to Kubernetes & GitOps talk:
We covered the most popular path that has brought success to many users already - GitOps as a natural evolution of Kubernetes. We'll give an overview of how you can benefit from Kubernetes and GitOps: greater security, reliability, velocity and more. Importantly, we cover definitions and principles standardized by the CNCF's OpenGitOps group and what it means for you.
Get Started with GitOps:
You'll have GitOps up and running in about 30 mins using our free and open source tools! We'll give a brief vision of where you want to be with those security, reliability, and velocity benefits, and then we'll support you while go through the getting started steps. During the workshop, you'll also experience in action and see demos for:
- an opinionated repo structure to minimize decision fatigue
- disaster recovery using GitOps
- Helm charts example
- Multi-cluster example
- all with free and open source tools mostly in the CNCF (eg. Flux and Helm).
If you have questions before or after the workshop, talk to us at #weave-gitops http://bit.ly/WeaveGitOpsSlack (If you need to invite yourself to the Slack, visit https://slack.weave.works/)
Kubernetes and CoreOS @ Athens Docker meetupMist.io
Using Kubernetes and CoreOS to increase scalability and availability. Presentation at the Athens Docker meetup http://www.meetup.com/Docker-Athens/events/226277352/
A Comprehensive Introduction to Kubernetes. This slide deck serves as the lecture portion of a full-day Workshop covering the architecture, concepts and components of Kubernetes. For the interactive portion, please see the tutorials here:
https://github.com/mrbobbytables/k8s-intro-tutorials
Kubernetes - how to orchestrate containersinovex GmbH
http://www.meetup.com/Docker-Karlsruhe/events/220797663/
mehr Meetups von inovex:
http://www.meetup.com/inovex-karlsruhe
http://www.meetup.com/inovex-munich
http://www.meetup.com/inovex-cologne
GL DevOps Experts are committed to sharing with our community as much knowledge about Docker and Kubernetes as possible.
Thinking about Kubernetes?
Join Vadym Fabiianskiy and Andrii Mandubyra, GlobalLogic Lviv DevOps Experts and learn:
Container Runtime specifics
What are the building blocks of K8S?
How does Kubernetes work?
Deployment and release strategies
Soft Introduction to Google's framework for taming containers in the cloud. For devs and architects that they just enter the world of cloud, microservices and containers
Netflix Container Scheduling and Execution - QCon New York 2016aspyker
Scheduling a Fuller House: Container Management At Netflix
Customers from over all over the world streamed Forty Two Billion hours of Netflix content last year. Various Netflix batch jobs and an increasing number of service applications use containers for their processing. In this talk Netflix will present a deep dive on the motivations and the technology powering container deployment on top of the AWS EC2 service. The talk will cover our approach to cloud resource management and scheduling with the open source Fenzo library, along with details on docker execution engine as a part of project Titus. As well, the talk will share some of the results so far, lessons learned, and end with a brief look at the developer experience for containers.
Speed & Agility of Innovation with Docker & KubernetesICS
Docker and Kubernetes pave the way for running federated, scalable and redundant systems on the Cloud and on the Edge with the same technology. In this introductory webinar we will go over the history, fundamentals and usage of Docker and Kubernetes and present key takeaways for CTOs and developers. We'll also cover the key benefits of using this technology in both the development and deployment of Qt applications.
We'll discuss:
Short history Docker and Kubernetes
Why are containers suddenly so popular?
The benefits of using Docker with Qt applications
Kubernetes for the Cloud and for the Edge
Moving to containers and Kubernetes: How and Why
Similar to DevOps Days Boston 2017: Real-world Kubernetes for DevOps (20)
Building Microservice Systems Without Cooking Your Laptop: Going “Remocal” wi...Ambassador Labs
When you adopt microservices, containers, and cloud native development, the technologies and architectures may change, but the need for fast feedback doesn’t. Kubernetes enables us to deploy and run applications at scale, but whether you’re coding or testing applications, you want to be able to get work done quickly without spinning up all of your microservices locally and driving your laptop fans into high speed!
Join me for a tour of coding, testing, and shipping microservices using remote-to-local “remocal” tools and techniques. You will:
Understand the challenges with scaling container-based application development – i.e. you can only run so many microservices locally before minikube melts your laptop.
Learn when to use various types of development practices and tooling based on your use case and requirements for production realism, speed, and practicality.
Explore how to utilize containerized dependencies and Docker for testing, including for both apps and services you own and those you don’t.
Learn how Telepresence can enable “remocal” development, expanding your local machine and Docker Desktop out into a remote Kubernetes cluster.
Ship Week 1: Intro to Continuous Delivery and GitOps
When building cloud native applications, software developers are no longer just responsible for coding new features. In the next module of Summer of Kubernetes, our expert guides (with the help of some special guests) will cover how to safely and effectively ship software without disrupting end users. To do this you will:
✅ Understand the basics of continuous delivery and GitOps
✅ Learn about how K8s enables declarative CD (via the use of reconciliation loops)
At GOTO Amsterdam in 2019 I presented how to create an effective cloud native developer workflow. Two years later and many new developer technologies have come and gone, but I still hear daily from cloud developers about the pain and friction associated with building, debugging, and deploying to the cloud. In this talk I'll share my latest learning on how to bring the fun and productivity back into delivering Kubernetes-based software.
In this talk, you will:
- Learn why the core tenets of continuous delivery -- speed and safety -- must be considered in all parts of the cloud native SDLC
- Explore how cloud native coding benefits from thinking separately about the inner development loop, continuous integration, continuous deployment, observability, and analysis
- Understand how cloud native best practices and tooling fit together. Learn about artifact syncing (e.g. Skaffold), dev environment bridging (e.g. Telepresence), GitOps (e.g. Argo), and observability-focused monitoring (e.g. Prometheus, Jaeger)
- Explore the importance of cultivating an effective cloud platform and associated team of experts
- Walk away with an overview of tools that can help you develop and debug effectively when using Kubernetes
Webinar: Accelerate Your Inner Dev Loop for Kubernetes Services Ambassador Labs
Many turn to static duplicate dev environments to shorten the dev loop and isolate code tests, but those bring about additional issues. The idea of safely sharing a dev environment and seeing your code changes in action immediately before sharing them probably seems impossible.
Service Preview, powered by Telepresence and the Ambassador Edge Stack, is here to help! This capability enables you to preview changes immediately and test locally with your tool of choice, while sharing a development cluster.
In this 45-minute webinar, Abhay Saxena will demonstrate using Service Preview to have a fast inner development loop while fixing a bug in a microservice, including stepping through the code in a debugger while other developers continue working unaffected.
[Confoo Montreal 2020] From Grief to Growth: The 7 Stages of Observability - ...Ambassador Labs
In this case-study talk, we will share Brent’s journey through the adoption of modern observability practices as he operated an architecture of distributed services. Facing difficulties using application logs as the primary tool to debug performance and reliability issues? Learn how to improve your company toolkit and engineering habits using existing monitoring tools with the addition of distributed tracing.
https://confoo.ca/en/yul2020/session/from-grief-to-growth-the-7-stages-of-observability
[Confoo Montreal 2020] Build Your Own Serverless with Knative - Alex GervaisAmbassador Labs
Google Cloud Run’s use of Knative introduced a portable Serverless solution built on top of Kubernetes. In this talk, we’ll recap the basic guidelines, use cases, and benefits of a Serverless architecture. Getting up and started, you will learn to take advantage of containers and the Ambassador API Gateway to serve event-driven application workloads and save costs using your existing Kubernetes resources.
https://confoo.ca/en/yul2020/session/build-your-own-serverless-with-knative
[QCon London 2020] The Future of Cloud Native API Gateways - Richard LiAmbassador Labs
The introduction of microservices, Kubernetes, and cloud technology has provided many benefits for developers. However, the age-old problem of getting user traffic routed correctly to the API of your backend applications can still be an issue, and may be complicated with the adoption of cloud native approaches: applications are now composed of multiple (micro)services that are built and released by independent teams; the underlying infrastructure is dynamically changing; services support multiple protocols, from HTTP/JSON to WebSockets and gRPC, and more; and many API endpoints require custom configuration of cross-cutting concerns, such as authn/z, rate limiting, and retry policies.
A cloud native API gateway is on the critical path of all requests, and also on the critical path for the workflow of any developer that is releasing functionality. Join this session to learn about the underlying technology and the required changes in engineering workflows. Key takeaways will include:
A brief overview of the evolution of API gateways over the past ten years, and how the original problems being solved have shifted in relation to cloud native technologies and workflow
Two important challenges when using an API gateway within Kubernetes: scaling the developer workflow; and supporting multiple architecture styles and protocols
Strategies for exposing Kubernetes services and APIs at the edge of your system
Insight into the (potential) future of cloud native API gateways
https://qconlondon.com/london2020/presentation/future-cloud-native-api-gateways
What's New in the Ambassador Edge Stack 1.0? Ambassador Labs
Before Kubernetes, the boundary between your users and your monolithic application was simple to manage. Now with Kubernetes, managing the edge has become dynamic and complex. More developers are involved, there are exponentially more edge operations, and each microservice has diverse requirements.
To fully capitalize on the benefits of Kubernetes, you need to provide a solution that supports the autonomy of application developers, the various requirements of your microservices, and your ability to scale.
You no longer need an API Gateway - you need a self-service, comprehensive edge stack.
In this 40 minute webinar on January 30th, we will discuss and demo the new functionality available with the Ambassador Edge Stack.
Edge Policy Console- graphical UI to visualize and manage all of your edge policies
Security Features- automatic TLS setup via ACME integration, OAuth/OpenID Connect integration, rate limiting, and fine-grained access control
Developer Onboarding- API catalog, Swagger/OpenAPI documentation support, and a fully customizable developer portal
Webinar: Effective Management of APIs and the Edge when Adopting Kubernetes Ambassador Labs
As you adopt Kubernetes, the requirements for your edge change. You now have teams working on multiple services all with different requirements. How can you make sure your edge is Kubernetes-ready?
[KubeCon NA 2018] Telepresence Deep Dive Session - Rafael Schloming & Luke Sh...Ambassador Labs
One of the challenges facing Telepresence is growing the contributor community. It’s a complex application that requires a good understanding of OS networking, VPNs, Kubernetes, and everything in between. We’ll kick off this meeting with a general architectural overview of Telepresence. We’ll talk about how we’ve managed the project to date, and our investments to make it easier. We want to then turn it over for an interactive discussion with participants to see what we can do to make it easier to contribute and grow the Telepresence community.
[KubeCon NA 2018] Effective Kubernetes Develop: Turbocharge Your Dev Loop - P...Ambassador Labs
Every software development cycle is rife with inefficiency. Seasoned devs know the pain of getting access to essential remote systems, waiting for tests to run (and then fail), or debugging with only log files. This talk teaches you how to best leverage Kubernetes, remote infrastructure and related tooling to create a dev cycle that maximizes velocity and minimizes developer friction and frustration.
Using tools such as Kubernetes, Docker and Telepresence, I will walk attendees through several advanced techniques that can be used to produce an effective developer experience and optimized dev loop. The goal of this is to eliminate many sources of frustrating inefficiency and reduce cycle time between releases. I will demonstrate how to incrementally adopt some of these techniques and how to approach introducing new and unfamiliar technology and techniques to skeptical dev teams.
The rise of Layer 7, microservices, and the proxy war with Envoy, NGINX, and ...Ambassador Labs
Modern cloud applications today are built as distributed microservices. These microservices talk to each other over L7 protocols: HTTP, gRPC, Redis, Kafka, and more. In this world, L7 proxies have assumed a crucial role in managing and observing L7 protocols. In this talk, I’ll discuss the evolution of service architectures, the role L7 proxies play in this world, and how there is now a battle raging between Envoy Proxy, HAProxy, and NGINX. I’ll wrap by talking about why we chose Envoy Proxy as the anchor of our Ambassador API Gateway and show how that has enabled a number of new capabilities.
The Simply Complex Task of Implementing Kubernetes Ingress - Velocity NYCAmbassador Labs
Getting traffic into a Kubernetes cluster should be simple, but it’s not. Richard Li explains how software architectures have evolved to take advantage of Kubernetes and discusses the implications that these changes have on ingress. Richard then covers some of the nuances of modern ingress, including authentication, resilience, and observability at the edge, explores how Kubernetes handles ingress today, with NodePorts, LoadBalancers, and ingress controllers, and shares his experience and lessons learned from using several real-world implementations of ingress on Kubernetes.
2017 Microservices Practitioner Virtual Summit: The Mechanics of Deploying En...Ambassador Labs
Abstract: The idea of the "service mesh" is becoming very popular in microservice design circles. However, the mechanics of deploying one into an existing infrastructure are far from simple. In this talk we will cover the logistical details of how Envoy was developed and deployed incrementally at Lyft, focusing primarily on the evolution of service mesh configuration management. We will also discuss why high level systems such as Istio are likely to be the main mechanism by which most customers ultimately get access to service mesh technology.
This talk was presented as part of the Microservices Practitioner Virtual Summit, https://www.microservices.com/summit/
2017 Microservices Practitioner Virtual Summit: How to Avoid Creating a GitHu...Ambassador Labs
As a former journalist, I tend to think in terms of storytelling. As an open source evangelist, I invite you to do the same. What you share on GitHub tells a story about you, your development practices, and your openness to others in the open source community. If you're motivated to gain users, contributors, and positive feedback about your projects, then building a compelling, coherent narrative is essential. In this talk, I'll share insights gained from "editing" Zalando's GitHub repository so we can tell a better story. From 400+ projects of widely differing quality, reliability and maintenance levels, we've winnowed our offerings to make our highest-quality work more discoverable. I'll share how we used GitHub and other tools to create guidelines, categories, and processes that bring sanity to our storytelling. If your organization is facing similar GitHub-bloat challenges, or looking for ways to manage your repos more effectively, you might find some help here.
Paketo Buildpacks : la meilleure façon de construire des images OCI? DevopsDa...Anthony Dahanne
Les Buildpacks existent depuis plus de 10 ans ! D’abord, ils étaient utilisés pour détecter et construire une application avant de la déployer sur certains PaaS. Ensuite, nous avons pu créer des images Docker (OCI) avec leur dernière génération, les Cloud Native Buildpacks (CNCF en incubation). Sont-ils une bonne alternative au Dockerfile ? Que sont les buildpacks Paketo ? Quelles communautés les soutiennent et comment ?
Venez le découvrir lors de cette session ignite
OpenFOAM solver for Helmholtz equation, helmholtzFoam / helmholtzBubbleFoamtakuyayamamoto1800
In this slide, we show the simulation example and the way to compile this solver.
In this solver, the Helmholtz equation can be solved by helmholtzFoam. Also, the Helmholtz equation with uniformly dispersed bubbles can be simulated by helmholtzBubbleFoam.
Accelerate Enterprise Software Engineering with PlatformlessWSO2
Key takeaways:
Challenges of building platforms and the benefits of platformless.
Key principles of platformless, including API-first, cloud-native middleware, platform engineering, and developer experience.
How Choreo enables the platformless experience.
How key concepts like application architecture, domain-driven design, zero trust, and cell-based architecture are inherently a part of Choreo.
Demo of an end-to-end app built and deployed on Choreo.
Climate Science Flows: Enabling Petabyte-Scale Climate Analysis with the Eart...Globus
The Earth System Grid Federation (ESGF) is a global network of data servers that archives and distributes the planet’s largest collection of Earth system model output for thousands of climate and environmental scientists worldwide. Many of these petabyte-scale data archives are located in proximity to large high-performance computing (HPC) or cloud computing resources, but the primary workflow for data users consists of transferring data, and applying computations on a different system. As a part of the ESGF 2.0 US project (funded by the United States Department of Energy Office of Science), we developed pre-defined data workflows, which can be run on-demand, capable of applying many data reduction and data analysis to the large ESGF data archives, transferring only the resultant analysis (ex. visualizations, smaller data files). In this talk, we will showcase a few of these workflows, highlighting how Globus Flows can be used for petabyte-scale climate analysis.
A Comprehensive Look at Generative AI in Retail App Testing.pdfkalichargn70th171
Traditional software testing methods are being challenged in retail, where customer expectations and technological advancements continually shape the landscape. Enter generative AI—a transformative subset of artificial intelligence technologies poised to revolutionize software testing.
Cyaniclab : Software Development Agency Portfolio.pdfCyanic lab
CyanicLab, an offshore custom software development company based in Sweden,India, Finland, is your go-to partner for startup development and innovative web design solutions. Our expert team specializes in crafting cutting-edge software tailored to meet the unique needs of startups and established enterprises alike. From conceptualization to execution, we offer comprehensive services including web and mobile app development, UI/UX design, and ongoing software maintenance. Ready to elevate your business? Contact CyanicLab today and let us propel your vision to success with our top-notch IT solutions.
First Steps with Globus Compute Multi-User EndpointsGlobus
In this presentation we will share our experiences around getting started with the Globus Compute multi-user endpoint. Working with the Pharmacology group at the University of Auckland, we have previously written an application using Globus Compute that can offload computationally expensive steps in the researcher's workflows, which they wish to manage from their familiar Windows environments, onto the NeSI (New Zealand eScience Infrastructure) cluster. Some of the challenges we have encountered were that each researcher had to set up and manage their own single-user globus compute endpoint and that the workloads had varying resource requirements (CPUs, memory and wall time) between different runs. We hope that the multi-user endpoint will help to address these challenges and share an update on our progress here.
SOCRadar Research Team: Latest Activities of IntelBrokerSOCRadar
The European Union Agency for Law Enforcement Cooperation (Europol) has suffered an alleged data breach after a notorious threat actor claimed to have exfiltrated data from its systems. Infamous data leaker IntelBroker posted on the even more infamous BreachForums hacking forum, saying that Europol suffered a data breach this month.
The alleged breach affected Europol agencies CCSE, EC3, Europol Platform for Experts, Law Enforcement Forum, and SIRIUS. Infiltration of these entities can disrupt ongoing investigations and compromise sensitive intelligence shared among international law enforcement agencies.
However, this is neither the first nor the last activity of IntekBroker. We have compiled for you what happened in the last few days. To track such hacker activities on dark web sources like hacker forums, private Telegram channels, and other hidden platforms where cyber threats often originate, you can check SOCRadar’s Dark Web News.
Stay Informed on Threat Actors’ Activity on the Dark Web with SOCRadar!
In software engineering, the right architecture is essential for robust, scalable platforms. Wix has undergone a pivotal shift from event sourcing to a CRUD-based model for its microservices. This talk will chart the course of this pivotal journey.
Event sourcing, which records state changes as immutable events, provided robust auditing and "time travel" debugging for Wix Stores' microservices. Despite its benefits, the complexity it introduced in state management slowed development. Wix responded by adopting a simpler, unified CRUD model. This talk will explore the challenges of event sourcing and the advantages of Wix's new "CRUD on steroids" approach, which streamlines API integration and domain event management while preserving data integrity and system resilience.
Participants will gain valuable insights into Wix's strategies for ensuring atomicity in database updates and event production, as well as caching, materialization, and performance optimization techniques within a distributed system.
Join us to discover how Wix has mastered the art of balancing simplicity and extensibility, and learn how the re-adoption of the modest CRUD has turbocharged their development velocity, resilience, and scalability in a high-growth environment.
Designing for Privacy in Amazon Web ServicesKrzysztofKkol1
Data privacy is one of the most critical issues that businesses face. This presentation shares insights on the principles and best practices for ensuring the resilience and security of your workload.
Drawing on a real-life project from the HR industry, the various challenges will be demonstrated: data protection, self-healing, business continuity, security, and transparency of data processing. This systematized approach allowed to create a secure AWS cloud infrastructure that not only met strict compliance rules but also exceeded the client's expectations.
Gamify Your Mind; The Secret Sauce to Delivering Success, Continuously Improv...Shahin Sheidaei
Games are powerful teaching tools, fostering hands-on engagement and fun. But they require careful consideration to succeed. Join me to explore factors in running and selecting games, ensuring they serve as effective teaching tools. Learn to maintain focus on learning objectives while playing, and how to measure the ROI of gaming in education. Discover strategies for pitching gaming to leadership. This session offers insights, tips, and examples for coaches, team leads, and enterprise leaders seeking to teach from simple to complex concepts.
TROUBLESHOOTING 9 TYPES OF OUTOFMEMORYERRORTier1 app
Even though at surface level ‘java.lang.OutOfMemoryError’ appears as one single error; underlyingly there are 9 types of OutOfMemoryError. Each type of OutOfMemoryError has different causes, diagnosis approaches and solutions. This session equips you with the knowledge, tools, and techniques needed to troubleshoot and conquer OutOfMemoryError in all its forms, ensuring smoother, more efficient Java applications.
Experience our free, in-depth three-part Tendenci Platform Corporate Membership Management workshop series! In Session 1 on May 14th, 2024, we began with an Introduction and Setup, mastering the configuration of your Corporate Membership Module settings to establish membership types, applications, and more. Then, on May 16th, 2024, in Session 2, we focused on binding individual members to a Corporate Membership and Corporate Reps, teaching you how to add individual members and assign Corporate Representatives to manage dues, renewals, and associated members. Finally, on May 28th, 2024, in Session 3, we covered questions and concerns, addressing any queries or issues you may have.
For more Tendenci AMS events, check out www.tendenci.com/events
Listen to the keynote address and hear about the latest developments from Rachana Ananthakrishnan and Ian Foster who review the updates to the Globus Platform and Service, and the relevance of Globus to the scientific community as an automation platform to accelerate scientific discovery.
Understanding Globus Data Transfers with NetSageGlobus
NetSage is an open privacy-aware network measurement, analysis, and visualization service designed to help end-users visualize and reason about large data transfers. NetSage traditionally has used a combination of passive measurements, including SNMP and flow data, as well as active measurements, mainly perfSONAR, to provide longitudinal network performance data visualization. It has been deployed by dozens of networks world wide, and is supported domestically by the Engagement and Performance Operations Center (EPOC), NSF #2328479. We have recently expanded the NetSage data sources to include logs for Globus data transfers, following the same privacy-preserving approach as for Flow data. Using the logs for the Texas Advanced Computing Center (TACC) as an example, this talk will walk through several different example use cases that NetSage can answer, including: Who is using Globus to share data with my institution, and what kind of performance are they able to achieve? How many transfers has Globus supported for us? Which sites are we sharing the most data with, and how is that changing over time? How is my site using Globus to move data internally, and what kind of performance do we see for those transfers? What percentage of data transfers at my institution used Globus, and how did the overall data transfer performance compare to the Globus users?
Strategies for Successful Data Migration Tools.pptxvarshanayak241
Data migration is a complex but essential task for organizations aiming to modernize their IT infrastructure and leverage new technologies. By understanding common challenges and implementing these strategies, businesses can achieve a successful migration with minimal disruption. Data Migration Tool like Ask On Data play a pivotal role in this journey, offering features that streamline the process, ensure data integrity, and maintain security. With the right approach and tools, organizations can turn the challenge of data migration into an opportunity for growth and innovation.
3. datawire.io
Why are we all here?
● You are curious about Kubernetes (and friends) and want a primer!
● You are invested in Kubernetes but are looking to learn about some techniques
and tools to make your developers lives better.
● This is the last preso and you feel guilty about leaving early :)
3
4. datawire.io
Agenda...
● Part 1: Containers and Docker and Kubernetes, Oh my!
● Part 2: Kubernetes Core Concepts
● Part 3: Development Workflow
● Part 4: Logging, Debugging and Resiliency
● Q & A
4
6. datawire.io
What is a container?
● Lightweight Linux environment. It is a form of virtualization… but very different
from a full virtual machine.
● Immutable, deployable artifact.
● Runnable.
● Popularized by Docker but there are many implementations (e.g. LXC, Rkt).
6
7. datawire.io
What is Docker?
● A tool, ecosystem and platform for building, pushing and
running containers.
● The most popular container runtime currently.
● Default container runtime in Kubernetes.
7
8. datawire.io
Why Containers?
● Easy and fast to produce.
● Great way to isolate different components in a complex system.
● Ensures a reproducible runtime for your app along the dev -> build -> test -> prod
pipeline.
● Easy to share in a team or with external partners.
8
9. datawire.io
What is Kubernetes?
● Runs massive numbers of containers based on
lessons learned by Google.
● Schedules and runs all kinds of containers
○ long-lived (e.g. services)
○ short-lived (e.g. pre-launch hooks, cronjobs etc)
● Kubernetes can be thought of as a Distributed OS or process manager
9
10. datawire.io
The Office Tower Analogy
Your product is the building
as a whole.
Your business logic is
the offices and workers
10
11. datawire.io
The Office Tower Analogy
Kubernetes provides the
infrastructure to build your
app around.
11
It is the foundational app
platform for your team to
build your businesses apps
around.
12. datawire.io
Why Kubernetes?
It is not the only kid in the neighborhood…
● Amazon ECS
● Docker Swarm
● Hashicorp Nomad
● Apache Mesos
12
13. datawire.io
Why Kubernetes?
Three big reasons to use Kubernetes over the other solutions:
1. Biggest ecosystem of the bunch and there is a hugely massive community
2. Runnable just about anywhere: cloud, bare-metal, and engineers laptops.
3. Unprecedented cloud portability.
13
14. datawire.io
Kubernetes Architecture
Types of nodes: Masters and Workers
14
Docker Kubelet
Kubeproxy
Kubernetes Node
Docker Kubelet
Kubeproxy
Kubernetes Node
Docker Kubelet
Kubeproxy
Kubernetes Node
Etcd API Server
Controller Manager
Kubernetes Master
Scheduler
16. datawire.io
The “Big Five” of Kubernetes Concepts
● Pods
● Deployments
● Services
● ConfigMaps
● Secrets
16
17. datawire.io
A Pod you say?
● One or more strongly-related containers…
17
Story
Server
name: blog
Redis
Comment
Server
Frontend
host: kube-worker-0
18. datawire.io
A Pod you say?
● Containers in a pod share the same host, pod IP and port space.
18
IP: 100.124.71.175
Story
Server
name: blog
Redis
Comment
Server
Frontend
host: kube-worker-0
19. datawire.io
A Pod you say?
● Unit of scaling is a Pod
19
IP: 100.124.71.175
blog-0
kube-worker-0
IP: 100.124.71.176
blog-1
kube-worker-1
Kubernetes cluster
20. datawire.io
Pods Summary
● Like a host. All containers inside of a pod are run on the same underlying worker
machine.
○ Can therefore reference localhost
○ … or share the filesystem
○ … or use unix domain sockets
● All containers in a Pod share the same IP and port space.
● Pods are not durable.
● Pods are a very low-level primitive construct. Necessary to know, but not
commonly used directly.
20
21. datawire.io
Deployments
● Simple mechanism to configure, scale and update applications.
● Kubernetes does the rest of the hard work of scheduling the Pods across the
cluster to meet desired capacity numbers.
● Works like a thermostat… Ensures the current state is always consistent with the
desired state.
21
22. datawire.io
Services
● Services are stable “names” in Kubernetes that enable you to route traffic to Pods
across the entire cluster.
● Every service gets its own IP address.
● Services route traffic to pods by matching labels.
22
23. datawire.io
Services Illustrated
How to talk to both apps despite different IP addresses?
23
IP: 100.124.71.175
blog-0
kube-worker-0
IP: 100.124.71.176
blog-1
kube-worker-1
Kubernetes cluster
24. datawire.io
Services Illustrated
Add a Service which becomes a DNS A record pointing the pod IP addresses
24
IP: 100.124.71.175
blog-0
kube-worker-0
IP: 100.124.71.176
blog-1
kube-worker-1
Kubernetes cluster
blog DNS (short) => blog
DNS (long) => blog.default.cluster.local
25. datawire.io
Services Illustrated
You can have multiple services target pods using labels and selectors.
25
app=blog
env=prod
blog-0
kube-worker-0
app=blog
env=prod
blog-1
kube-worker-1
Kubernetes cluster
blog blog-staging
app=blog
blog-0
kube-worker-1
app=blog
env=prod
app=blog
26. datawire.io
Service Flavors
● Many different flavors of “Service” in Kubernetes
○ ClusterIP
○ NodePort
○ LoadBalancer
○ ExternalName - often forgotten, but very useful!
26
27. datawire.io
Services Summary
● Creates DNS A records pointing at Pod IP addresses
● Powerful label matching capabilities that enable you to route traffic to particular
pods (e.g. for blue-green or canary releases).
● Supports DNS SRV records so you can avoid hard coding port numbers in your
app code as well.
27
28. datawire.io
ConfigMap
● Containers are immutable… so how do you provide runtime configuration to
them?
● Age old question for immutable infrastructure lots of good (and bad) solutions
have been built over the years.
● Kubernetes solution is built-in as the ConfigMap. Inject configuration information
as…
○ Environment variables
○ Volumes
28
29. datawire.io
Secret
● Cousin of the ConfigMap
● Operates almost exactly the same as a ConfigMap but designed for storing
sensitive information.
● Secret information only sent from master to worker nodes when needed by a
pod. The data lives in memory so it is not on the disk.
● One important aspect of secrets… the master currently stores them in plaintext.
Work in Progress to eliminate this in the future, but worth being aware of.
29
32. datawire.io
Developers...
● Part of our role involves aiding Developers and making them faster and more
productive.
● Kubernetes is awesome and it comes with a lot of power.
● Great power comes with lots of potential for learning pain.
● How do we make developers productive?
32
33. datawire.io
Manifests
● A declarative YAML/JSON config format that describe at a high level how
Kubernetes should operate.
● Kubernetes operates like a thermostat. Transforms current state -> desired state
based on config in the Manifest.
● Manifests often need to be parameterized (e.g. to change the container image).
General approach is to use some kind of templating.
33
34. datawire.io
Structuring an Application
● Often asked, all this stuff is cool, but how should we structure our apps to be
consistent and compatible with tooling?
● Strongly recommend a k8s/ directory in the top of your project
○ Manifests can be concrete and ready to use by just running `kubectl apply -f k8s/ `
○ … or you can take an alternate approach and put templates in that directory and do some kind of
config generation with say Python + Jinja2 (for example).
34
35. datawire.io
Writing Manifests
● Avoid hard coding environment configuration into manifests.
○ Templating
○ Kubectl switches (e.g. for namespaces)
● Stick to YAML even if Kubernetes supports JSON… not uncommon to want
comments in the manifests.
● Kubernetes manifests can be spread across multiple files or kept in a single file.
Strongly recommend using a single file until it becomes bothersome to maintain.
35
36. datawire.io
Development Workflows
● No single workflow that works for all developers or teams
● Need tools that can adapt to changing requirements and process
● Personally, great success with:
○ Trunk-based development model.
○ Using parameterized templates in k8s/ directory.
○ monorepo or “pseudo-monorepo”.
○ Dev-tooling the focuses on speed and maintaining fast iteration cycles
36
37. datawire.io
Forge (https://forge.sh)
● Build and deploy Kubernetes-based microservices quickly.
● Can deploy 1 or 100 services from source to Kubernetes in seconds.
● Changes are applied incrementally. Computes the diff of a change and then
pushes the update to Kubernetes.
37
39. datawire.io
Logging...
● Kubernetes has a built-in log aggregation but it is limited (STDOUT, STDERR only).
● `kubectl logs` is good enough for devs but invest in a real logging solution
for prod.
● You will want something like fluentd and elasticsearch because Kubernetes does
not track historical logs for crashed or terminated pods.
○ Also search capabilities are limited to how much of a grep wizard you are.
39
40. datawire.io
Logging...
● There is more to logging than just application logs.
● Consider introducing a service mesh to your cluster that allows you to do
per-request logging and tracing.
40
41. datawire.io
Services Mesh
● This was covered in a presentation yesterday so here is the recap:
A dedicated infrastructure layer for making service-to-service communication
safe and reliable.
● Kubernetes and CNCF are really pushing Lyft Envoy as the mechanism to build a
service mesh.
41
42. datawire.io
Simple Log Query Tools
● Because the kubectl logs command is so limited many tools have been
written to make it easier...
○ ktail https://github.com/atombender/ktail
○ kubetail https://github.com/johanhaleby/kubetail
○ stern https://github.com/wercker/stern
42
43. datawire.io
Debugging...
Kubernetes IS complex. There are a lot of failure scenarios in all kinds of places. The
Kubernetes docs are pretty helpful for doing some troubleshooting
Application Troubleshooting:
https://kubernetes.io/docs/tasks/debug-application-cluster/debug-application/
Cluster Troubleshooting:
https://kubernetes.io/docs/tasks/debug-application-cluster/debug-cluster/
43
44. datawire.io
Debugging The Cool Way
● Sometimes you need something more powerful than logs…
● Or your developers use a shared development-staging cluster and something is
broken...
● Classic problem with building web services… How do I attach a local process to
the running cloud environment? What about a debugger?
● Super easy in Kubernetes with a tool called Telepresence!
http://www.telepresence.io/
44
45. datawire.io
● Telepresence proxies network
requests, environment variables,
and volumes to local
Telepresence client
● Code locally on your laptop using
your favorite editor and your local
filesystem :)
Telepresence
45
46. datawire.io
Wrapping Up
● Kubernetes is awesome!
● There’s a lot of power and flexibility
● We need to empower developers by providing them excellent tools that make
their lives easier!
● As infrastructure and ops engineers we need to build a stable platform that
developers can use without feeling restricted. The service mesh makes this easier.
46
47. datawire.io
Preso Over! Thank you!
● If you’re building cloud applications on top of Kubernetes, check out our open
source tools:
● Contact us @datawireio or hello@datawire.io
47