This talks covers the current challenges and opportunities for using cloud computing for data-heavy, research computing.
Talk given at the Marcus Evans "Cloud Computing in the Pharmaceutical Industry" conference, Frankfurt 2011.
GitOps, Driving NGN Operations Teams 211127 #kcdgt 2021William Caban
The adoption of cloud-native principles brings new challenges. Scaling and evolving operations teams and staying up to date requires the adoption of new operational models and paradigms.
This deck presents how modern paradigms map to GitOps principles and the charactersitics that must be supported by any software used for GitOps.
Join this workshop and accelerate your journey to production-ready Kubernetes by learning the practical techniques for reliably operating your software lifecycle using the GitOps pattern. The Weaveworks team will be running a full-day workshop, sharing their expertise as users and contributors of Kubernetes and Prometheus, as well as followers of GitOps (operations by pull request) practices.
Using a combination of instructor led demonstrations and hands-on exercises, the workshop will enable the attendee to go into detail on the following topics:
• Developing and operating your Kubernetes microservices at scale
• DevOps best practices and the movement towards a “GitOps” approach
• Building with Kubernetes in production: caring for your apps, implementing CI/CD best practices, and utilizing the right metrics, monitoring tools, and automated alerts
• Operating Kubernetes in production: Upgrading and managing Kubernetes, managing incident response, and adhering to security best practices for Kubernetes
Continuous Delivery NYC: From GitOps to an adaptable CI/CD Pattern for Kubern...Andrew Phillips
Slides from the presentation "From GitOps to an adaptable CI/CD Pattern for Kubernetes" at the Continuous Delivery NYC meetup, by Andrew Phillips. See https://www.meetup.com/ContinuousDeliveryNYC/events/255366708/ and https://www.youtube.com/watch?v=SYeZ0uIwbLc
Hardening Your CI/CD Pipelines with GitOps and Continuous SecurityWeaveworks
Join us for a webinar on how to secure your CI/CD pipeline for Kubernetes with GitOps best practices and continuous runtime protection. As modern developers and DevOps teams are embarking on a quest for speed and reliability through automated CI/CD pipelines for Kubernetes, enterprises still need to ensure security and regulatory compliance.
Together with Deepfence, the Weaveworks team will explain and demonstrate how GitOps continuous delivery pipelines, combined with continuous security observability, improves the overall security of your development workflow - from Git to production.
In this webinar we will demonstrate:
Deepfence container scanning
Git-to-Kubernetes using FluxCD
Deepfence continuous runtime security
Continuous Lifecycle London 2018 Event KeynoteWeaveworks
Today it’s all about delivering velocity without compromising on quality, yet it’s becoming increasingly difficult for organisations to keep up with the challenges of current release management and traditional operations. The demand for developers to own the end-to-end delivery, including operational ownership, is increasing. A “you build it, you own it” development process requires tools that developers know and understand. So I’d like to introduce “GitOps”- an agile software lifecycle for modern applications.
In this session, I will discuss these industry challenges, including current CICD trends and how they’re converging with operations and monitoring. I’ll also illustrate the GitOps model, identify best practices and tools to use, and explain how you can benefit from adopting this methodology inherited from best practices going back 10-15 years.
Docker New York City: From GitOps to a scalable CI/CD Pattern for KubernetesAndrew Phillips
Slides from the presentation "From GitOps to a scalable CI/CD Pattern for Kubernetes" at the Docker New York City meetup, by Andrew Phillips. See https://www.meetup.com/Docker-NewYorkCity/events/257539512/
Deploying software and controlling infrastructure quickly and safely is a hard task.
In this talk, Brice Fernandes, Customer Success Engineer at Weaveworks, discusses GitOps, an operational model for Kubernetes and beyond to speed up development, while retaining extremely strong security guarantees. Brice describes and shows several open source tools developed at Weaveworks to support this approach. You will have a good idea of how to use the GitOps principles to create software pipelines that are fast, safe, and reproducible, while creating clear and high quality audit trails.
Check out the full presentation on YouTube: https://youtu.be/QdCwUUtcj4I
Observe and command your fleets across any kubernetes with weave git opsWeaveworks
Modern day deployments can often resemble the chaos of navigating the high seas with poor visibility and the dangers of unexpected events. Dev and test environments, running test data sets and feature flags in the public cloud, and production being served from a self-managed site that securely hosts client data can all be a challenge without full observability and control.
In this webinar, we show how you can reliably expand your Kubernetes footprint with Weave GitOps. Confidently observe and control your fleets, all from a single pane of glass across any environment.
Join this webinar to learn how to:
Control the health and propagation of customized clusters
Easily assign and secure clusters across multiple teams for multiple purposes
Observe all actions across all environments all from within Git
Understand managing all deployments across your cluster and fleets
GitOps, Driving NGN Operations Teams 211127 #kcdgt 2021William Caban
The adoption of cloud-native principles brings new challenges. Scaling and evolving operations teams and staying up to date requires the adoption of new operational models and paradigms.
This deck presents how modern paradigms map to GitOps principles and the charactersitics that must be supported by any software used for GitOps.
Join this workshop and accelerate your journey to production-ready Kubernetes by learning the practical techniques for reliably operating your software lifecycle using the GitOps pattern. The Weaveworks team will be running a full-day workshop, sharing their expertise as users and contributors of Kubernetes and Prometheus, as well as followers of GitOps (operations by pull request) practices.
Using a combination of instructor led demonstrations and hands-on exercises, the workshop will enable the attendee to go into detail on the following topics:
• Developing and operating your Kubernetes microservices at scale
• DevOps best practices and the movement towards a “GitOps” approach
• Building with Kubernetes in production: caring for your apps, implementing CI/CD best practices, and utilizing the right metrics, monitoring tools, and automated alerts
• Operating Kubernetes in production: Upgrading and managing Kubernetes, managing incident response, and adhering to security best practices for Kubernetes
Continuous Delivery NYC: From GitOps to an adaptable CI/CD Pattern for Kubern...Andrew Phillips
Slides from the presentation "From GitOps to an adaptable CI/CD Pattern for Kubernetes" at the Continuous Delivery NYC meetup, by Andrew Phillips. See https://www.meetup.com/ContinuousDeliveryNYC/events/255366708/ and https://www.youtube.com/watch?v=SYeZ0uIwbLc
Hardening Your CI/CD Pipelines with GitOps and Continuous SecurityWeaveworks
Join us for a webinar on how to secure your CI/CD pipeline for Kubernetes with GitOps best practices and continuous runtime protection. As modern developers and DevOps teams are embarking on a quest for speed and reliability through automated CI/CD pipelines for Kubernetes, enterprises still need to ensure security and regulatory compliance.
Together with Deepfence, the Weaveworks team will explain and demonstrate how GitOps continuous delivery pipelines, combined with continuous security observability, improves the overall security of your development workflow - from Git to production.
In this webinar we will demonstrate:
Deepfence container scanning
Git-to-Kubernetes using FluxCD
Deepfence continuous runtime security
Continuous Lifecycle London 2018 Event KeynoteWeaveworks
Today it’s all about delivering velocity without compromising on quality, yet it’s becoming increasingly difficult for organisations to keep up with the challenges of current release management and traditional operations. The demand for developers to own the end-to-end delivery, including operational ownership, is increasing. A “you build it, you own it” development process requires tools that developers know and understand. So I’d like to introduce “GitOps”- an agile software lifecycle for modern applications.
In this session, I will discuss these industry challenges, including current CICD trends and how they’re converging with operations and monitoring. I’ll also illustrate the GitOps model, identify best practices and tools to use, and explain how you can benefit from adopting this methodology inherited from best practices going back 10-15 years.
Docker New York City: From GitOps to a scalable CI/CD Pattern for KubernetesAndrew Phillips
Slides from the presentation "From GitOps to a scalable CI/CD Pattern for Kubernetes" at the Docker New York City meetup, by Andrew Phillips. See https://www.meetup.com/Docker-NewYorkCity/events/257539512/
Deploying software and controlling infrastructure quickly and safely is a hard task.
In this talk, Brice Fernandes, Customer Success Engineer at Weaveworks, discusses GitOps, an operational model for Kubernetes and beyond to speed up development, while retaining extremely strong security guarantees. Brice describes and shows several open source tools developed at Weaveworks to support this approach. You will have a good idea of how to use the GitOps principles to create software pipelines that are fast, safe, and reproducible, while creating clear and high quality audit trails.
Check out the full presentation on YouTube: https://youtu.be/QdCwUUtcj4I
Observe and command your fleets across any kubernetes with weave git opsWeaveworks
Modern day deployments can often resemble the chaos of navigating the high seas with poor visibility and the dangers of unexpected events. Dev and test environments, running test data sets and feature flags in the public cloud, and production being served from a self-managed site that securely hosts client data can all be a challenge without full observability and control.
In this webinar, we show how you can reliably expand your Kubernetes footprint with Weave GitOps. Confidently observe and control your fleets, all from a single pane of glass across any environment.
Join this webinar to learn how to:
Control the health and propagation of customized clusters
Easily assign and secure clusters across multiple teams for multiple purposes
Observe all actions across all environments all from within Git
Understand managing all deployments across your cluster and fleets
Slides of talk given at London Study of Enterprise Agile Meetup in June 2019.
We go over GitOps and how it affects delivery speed in software development and release.
Hybrid and Multi-Cloud Strategies for Kubernetes with GitOpsSonja Schweigert
One of the biggest advantages Kubernetes has to offer is that it is agnostic to infrastructure and capable of managing diverse workloads running on different compute resources. This allows organizations to manage multiple developer platforms, who can operate across many environments such as on premise, hybrid and multiple clouds.
Streamlined processes and automation is pivotal for operations when managing clusters at scale and maintaining security and policy checks. Paul Curtis, Principal Solutions Architect will demonstrate GitOps and Weave Kubernetes Platform in a hybrid and multi-cloud setup.
Learn how to:
Use model-driven automation to increases reliability and stability across environments
Simplify multi-cluster management with GitOps
Enable developers to push code to production daily (self-service)
Improve utilization and capacity management through Kubernetes platforms on cloud and on-premise infrastructure
Introducing Flagger: a progressive delivery Kubernetes operator for Istio.
Flagger automates the promotion of canary deployments, and uses Istio routing for traffic shifting and Prometheus metrics for canary analysis.
GitOps - Modern best practices for high velocity app dev using cloud native t...Weaveworks
Alexis Richardson, Weaveworks CEO, recently presented this slide deck at the KubeCon + CloudNativeCon event. He covers GitOps - modern best practices for developing apps faster using cloud native tools.
Secure GitOps pipelines for Kubernetes with Snyk & WeaveworksWeaveworks
Together with Snyk, the Weaveworks team will explain and demonstrate how GitOps continuous delivery pipelines combined with good security practices improves the overall security of your development workflow - from Git to production. In the webinar we will:
- Examine security concerns in a typical CICD pipeline
- Operate continuous delivery via pull request
- Discuss Read/Write access in a GitOps pipeline
- Share 5 tips and tricks on securing your source code repos from the beginning
Blog on this topic: https://www.weave.works/blog/secure-gitops-pipelines-for-kubernetes-with-snyk-and-weaveworks
Cloud Native Engineering with SRE and GitOpsWeaveworks
Site reliability engineering (SRE), a model championed by Google, is a software engineering approach to IT operations. For companies striving to become cloud native and adopting modern tools such as Kubernetes, SRE best practices are crucial for success.
In this webinar, Brice, one of our seasoned Customer Reliability Engineers will show how to design a fail-proof Kubernetes platform using tried and tested SRE and GitOps methods.
He will share best practices on:
Increasing performance and ensuring scalability
Managing incident responses through disaster recovery
Designing for High Availability in Kubernetes
Achieving 360 visibility and alerts for your platform
Hands-on GitOps Patterns for Helm Users YouTube Recording: https://youtu.be/ljouUBPtnuI
There are a lot of opinions on how to structure Flux 2 manifests the "GitOps Way." Flux maintainers have given specific examples of how to properly do this in the Flux user guides, demos, and example repos. But most of these focus on Kustomize, and not yet on patterns for users who want to only use Helm.
In this session, Scott Rigby, Developer Experience Engineer at Weaveworks, shares current work towards GitOps patterns for those who want to only use Helm with Flux 2. We welcome your feedback about use-cases and challenges!
GitOps is the best modern practice for CD with KubernetesVolodymyr Shynkar
Evolution of infrastructure as code, a framework that can drastically improve deployment speed and development efficiency.
Youtube version: https://www.youtube.com/watch?v=z2kHFpCPum8
Ship Week 1: Intro to Continuous Delivery and GitOps
When building cloud native applications, software developers are no longer just responsible for coding new features. In the next module of Summer of Kubernetes, our expert guides (with the help of some special guests) will cover how to safely and effectively ship software without disrupting end users. To do this you will:
✅ Understand the basics of continuous delivery and GitOps
✅ Learn about how K8s enables declarative CD (via the use of reconciliation loops)
The Power of GitOps with Flux & GitOps ToolkitWeaveworks
GitOps Days Community Special
Watch the video here: https://youtu.be/0v5bjysXTL8
New to GitOps or been a long-time Flux user?
We'll walk you through the benefits of GitOps and then demo it in action with a sneak peak into the next gen Flux and GitOps Toolkit!
* Automation!
* Visibility!
* Reconciliation!
* Powerful use of Prometheus and Grafana!
* GitOps for Helm!
For Flux users, Flux v1 is decoupled into Flux v2 and GitOps Toolkit. We'll demo how this decoupling gives you more control over how you can do GitOps and with fewer steps!
Join Leigh Capili and Tamao Nakahara as they show you GitOps in action with Flux and GitOps Toolkit.
Note to our Flux community that Flux v2 and the GitOps Toolkit is in development and Flux v1 is in maintenance mode. These talks and upcoming guides will give you the most up-to-date info and steps to migrate once we reach feature parity and start the migration process. We are dedicated to the smoothest experience possible for our Flux community, so please join us if you'd like early access and to give us feedback for the migration process.
We are really excited by the improvements and want to take this opportunity to show you what the GitOps Toolkit is all about, walk you through the guides and get your feedback!
For more info, see https://toolkit.fluxcd.io/.
Here's our latest blog post on Flux v2 and GitOps Toolkit updates: https://www.weave.works/blog/the-road-to-flux-v2-october-update
These are the slides for a talk/workshop delivered to the Cloud Native Wales user group (@CloudNativeWal) on 2019-01-10.
In these slides, we go over some principles of gitops and a hands on session to apply these to manage a microservice.
You can find out more about GitOps online https://www.weave.works/technologies/gitops/
Join us for a webinar on securing the DevOps lifecycle with GitOps. Explore the best defenses for common security threats to code repositories, and see how to apply GitOps best practices to your CICD pipelines for Kubernetes.
The adoption of GitOps already increases the security and stability of your Kubernetes deployment pipelines, keeping your deployment credentials and other secrets inside of the cluster. Although GitOps improves CICD pipeline security, it shifts the security burden to Git itself.
For organizations who wish to defend themselves from malicious internal or external actors, or who operate under high compliance requirements, implementing additional security measures to Git provides identity guarantees, automation of change control, and detailed audit trails.
In this webinar, we’ll discuss 4 common Git attacks and how to mitigate them:
1. User impersonation
2. Malicious user tampering with the repository’s history
3. Malicious user attacking the Git platform
4. Historical attacks on Git clients and their impact
Why performance testing? 2012: Research showed that Amazon would lose $1.6 billion in sales every year if its site took one more second to load. 2013: 39% of e-retailers claimed they lost money last year due to performance or stability problems. 2014: The web performance monitoring company Catchpoint Systems looked at aggregate performance on Black Friday and compared it to the same timeframe in 2013.The results are notable: desktop web pages were 19.85 percent slower, while mobile web pages were a whopping 57.21 percent slower. 2015: Some major e-retailers’ sites buckled under the pressure of heavy holiday traffic during 2015’s Cyber Monday peak traffic times.
Why observability matters - now and in the future (w/guest Grafana)Weaveworks
Carl Bergquist (Grafana) and Neil Gehani (Weaveworks) discuss best practices on how to get started with monitoring your application. Start capturing metrics that matter, aggregate and visualize them in a useful way that allows for identifying bottlenecks and preventing incidents before they happen.
Setting up Notifications, Alerts & Webhooks with Flux v2 by Alison DowdneyWeaveworks
Watch the recording here: https://youtu.be/cakxixc-yQk
❗️ Notifications & Alerts ⚠️
When operating a cluster, different teams may wish to receive notifications about the status of their GitOps pipelines. For example, the on-call team would receive alerts about reconciliation failures in the cluster, while the dev team may wish to be alerted when a new version of an app was deployed and if the deployment is healthy.
Webhook Receivers
The GitOps toolkit controllers are by design pull-based. In order to notify the controllers about changes in Git or Helm repositories, you can setup webhooks and trigger a cluster reconciliation every time a source changes. Using webhook receivers, you can build push-based GitOps pipelines that react to external events.
Alison Dowdney, Developer Experience Engineer at Weaveworks and CNCF Ambassador, walks through how to define a provider, an alert, git commit status, exposing the webhook receiver and defining a git repository and receiver.
Resources
Flux2 Documentation: https://fluxcd.io/docs/
Flux Guide: Setup Notifications: https://fluxcd.io/docs/guides/notifications/
Flux Guide: Setup Webhook receivers: https://fluxcd.io/docs/guides/webhook-receivers/
Flux Roadmap: https://fluxcd.io/docs/roadmap/
Alison's Demo Repo: https://github.com/alisondy/flux-demos
Kubernetes GitOps featuring GitHub, Kustomize and ArgoCDSunnyvale
A brief dissertation about using GitOps paradigm to operate an application on multiple Kubernetes environments thanks to GitHub, ArgoCD and Kustomize. A talk about this matters has been taken at the event #CloudConf2020
Improvements in DNA sequencing technology have lead to a 10,000 fold increase in our data output over the past 5 years. I will describe the lessons we learned whilst scaling our IT infrastructure and tools to cope with the vast amount of data.
Slides of talk given at London Study of Enterprise Agile Meetup in June 2019.
We go over GitOps and how it affects delivery speed in software development and release.
Hybrid and Multi-Cloud Strategies for Kubernetes with GitOpsSonja Schweigert
One of the biggest advantages Kubernetes has to offer is that it is agnostic to infrastructure and capable of managing diverse workloads running on different compute resources. This allows organizations to manage multiple developer platforms, who can operate across many environments such as on premise, hybrid and multiple clouds.
Streamlined processes and automation is pivotal for operations when managing clusters at scale and maintaining security and policy checks. Paul Curtis, Principal Solutions Architect will demonstrate GitOps and Weave Kubernetes Platform in a hybrid and multi-cloud setup.
Learn how to:
Use model-driven automation to increases reliability and stability across environments
Simplify multi-cluster management with GitOps
Enable developers to push code to production daily (self-service)
Improve utilization and capacity management through Kubernetes platforms on cloud and on-premise infrastructure
Introducing Flagger: a progressive delivery Kubernetes operator for Istio.
Flagger automates the promotion of canary deployments, and uses Istio routing for traffic shifting and Prometheus metrics for canary analysis.
GitOps - Modern best practices for high velocity app dev using cloud native t...Weaveworks
Alexis Richardson, Weaveworks CEO, recently presented this slide deck at the KubeCon + CloudNativeCon event. He covers GitOps - modern best practices for developing apps faster using cloud native tools.
Secure GitOps pipelines for Kubernetes with Snyk & WeaveworksWeaveworks
Together with Snyk, the Weaveworks team will explain and demonstrate how GitOps continuous delivery pipelines combined with good security practices improves the overall security of your development workflow - from Git to production. In the webinar we will:
- Examine security concerns in a typical CICD pipeline
- Operate continuous delivery via pull request
- Discuss Read/Write access in a GitOps pipeline
- Share 5 tips and tricks on securing your source code repos from the beginning
Blog on this topic: https://www.weave.works/blog/secure-gitops-pipelines-for-kubernetes-with-snyk-and-weaveworks
Cloud Native Engineering with SRE and GitOpsWeaveworks
Site reliability engineering (SRE), a model championed by Google, is a software engineering approach to IT operations. For companies striving to become cloud native and adopting modern tools such as Kubernetes, SRE best practices are crucial for success.
In this webinar, Brice, one of our seasoned Customer Reliability Engineers will show how to design a fail-proof Kubernetes platform using tried and tested SRE and GitOps methods.
He will share best practices on:
Increasing performance and ensuring scalability
Managing incident responses through disaster recovery
Designing for High Availability in Kubernetes
Achieving 360 visibility and alerts for your platform
Hands-on GitOps Patterns for Helm Users YouTube Recording: https://youtu.be/ljouUBPtnuI
There are a lot of opinions on how to structure Flux 2 manifests the "GitOps Way." Flux maintainers have given specific examples of how to properly do this in the Flux user guides, demos, and example repos. But most of these focus on Kustomize, and not yet on patterns for users who want to only use Helm.
In this session, Scott Rigby, Developer Experience Engineer at Weaveworks, shares current work towards GitOps patterns for those who want to only use Helm with Flux 2. We welcome your feedback about use-cases and challenges!
GitOps is the best modern practice for CD with KubernetesVolodymyr Shynkar
Evolution of infrastructure as code, a framework that can drastically improve deployment speed and development efficiency.
Youtube version: https://www.youtube.com/watch?v=z2kHFpCPum8
Ship Week 1: Intro to Continuous Delivery and GitOps
When building cloud native applications, software developers are no longer just responsible for coding new features. In the next module of Summer of Kubernetes, our expert guides (with the help of some special guests) will cover how to safely and effectively ship software without disrupting end users. To do this you will:
✅ Understand the basics of continuous delivery and GitOps
✅ Learn about how K8s enables declarative CD (via the use of reconciliation loops)
The Power of GitOps with Flux & GitOps ToolkitWeaveworks
GitOps Days Community Special
Watch the video here: https://youtu.be/0v5bjysXTL8
New to GitOps or been a long-time Flux user?
We'll walk you through the benefits of GitOps and then demo it in action with a sneak peak into the next gen Flux and GitOps Toolkit!
* Automation!
* Visibility!
* Reconciliation!
* Powerful use of Prometheus and Grafana!
* GitOps for Helm!
For Flux users, Flux v1 is decoupled into Flux v2 and GitOps Toolkit. We'll demo how this decoupling gives you more control over how you can do GitOps and with fewer steps!
Join Leigh Capili and Tamao Nakahara as they show you GitOps in action with Flux and GitOps Toolkit.
Note to our Flux community that Flux v2 and the GitOps Toolkit is in development and Flux v1 is in maintenance mode. These talks and upcoming guides will give you the most up-to-date info and steps to migrate once we reach feature parity and start the migration process. We are dedicated to the smoothest experience possible for our Flux community, so please join us if you'd like early access and to give us feedback for the migration process.
We are really excited by the improvements and want to take this opportunity to show you what the GitOps Toolkit is all about, walk you through the guides and get your feedback!
For more info, see https://toolkit.fluxcd.io/.
Here's our latest blog post on Flux v2 and GitOps Toolkit updates: https://www.weave.works/blog/the-road-to-flux-v2-october-update
These are the slides for a talk/workshop delivered to the Cloud Native Wales user group (@CloudNativeWal) on 2019-01-10.
In these slides, we go over some principles of gitops and a hands on session to apply these to manage a microservice.
You can find out more about GitOps online https://www.weave.works/technologies/gitops/
Join us for a webinar on securing the DevOps lifecycle with GitOps. Explore the best defenses for common security threats to code repositories, and see how to apply GitOps best practices to your CICD pipelines for Kubernetes.
The adoption of GitOps already increases the security and stability of your Kubernetes deployment pipelines, keeping your deployment credentials and other secrets inside of the cluster. Although GitOps improves CICD pipeline security, it shifts the security burden to Git itself.
For organizations who wish to defend themselves from malicious internal or external actors, or who operate under high compliance requirements, implementing additional security measures to Git provides identity guarantees, automation of change control, and detailed audit trails.
In this webinar, we’ll discuss 4 common Git attacks and how to mitigate them:
1. User impersonation
2. Malicious user tampering with the repository’s history
3. Malicious user attacking the Git platform
4. Historical attacks on Git clients and their impact
Why performance testing? 2012: Research showed that Amazon would lose $1.6 billion in sales every year if its site took one more second to load. 2013: 39% of e-retailers claimed they lost money last year due to performance or stability problems. 2014: The web performance monitoring company Catchpoint Systems looked at aggregate performance on Black Friday and compared it to the same timeframe in 2013.The results are notable: desktop web pages were 19.85 percent slower, while mobile web pages were a whopping 57.21 percent slower. 2015: Some major e-retailers’ sites buckled under the pressure of heavy holiday traffic during 2015’s Cyber Monday peak traffic times.
Why observability matters - now and in the future (w/guest Grafana)Weaveworks
Carl Bergquist (Grafana) and Neil Gehani (Weaveworks) discuss best practices on how to get started with monitoring your application. Start capturing metrics that matter, aggregate and visualize them in a useful way that allows for identifying bottlenecks and preventing incidents before they happen.
Setting up Notifications, Alerts & Webhooks with Flux v2 by Alison DowdneyWeaveworks
Watch the recording here: https://youtu.be/cakxixc-yQk
❗️ Notifications & Alerts ⚠️
When operating a cluster, different teams may wish to receive notifications about the status of their GitOps pipelines. For example, the on-call team would receive alerts about reconciliation failures in the cluster, while the dev team may wish to be alerted when a new version of an app was deployed and if the deployment is healthy.
Webhook Receivers
The GitOps toolkit controllers are by design pull-based. In order to notify the controllers about changes in Git or Helm repositories, you can setup webhooks and trigger a cluster reconciliation every time a source changes. Using webhook receivers, you can build push-based GitOps pipelines that react to external events.
Alison Dowdney, Developer Experience Engineer at Weaveworks and CNCF Ambassador, walks through how to define a provider, an alert, git commit status, exposing the webhook receiver and defining a git repository and receiver.
Resources
Flux2 Documentation: https://fluxcd.io/docs/
Flux Guide: Setup Notifications: https://fluxcd.io/docs/guides/notifications/
Flux Guide: Setup Webhook receivers: https://fluxcd.io/docs/guides/webhook-receivers/
Flux Roadmap: https://fluxcd.io/docs/roadmap/
Alison's Demo Repo: https://github.com/alisondy/flux-demos
Kubernetes GitOps featuring GitHub, Kustomize and ArgoCDSunnyvale
A brief dissertation about using GitOps paradigm to operate an application on multiple Kubernetes environments thanks to GitHub, ArgoCD and Kustomize. A talk about this matters has been taken at the event #CloudConf2020
Improvements in DNA sequencing technology have lead to a 10,000 fold increase in our data output over the past 5 years. I will describe the lessons we learned whilst scaling our IT infrastructure and tools to cope with the vast amount of data.
Marco Cattaneo "Event data processing in LHCb"Yandex
Семинар «Использование современных информационных технологий для решения современных задач физики частиц» в московском офисе Яндекса, 3 июля 2012
Marco Cattaneo, CERN
IBM Business Analytics and Optimization - Traffic Management with IBM InfoSph...IBM Sverige
Professor Haris och hans forskare på KTH har ett samarbete med de som utvecklar InfoSphere Streams på IBM Research. I detta projektet analyseras trafikdata från Stockholmsområdet för att se hur man kan nyttja informationen på bästa sätt för att styra trafiken smartare och informera resenärerna om hur man tar sig fram på bästa sätt.
Förutom själva Stockholmsområdet, så analyserar man trafiken till/från Arlanda. Bland annat vill man prediktera sannolikheten för att man kommer i tid till sin flygavgång på Arlanda beroende på vilken tid man skall åka och beroende på vilket transportsätt man väljer. KTH är vinnare till priset för en smartare planet 2010.
Talare: Haris N. Koutsopoulos, Professor and Head of Transportation and Logistics Division, KTH
Denna presentation hölls på ett seminariepass för Business Analytics & Optimization under IBM Software Day 2010.
Automating the process of continuously prioritising data, updating and deploy...Ola Spjuth
Presentation at Data Innovation Summit 2019 in Stockholm, Sweden.
ABSTRACT
Microscopes are capable of producing vast amounts of data, and when used in automated laboratories both the number and size of images present many challenges for storing, categorizing, analyzing, annotating, and transforming the data into actionable information that can used for decision making; either by humans or machines. In this presentation I will describe the informatics system we have established at the Department of Pharmaceutical Biosciences at Uppsala University, which consists of computational hardware (CPUs, GPUs, storage), middleware (Kubernetes), imaging database (OMERO), and workflow system (Pachyderm) to perform online prioritization of new data, as well as the continuous analytics system to automate the process from captured images to continuously updated and deployed AI models. The AI methodologies include Deep Learning models trained on image data, and conventional machine learning models trained on features extracted from images or chemical structures. Due to the microservice architecture the system is scalable and can be expanded using hybrid-architectures with cloud computing resources. The informatics system serves a robotized cell profiling setup with incubators, liquid handling and high-content microscopy. The lab is quite young and is targeting applications primarily in drug screening and toxicity assessment, with the aim to improve research using AI and intelligent design of experiments.
Observing Intraday Indicators Using Real-Time Tick Data on Apache Superset an...DataWorks Summit
The Central Bank of the Republic of Turkey is primarily responsible for steering the monetary and exchange rate policies in Turkey.
One of the major core functions of the Bank is market operations. In this context, analyzing and interpreting real-time tick data related to money market instruments has become not only a requirement but also a challenge.
For this use case, an API provided by one of the financial data vendors has been used to gather real-time tick data and data routing has been orchestrated by Apache NiFi.
Gathered data is being transferred to Kafka topics and then handed off to Druid for real-time indexing tasks.
Indicators such as effective cost, bid-ask spread, price impact measures, return reversal are calculated using Apache Storm and finally visualized by means of Apache Superset in order to provide decision-makers with a new set of tools.
Supercharging Data Performance for Real-Time Data Analysis Ryft
The velocity and volume of data are growing faster than ever before, and companies are looking for new methods to speed their data analytics. Using an innovative FPGA-based architecture, the Ryft ONE supercharges data analytics and provides you more value from your data.
Machine Learning and Apache Edgent with STM32F401 to Firebase Mostafa Ramezani
The STM32F401 Nucleo board detects the heartbeats through the pulse sensor. Do real-time analytics on the continuous streams of data coming from sensor by Apache Edgent. Such as Aggregation, Categorize data in order to Reduce the amount of data transmitted to analytics servers and Reduce the amount of data to be stored . Data Pre-processed with Apache Edgent and then submitted to the Machine Learning algorithm. ML algorithm does Classification of Heart Disease Using K- Nearest Neighbor and detect Heart Disease such as Atrial fibrillation(Atrial fibrillation (AF or A-fib) is an abnormal heart rhythm characterized by rapid and irregular beating ) Finally all Data sent to Firebase Realtime Database by Firebase Admin in Java Application.
Presentation: Overview of Kognitio, Kognitio Cloud and the Kognitio Analytical Platform
Kognitio is driving the convergence of Big Data, in-memory analytics and cloud computing. Having delivered the first in-memory analytical platform in 1989, it was designed from the ground up to provide the highest amount of scalable compute power to allow rapid execution of complex analytical queries without the administrative overhead of manipulating data. Kognitio software runs on industry-standard x86 servers, or as an appliance, or in Kognitio Cloud, a ready-to-use analytical platform. Kognitio Cloud is a secure, private or public cloud Platform-as-a-Service (PaaS), leveraging the cloud computing model to make the Kognitio Analytical Platform available on a subscription basis. Clients span industries, including market research, consumer packaged goods, retail, telecommunications, financial services, insurance, gaming, media and utilities.
To learn more, visit www.kognitio.com and follow us on Facebook, LinkedIn and Twitter.
Are blade server suitable for HPTC? This talk covers the pros and cons of building your next cluster using blades.
Talk given at International Supercomputing blade workshop in 2007.
Keynote given at BOSC, 2010.
Does the hype surrounding cloud match the reality?
Can we use them to solve the problems in provisioning IT services to support next-generation sequencing?
Next generation genomics: Petascale data in the life sciencesGuy Coates
Keynote presentation at OGF 28.
The year 2000 saw the release of "The" human genome, the product of a the combined sequencing effort of the whole planet. In 2010, single institutions are sequencing thousands of genomes a year, producing petabytes of data. Furthermore, many of the large scale sequencing projects are based around international collaboration and consortia. The talk will explore how Grid and Cloud technologies are being used to share genomics data around the planet, revolutionizing life science research.
Next-generation sequencing: Data mangementGuy Coates
Next-generation sequencing is producing vast amounts of data. Providing storage and compute is only half the battle. Researchers and IT staff need to be able to "manage" data, in order to stay productive.
Talk given at BIO-IT World, Europe 2010.
The computational requirements of next generation sequencing is placing a huge demand on IT organisations .
Building compute clusters is now a well understood and relatively straightforward problem. However, NGS sequencing applications require large amounts of storage, and high IO rates.
This talk details our approach for providing storage for next-gen sequencing applications.
Talk given at BIO-IT World, Europe, 2009.
The Next-Generation sequencing data-deluge requires storage and compute services to be provisioned at an ever-increasing rate. Can Cloud (and last decade's buzzword, Grid), help us?
Talk given at the NHGRI Cloud computing workshop, 2010.
Climate Impact of Software Testing at Nordic Testing DaysKari Kakkonen
My slides at Nordic Testing Days 6.6.2024
Climate impact / sustainability of software testing discussed on the talk. ICT and testing must carry their part of global responsibility to help with the climat warming. We can minimize the carbon footprint but we can also have a carbon handprint, a positive impact on the climate. Quality characteristics can be added with sustainability, and then measured continuously. Test environments can be used less, and in smaller scale and on demand. Test techniques can be used in optimizing or minimizing number of tests. Test automation can be used to speed up testing.
Threats to mobile devices are more prevalent and increasing in scope and complexity. Users of mobile devices desire to take full advantage of the features
available on those devices, but many of the features provide convenience and capability but sacrifice security. This best practices guide outlines steps the users can take to better protect personal devices and information.
Pushing the limits of ePRTC: 100ns holdover for 100 daysAdtran
At WSTS 2024, Alon Stern explored the topic of parametric holdover and explained how recent research findings can be implemented in real-world PNT networks to achieve 100 nanoseconds of accuracy for up to 100 days.
Dr. Sean Tan, Head of Data Science, Changi Airport Group
Discover how Changi Airport Group (CAG) leverages graph technologies and generative AI to revolutionize their search capabilities. This session delves into the unique search needs of CAG’s diverse passengers and customers, showcasing how graph data structures enhance the accuracy and relevance of AI-generated search results, mitigating the risk of “hallucinations” and improving the overall customer journey.
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
In the rapidly evolving landscape of technologies, XML continues to play a vital role in structuring, storing, and transporting data across diverse systems. The recent advancements in artificial intelligence (AI) present new methodologies for enhancing XML development workflows, introducing efficiency, automation, and intelligent capabilities. This presentation will outline the scope and perspective of utilizing AI in XML development. The potential benefits and the possible pitfalls will be highlighted, providing a balanced view of the subject.
We will explore the capabilities of AI in understanding XML markup languages and autonomously creating structured XML content. Additionally, we will examine the capacity of AI to enrich plain text with appropriate XML markup. Practical examples and methodological guidelines will be provided to elucidate how AI can be effectively prompted to interpret and generate accurate XML markup.
Further emphasis will be placed on the role of AI in developing XSLT, or schemas such as XSD and Schematron. We will address the techniques and strategies adopted to create prompts for generating code, explaining code, or refactoring the code, and the results achieved.
The discussion will extend to how AI can be used to transform XML content. In particular, the focus will be on the use of AI XPath extension functions in XSLT, Schematron, Schematron Quick Fixes, or for XML content refactoring.
The presentation aims to deliver a comprehensive overview of AI usage in XML development, providing attendees with the necessary knowledge to make informed decisions. Whether you’re at the early stages of adopting AI or considering integrating it in advanced XML development, this presentation will cover all levels of expertise.
By highlighting the potential advantages and challenges of integrating AI with XML development tools and languages, the presentation seeks to inspire thoughtful conversation around the future of XML development. We’ll not only delve into the technical aspects of AI-powered XML development but also discuss practical implications and possible future directions.
UiPath Test Automation using UiPath Test Suite series, part 6DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 6. In this session, we will cover Test Automation with generative AI and Open AI.
UiPath Test Automation with generative AI and Open AI webinar offers an in-depth exploration of leveraging cutting-edge technologies for test automation within the UiPath platform. Attendees will delve into the integration of generative AI, a test automation solution, with Open AI advanced natural language processing capabilities.
Throughout the session, participants will discover how this synergy empowers testers to automate repetitive tasks, enhance testing accuracy, and expedite the software testing life cycle. Topics covered include the seamless integration process, practical use cases, and the benefits of harnessing AI-driven automation for UiPath testing initiatives. By attending this webinar, testers, and automation professionals can gain valuable insights into harnessing the power of AI to optimize their test automation workflows within the UiPath ecosystem, ultimately driving efficiency and quality in software development processes.
What will you get from this session?
1. Insights into integrating generative AI.
2. Understanding how this integration enhances test automation within the UiPath platform
3. Practical demonstrations
4. Exploration of real-world use cases illustrating the benefits of AI-driven test automation for UiPath
Topics covered:
What is generative AI
Test Automation with generative AI and Open AI.
UiPath integration with generative AI
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
How to Get CNIC Information System with Paksim Ga.pptxdanishmna97
Pakdata Cf is a groundbreaking system designed to streamline and facilitate access to CNIC information. This innovative platform leverages advanced technology to provide users with efficient and secure access to their CNIC details.
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...SOFTTECHHUB
The choice of an operating system plays a pivotal role in shaping our computing experience. For decades, Microsoft's Windows has dominated the market, offering a familiar and widely adopted platform for personal and professional use. However, as technological advancements continue to push the boundaries of innovation, alternative operating systems have emerged, challenging the status quo and offering users a fresh perspective on computing.
One such alternative that has garnered significant attention and acclaim is Nitrux Linux 3.5.0, a sleek, powerful, and user-friendly Linux distribution that promises to redefine the way we interact with our devices. With its focus on performance, security, and customization, Nitrux Linux presents a compelling case for those seeking to break free from the constraints of proprietary software and embrace the freedom and flexibility of open-source computing.
3. The Sanger Institute
Funded by Wellcome Trust.
• 2nd largest research charity in the world.
• ~700 employees.
• Based in Hinxton Genome Campus,
Cambridge, UK.
Large scale genomic research.
• Sequenced 1/3 of the human genome.
(largest single contributor).
• We have active cancer, malaria,
pathogen and genomic variation / human
health studies.
All data is made publicly
available.
• Websites, ftp, direct database. access,
programmatic APIs.
8. Ensembl
Ensembl is a system for genome Annotation.
Data visualisation / Mining web services.
• www.ensembl.org
• Provides web / programmatic interfaces to genomic data.
• 10k visitors / 126k page views per day.
Compute Pipeline (HPTC Workload)
• Take a raw genome and run it through a compute pipeline to find genes
and other features of interest.
• Ensembl at Sanger/EBI provides automated analysis for 51 vertebrate
genomes.
• Software is Open Source (apache license).
• Data is free for download.
We have web services and HPTC workloads running on
Iaas.
9. Why Cloud?
Web services
• Was hosted in a single datacentre at the Genome Campus, UK.
• 1 datacentre = Single point of failure.
• Access slow if you were not in western Europe.
Cloud Application
• Build worldwide network of mirrors on IaaS.
HPC
• People want to run Ensembl HPC pipeline on their own data.
• Requires skilled bioinformatician to get the software running and access
to a HPC cluster.
Cloud Application
• Build HPC SaaS.
• Users deploy ready-to-run Ensembl code on AWS, self-assembles into a
HPC cluster and analyses their data.
14. Economic Trends:
As cost of sequencing halves every 12
months.
• cf Moore's Law
The Human genome project:
• 13 years.
• 23 labs.
• $500 Million.
A Human genome today:
• 3 days.
• 1 machine.
• $10,000.
• Large centres are now doing studies with 10,000s of
genomes.
Trend will continue:
• Generation 3 sequencers are on their way.
• $500 genome is probable within 5 years.
15. The scary graph
Peak Yearly capillary Current weeky sequencing:
sequencing: 30 Gbase 3000 Gbase
16. Managing Growth
We have exponential growth in
storage and compute.
• Storage /compute doubles every 12 Disk Storage
months. 6000
• 2009 ~7 PB raw
5000
Gigabase of sequence ≠ Gigbyte 4000
of storage.
• 16 bytes per base for for sequence
Terabytes
3000
data.
• Intermediate analysis typically need 10x 2000
disk space of the raw data. 1000
Moore's law will not save us. 0
• Transistor/disk density: Td=18 months
1995 1997 1999 2001 2003 2005 2007 2009
1994 1996 1998 2000 2002 2004 2006 2008
• Sequencing cost: Td=12 months Year
• Sequencing output: Td=3-6 months
17. What do you need to do
sequencing?
LIMS System / Data Tracking
External
External
analysis
analysis Data
Data
Sample prep
Sample prep Sequencer
Sequencer repository
repository
software
software repository
repository
Integrated
Integrated
compute
compute
HPC
HPC
Resource
Resource
18. What IT do you need to do
sequencing?
LIMS System / Data Tracking
External
External
analysis
analysis Data
Data
Sample prep
Sample prep Sequencer
Sequencer repository
repository
software
software repository
repository
Integrated
Integrated
compute
compute
HPC
HPC
Resource
Resource
Part covered in the grant
19. This is really hard...
We have a whole division of HPC specialists, LIMs
developers, bio-informaticians.
What about smaller labs with 1 or 2 sequencers?
20. ...and then change it.
Sequencing informatics is massively fluid.
• New chemistry.
• More sequencing machines.
• New analysis software.
Constant cycle of development and deployment.
22. What can we put on the Cloud?
LIMS System / Data Tracking
External
External
analysis
analysis Data
Data
Sample prep
Sample prep Sequencer
Sequencer repository
repository
software
software repository
repository
Integrated
Integrated
compute
compute
HPC
HPC
Resource
Resource
23. Does it Cloud?
How do we decide what to cloud?
Rule of thumb borrowed from HPC.
• Small data / High CPU work better in distributed environments.
IO Bound CPU Bound
/ Large data / small data
24. Sequencing Data
Data size per Genome
Tracking / LIMs Structured data
(100s Kbytes) (databases)
Individual
features (3MB)
Variation data (1GB)
Alignments (200 GB)
Sequence + quality data (500 GB)
Unstructured data
(flat files)
( Raw data (TB) )
25. Sequencing Data
Data size per Genome
Cloud Friendly
Tracking / LIMs Structured data
(100s Kbytes) (databases)
Individual
features (3MB)
Variation data (1GB)
Alignments (200 GB)
Sequence + quality data (500 GB)
Unstructured data
Cloud Unfriendly (flat files)
( Raw data (TB) )
26. Can we Cloudify Sequencing?
LIMS System / Data Tracking
External
External
analysis
analysis Data
Data
Sample prep
Sample prep Sequencer
Sequencer repository
repository
software
software repository
repository
Integrated
Integrated
compute
compute
HPC
HPC
Resource
Resource
27. What are the blockers?
HPC infrastructure is now available in the cloud.
• Good enough for 95% of sequencing.
Doing big data is hard:
1. You have to get the data there first.
2. You may not be allowed to put the data there.
28. Moving data is hard
Tools:
• (FTP,ssh/rsync) are not suited to wide-area networks.
• WAN tools: gridFTP/FDT/Aspera.
Data transfer rates (gridFTP/FDT via our 2 Gbit/s site link).
• Cambridge → EC2 East coast: 12 Mbytes/s (96 Mbits/s)
• Cambridge → EC2 Dublin: 25 Mbytes/s (200 Mbits/s)
• 11 hours to move 1TB to Dublin.
• 23 hours to move 1 TB to East coast.
What speed should we get?
• Once we leave JANET (UK academic network) finding out what the
connectivity is and what we should expect is almost impossible.
Do you have fast enough disks at each end to keep the
network full?
Why not just ship disks?
• Logistical nightmare.
• Format issues, corruption, slow.
29. Networking
How do we improve data
transfers across the public
internet?
• CERN approach; don't.
• Dedicated networking has been
put in between CERN and the T1
centres who get all of the CERN
data.
Can it work for cloud?
• Buy dedicated bandwidth to a
provider.
• Ties you in.
• Should they pay?
We need good connectivity
to everywhere.
31. Are you allowed to put data on
the cloud?
Default policy:
“Our data is confidential/important/critical to our business.
We must keep our data on our computers.”
32. What does “My System”
mean?
My System Not my system
Purchased computer in Purchased computer in IaaS on a cloud
my data centre a co-lo facility provider
Leased computer in
Traditionally outsourced IT SaaS on a cloud
my data centre
service provider
Root / Admin Access?
VPN / inside or outside firewall?
Encrypted/ Non encrypted? Legal / IP agreement in place?
33. How confidential is the data?
Low Risk High Risk
Anonymised Personally
Publically available datasets identifiable datasets Trade Secret /
Genome data (eg individual Patentable data
genomes with no
identifiers)
34. Reasons to be optimistic:
Most (all?) data security issues can be dealt with.
• But the devil is in the details.
• Data can be put on the cloud, if care is taken.
It is probably more secure there than in your own data-
centre.
• Can you match AWS data availability guarantees?
Are cloud providers different from any other organisation
you outsource to?
35. Outstanding Issues
Audit and compliance:
• If you need IP agreements, above your providers standard T&Cs, how do
you push them through?
Geographical boundaries mean little in the cloud.
• Data can be replicated across national boundaries, without end user
being aware.
Moving personally identifiable data outside of the EU is
potentially problematic.
• (Can be problematic within the EU; privacy laws are not as harmonised as
you might think.)
• More sequencing experiments are trying to link with phenotype data. (ie
personally identifiable medical records).
36. Private Cloud to rescue?
Sequencing increasingly takes place in large consortiums.
• Eg International Cancer Genome Consortium http://www.icgc.org)
Can we do private clouds within the consortium?
37. Traditional Collaboration
IT
IT
IT
IT Sequencing
Sequencing IT
IT
Sequencing centre
centre Sequencing
Sequencing Sequencing
centre
centre centre
centre
Sequencing
Sequencing
Centre + DCC
Centre + DCC
IT
IT
38. Cloud Collaborations
Sequencing
Sequencing
centre
centre
Private Cloud
Private Cloud
IaaS // SaaS
IaaS SaaS
Sequencing
Sequencing Sequencing
Sequencing
centre
centre centre
centre
Private Cloud
Private Cloud
IaaS // SaaS
IaaS SaaS
Sequencing
Sequencing
Centre
Centre
39. Private Cloud
Advantages:
• LIMS / analysis software easily shared with consortium.
• Small organisations leverage expertise of big IT organisations.
• Academia tends to be linked by fast research networks.
• Moving data is easier.
• Consortium will be signed up to data-access agreements.
• Simplifies data governance.
Problems:
• Big change in funding model.
• Are big centres set up to provide private cloud services?
•Selling services is hard if you are a charity.
• Can we do it as well as the big internet companies?
41. Dark Archives
Storing data in an archive is not
particularly useful.
• You need to be able to access the
data and do something useful with it.
Data in current archives is
“dark”.
• You can put/get data, but cannot
compute across it.
• Is data in an inaccessible archive
really useful?
42. Example problem:
“We want to run out pipeline across 100TB of data
currently in EGA/SRA.”
We will need to de-stage the data to Sanger, and then run
the compute.
• Extra 0.5 PB of storage, 1000 cores of compute.
• 3 month lead time.
• ~$1.5M capex.
43. Cloud / Computable archives
Move the compute to the
data.
• Upload workload onto VMs.
• Put VMs on compute that is
“attached” to the data.
CPU CPU CPU CPU
CPU CPU CPU CPU
Federated between
centres Data
Data
• Grid software build on top of CPU CPU CPU CPU
CPU CPU CPU CPU
cloud components.
• Avoids scaling problems VM
VM
Data
inherent in putting everything Data
on one place.
44. Acknowledgements
Sanger EBI
• Phil Butcher Glenn Proctor
• James Beal Steve Keenan
• Pete Clapham
• Simon Kelley
• Gen-Tao Chiang
• Steve Searle
• Jan-Hinnerk Vogel
• Bronwen Aken