The primary requirements for OpenStack based clouds (public, private or hybrid) is that they must be massively scalable and highly available. There are a number of interrelated concepts which make the understanding and implementation of HA complex. The potential for not implementing HA correctly would be disastrous.
This session was presented at the OpenStack Meetup in Boston Feb 2014. We discussed interrelated concepts as a basis for implementing HA and examples of HA for MySQL, Rabbit MQ and the OpenStack APIs primarily using Keepalived, VRRP and HAProxy which will reinforce the concepts and show how to connect the dots.
OpenStack 운영을 통해 얻은 교훈을 공유합니다.
목차
1. TOAST 클라우드 지금의 모습
2. OpenStack 선택의 이유
3. 구성의 어려움과 극복 사례
4. 활용 사례
5. 풀어야 할 문제들
대상
- TOAST 클라우드를 사용하고 싶은 분
- WMI를 처음 들어보시는 분
Scylla on Kubernetes: Introducing the Scylla OperatorScyllaDB
The document introduces the Scylla Operator for Kubernetes, which provides a management layer for Scylla on Kubernetes. It addresses some limitations of using StatefulSets alone to run Scylla, such as safe scale down operations and tracking member identity. The operator implements the controller pattern with custom resources to deploy and manage Scylla clusters on Kubernetes. It handles tasks like cluster creation and scale up/down while addressing issues like local storage failures.
The document discusses high availability (HA) techniques in OpenStack. It covers HA concepts for both stateless and stateful services. For compute HA, it discusses server evacuation and instance migration without and with shared storage. It then covers different HA options for OpenStack controllers, including Pacemaker/Corosync/DRBD for active-passive HA and Galera for active-active MySQL HA. It also discusses using Keepalived, HAProxy and VRRP for load balancing and failover of API services. Finally, it presents a sample highly available OpenStack architecture and lists additional resources.
Using eBPF for High-Performance Networking in CiliumScyllaDB
The Cilium project is a popular networking solution for Kubernetes, based on eBPF. This talk uses eBPF code and demos to explore the basics of how Cilium makes network connections, and manipulates packets so that they can avoid traversing the kernel's built-in networking stack. You'll see how eBPF enables high-performance networking as well as deep network observability and security.
OVN provides virtual networking capabilities for Open vSwitch including logical switches, routers, security groups, and ACLs. It uses OVSDB to configure OVN components and provides native integration with OpenStack Neutron. OVN's architecture includes a northbound database for logical network definitions, a southbound database for physical mappings, and daemons like ovn-northd and ovn-controller that translate between the databases.
The primary requirements for OpenStack based clouds (public, private or hybrid) is that they must be massively scalable and highly available. There are a number of interrelated concepts which make the understanding and implementation of HA complex. The potential for not implementing HA correctly would be disastrous.
This session was presented at the OpenStack Meetup in Boston Feb 2014. We discussed interrelated concepts as a basis for implementing HA and examples of HA for MySQL, Rabbit MQ and the OpenStack APIs primarily using Keepalived, VRRP and HAProxy which will reinforce the concepts and show how to connect the dots.
OpenStack 운영을 통해 얻은 교훈을 공유합니다.
목차
1. TOAST 클라우드 지금의 모습
2. OpenStack 선택의 이유
3. 구성의 어려움과 극복 사례
4. 활용 사례
5. 풀어야 할 문제들
대상
- TOAST 클라우드를 사용하고 싶은 분
- WMI를 처음 들어보시는 분
Scylla on Kubernetes: Introducing the Scylla OperatorScyllaDB
The document introduces the Scylla Operator for Kubernetes, which provides a management layer for Scylla on Kubernetes. It addresses some limitations of using StatefulSets alone to run Scylla, such as safe scale down operations and tracking member identity. The operator implements the controller pattern with custom resources to deploy and manage Scylla clusters on Kubernetes. It handles tasks like cluster creation and scale up/down while addressing issues like local storage failures.
The document discusses high availability (HA) techniques in OpenStack. It covers HA concepts for both stateless and stateful services. For compute HA, it discusses server evacuation and instance migration without and with shared storage. It then covers different HA options for OpenStack controllers, including Pacemaker/Corosync/DRBD for active-passive HA and Galera for active-active MySQL HA. It also discusses using Keepalived, HAProxy and VRRP for load balancing and failover of API services. Finally, it presents a sample highly available OpenStack architecture and lists additional resources.
Using eBPF for High-Performance Networking in CiliumScyllaDB
The Cilium project is a popular networking solution for Kubernetes, based on eBPF. This talk uses eBPF code and demos to explore the basics of how Cilium makes network connections, and manipulates packets so that they can avoid traversing the kernel's built-in networking stack. You'll see how eBPF enables high-performance networking as well as deep network observability and security.
OVN provides virtual networking capabilities for Open vSwitch including logical switches, routers, security groups, and ACLs. It uses OVSDB to configure OVN components and provides native integration with OpenStack Neutron. OVN's architecture includes a northbound database for logical network definitions, a southbound database for physical mappings, and daemons like ovn-northd and ovn-controller that translate between the databases.
Producer Performance Tuning for Apache KafkaJiangjie Qin
Kafka is well known for high throughput ingestion. However, to get the best latency characteristics without compromising on throughput and durability, we need to tune Kafka. In this talk, we share our experiences to achieve the optimal combination of latency, throughput and durability for different scenarios.
This document summarizes CloudStack networking features and architecture. CloudStack provides orchestration of network services like IPAM, DNS, firewalls, load balancing, and VPN. It supports various network isolation techniques including VLANs, L3 isolation, and overlay networks. The CloudStack virtual router provides default network services, and external devices can also be integrated. CloudStack networking supports advanced configurations including multi-tier networks, bring your own services, and software defined networking.
The document provides an introduction and overview of Prometheus for monitoring systems. It discusses key concepts such as metric types, exposing metrics through HTTP endpoints, querying metrics with PromQL, and creating alerts. It also covers visualizing data with Grafana, using ServiceMonitors to monitor Kubernetes services, and leveraging exporters to get additional system metrics. The overall document serves as a tutorial for getting started with Prometheus and its monitoring capabilities.
Systems Monitoring with Prometheus (Devops Ireland April 2015)Brian Brazil
Monitoring means many things to many people. This talk looks at Systems Monitoring, that is how to keep an eye on a given system and use this as part of overall management of a system. This talk will cover Why one monitors, What to monitor, How to monitor, the general design of a monitoring system and how Prometheus is a good fit for this in terms of instrumentation, consoles, alerts, general system health and sanity.
Prometheus is a next-generation monitoring system publicly announced earlier this year, developed by companies including SoundCloud, locals Boxever and Docker. Since launch there has been wide-spread interest, and many community contributions.
For more information see http://prometheus.io or http://www.boxever.com/tag/monitoring
Full recorded presentation at https://www.youtube.com/watch?v=2UfAgCSKPZo for Tetrate Tech Talks on 2022/05/13.
Envoy's support for Kafka protocol, in form of broker-filter and mesh-filter.
Contents:
- overview of Kafka (usecases, partitioning, producer/consumer, protocol);
- proxying Kafka (non-Envoy specific);
- proxying Kafka with Envoy;
- handling Kafka protocol in Envoy;
- Kafka-broker-filter for per-connection proxying;
- Kafka-mesh-filter to provide front proxy for multiple Kafka clusters.
References:
- https://adam-kotwasinski.medium.com/deploying-envoy-and-kafka-8aa7513ec0a0
- https://adam-kotwasinski.medium.com/kafka-mesh-filter-in-envoy-a70b3aefcdef
The document discusses using Senlin, an OpenStack clustering service, to provide autoscaling capabilities for multicloud platforms. Senlin allows for managing clusters of nodes across different cloud providers and includes features like load balancing, auto-healing, and scaling policies. It describes how Senlin was implemented at a company to provide a centralized autoscaling solution across OpenStack and VMware cloud environments. Some drawbacks of Senlin are also outlined, along with potential future work like multi-region clusters and global load balancing.
Presentation at Strata Data Conference 2018, New York
The controller is the brain of Apache Kafka. A big part of what the controller does is to maintain the consistency of the replicas and determine which replica can be used to serve the clients, especially during individual broker failure.
Jun Rao outlines the main data flow in the controller—in particular, when a broker fails, how the controller automatically promotes another replica as the leader to serve the clients, and when a broker is started, how the controller resumes the replication pipeline in the restarted broker.
Jun then describes recent improvements to the controller that allow it to handle certain edge cases correctly and increase its performance, which allows for more partitions in a Kafka cluster.
This document discusses Zero touch on-premise storage infrastructure with OpenStack Cinder. It describes Viettel's IT infrastructure with mixed storage resources and the challenges of managing it. The solution presented uses OpenStack Cinder and additional tools to automate the management and provisioning of block storage for bare metal servers and OpenStack instances. This removes manual configuration steps and improves performance by pre-zoning storage connections. The goal is to make volume management simpler and allow adding new storage resources without additional configuration through the unified management solution.
Wars of MySQL Cluster ( InnoDB Cluster VS Galera ) Mydbops
MySQL Clustering over InnoDB engines has grown a lot over the last decade. Galera began working with InnoDB early and then Group Replication came to the environment later, where the features are now rich and robust. This presentation offers a technical comparison of both of them.
Optimizing Kubernetes Resource Requests/Limits for Cost-Efficiency and Latenc...Henning Jacobs
Kubernetes has the concept of resource requests and limits. Pods get scheduled on the nodes based on their requests and optionally limited in how much of the resource they can consume. Understanding and optimizing resource requests/limits is crucial both for reducing resource "slack" and ensuring application performance/low-latency. This talk shows our approach to monitoring and optimizing Kubernetes resources for 80+ clusters to achieve cost-efficiency and reducing impact for latency-critical applications. All shown tools are Open Source and can be applied to most Kubernetes deployments.
Redis Cluster is an approach to distributing Redis across multiple nodes. Key-value pairs are partitioned across nodes using consistent hashing on the key's hash slot. Nodes specialize as masters or slaves of data partitions for redundancy. Clients can query any node, which will redirect requests as needed. Nodes continuously monitor each other to detect and address failures, maintaining availability as long as each partition has at least one responsive node. The redis-trib tool is used to setup, check, resize, and repair clusters as needed.
Prometheus is an open-source monitoring system that collects metrics from configured targets, stores time series data, and allows users to query and alert on that data. It is designed for dynamic cloud environments and has built-in service discovery integration. Core features include simplicity, efficiency, a dimensional data model, the PromQL query language, and service discovery.
Prometheus was recently accepted into the Cloud Native Computing Foundation, making it the second project after Kubernetes to be given their blessing and acknowledging that Prometheus and Kubernetes make an awesome combination. In this talk we'll cover common patterns for running Prometheus on Kubernetes, how to monitor services on Kubernetes, and some cool tips and hacks to ensure you get the most out of your Prometheus + Kubernetes deployment.
MoP(MQTT on Pulsar) - a Powerful Tool for Apache Pulsar in IoT - Pulsar Summi...StreamNative
MQTT (Message Queuing Telemetry Transport,) is a message protocol based on the pub/sub model with the advantages of compact message structure, low resource consumption, and high efficiency, which is suitable for IoT applications with low bandwidth and unstable network environments.
This session will introduce MQTT on Pulsar, which allows developers users of MQTT transport protocol to use Apache Pulsar. I will share the architecture, principles and future planning of MoP, to help you understand Apache Pulsar's capabilities and practices in the IoT industry.
Kafka Tutorial - Introduction to Apache Kafka (Part 1)Jean-Paul Azar
Why is Kafka so fast? Why is Kafka so popular? Why Kafka? This slide deck is a tutorial for the Kafka streaming platform. This slide deck covers Kafka Architecture with some small examples from the command line. Then we expand on this with a multi-server example to demonstrate failover of brokers as well as consumers. Then it goes through some simple Java client examples for a Kafka Producer and a Kafka Consumer. We have also expanded on the Kafka design section and added references. The tutorial covers Avro and the Schema Registry as well as advance Kafka Producers.
This document discusses running MySQL on Kubernetes with Percona Kubernetes Operators. It provides an introduction to cloud native applications and Kubernetes. It then discusses the benefits and challenges of running MySQL on Kubernetes compared to database-as-a-service options. It introduces Percona Kubernetes Operators for MySQL, which help manage and configure MySQL deployments on Kubernetes. Finally, it discusses how to deploy MySQL with the Percona Kubernetes Operators, including prerequisites, connectivity, architecture, high availability, and monitoring.
Kubernetes Observability with Prometheus by ExampleThomas Riley
This talk was given at Cloud Native + Kubernetes Manchester, July 2019.
Prometheus is quickly becoming the de factor open-source monitoring and alerting tool for Kubernetes. Through a series of live demos I will explain how to deploy Prometheus into Kubernetes and make use of it for monitoring Kubernetes. I will also demonstrate how to successfully run Prometheus in HA with the Thanos project and how to store years worth of metrics without requiring heaps of CPU, memory and storage for Prometheus.
Docker and Go: why did we decide to write Docker in Go?Jérôme Petazzoni
Docker is currently one of the most popular Go projects. After a (quick) Docker intro, we will discuss why we picked Go, and how it turned out for us.
We tried to list all the drawbacks and minor inconveniences that we met while developing Docker; not to complain about Go, but to give the audience an idea of what to expect. Depending on your project, those drawbacks could be minor inconveniences or showstoppers; we thought you would want to know about them to help you to make the right choice!
A study and practice of OpenStack release Kilo HA deployment. The Kilo document has some errors, and it's hardly find a detailed document to describe how to deploy a HA cloud based on Kilo release. Hope this slides can provide some clues.
Azure Event Hubs - Behind the Scenes With Kasun Indrasiri | Current 2022HostedbyConfluent
Azure Event Hubs - Behind the Scenes With Kasun Indrasiri | Current 2022
Azure Event Hubs is a hyperscale PaaS event stream broker with protocol support for HTTP, AMQP, and Apache Kafka RPC that accepts and forwards several trillion (!) events per day and is available in all global Azure regions. This session is a look behind the curtain where we dive deep into the architecture of Event Hubs and look at the Event Hubs cluster model, resource isolation, and storage strategies and also review some performance figures.
Producer Performance Tuning for Apache KafkaJiangjie Qin
Kafka is well known for high throughput ingestion. However, to get the best latency characteristics without compromising on throughput and durability, we need to tune Kafka. In this talk, we share our experiences to achieve the optimal combination of latency, throughput and durability for different scenarios.
This document summarizes CloudStack networking features and architecture. CloudStack provides orchestration of network services like IPAM, DNS, firewalls, load balancing, and VPN. It supports various network isolation techniques including VLANs, L3 isolation, and overlay networks. The CloudStack virtual router provides default network services, and external devices can also be integrated. CloudStack networking supports advanced configurations including multi-tier networks, bring your own services, and software defined networking.
The document provides an introduction and overview of Prometheus for monitoring systems. It discusses key concepts such as metric types, exposing metrics through HTTP endpoints, querying metrics with PromQL, and creating alerts. It also covers visualizing data with Grafana, using ServiceMonitors to monitor Kubernetes services, and leveraging exporters to get additional system metrics. The overall document serves as a tutorial for getting started with Prometheus and its monitoring capabilities.
Systems Monitoring with Prometheus (Devops Ireland April 2015)Brian Brazil
Monitoring means many things to many people. This talk looks at Systems Monitoring, that is how to keep an eye on a given system and use this as part of overall management of a system. This talk will cover Why one monitors, What to monitor, How to monitor, the general design of a monitoring system and how Prometheus is a good fit for this in terms of instrumentation, consoles, alerts, general system health and sanity.
Prometheus is a next-generation monitoring system publicly announced earlier this year, developed by companies including SoundCloud, locals Boxever and Docker. Since launch there has been wide-spread interest, and many community contributions.
For more information see http://prometheus.io or http://www.boxever.com/tag/monitoring
Full recorded presentation at https://www.youtube.com/watch?v=2UfAgCSKPZo for Tetrate Tech Talks on 2022/05/13.
Envoy's support for Kafka protocol, in form of broker-filter and mesh-filter.
Contents:
- overview of Kafka (usecases, partitioning, producer/consumer, protocol);
- proxying Kafka (non-Envoy specific);
- proxying Kafka with Envoy;
- handling Kafka protocol in Envoy;
- Kafka-broker-filter for per-connection proxying;
- Kafka-mesh-filter to provide front proxy for multiple Kafka clusters.
References:
- https://adam-kotwasinski.medium.com/deploying-envoy-and-kafka-8aa7513ec0a0
- https://adam-kotwasinski.medium.com/kafka-mesh-filter-in-envoy-a70b3aefcdef
The document discusses using Senlin, an OpenStack clustering service, to provide autoscaling capabilities for multicloud platforms. Senlin allows for managing clusters of nodes across different cloud providers and includes features like load balancing, auto-healing, and scaling policies. It describes how Senlin was implemented at a company to provide a centralized autoscaling solution across OpenStack and VMware cloud environments. Some drawbacks of Senlin are also outlined, along with potential future work like multi-region clusters and global load balancing.
Presentation at Strata Data Conference 2018, New York
The controller is the brain of Apache Kafka. A big part of what the controller does is to maintain the consistency of the replicas and determine which replica can be used to serve the clients, especially during individual broker failure.
Jun Rao outlines the main data flow in the controller—in particular, when a broker fails, how the controller automatically promotes another replica as the leader to serve the clients, and when a broker is started, how the controller resumes the replication pipeline in the restarted broker.
Jun then describes recent improvements to the controller that allow it to handle certain edge cases correctly and increase its performance, which allows for more partitions in a Kafka cluster.
This document discusses Zero touch on-premise storage infrastructure with OpenStack Cinder. It describes Viettel's IT infrastructure with mixed storage resources and the challenges of managing it. The solution presented uses OpenStack Cinder and additional tools to automate the management and provisioning of block storage for bare metal servers and OpenStack instances. This removes manual configuration steps and improves performance by pre-zoning storage connections. The goal is to make volume management simpler and allow adding new storage resources without additional configuration through the unified management solution.
Wars of MySQL Cluster ( InnoDB Cluster VS Galera ) Mydbops
MySQL Clustering over InnoDB engines has grown a lot over the last decade. Galera began working with InnoDB early and then Group Replication came to the environment later, where the features are now rich and robust. This presentation offers a technical comparison of both of them.
Optimizing Kubernetes Resource Requests/Limits for Cost-Efficiency and Latenc...Henning Jacobs
Kubernetes has the concept of resource requests and limits. Pods get scheduled on the nodes based on their requests and optionally limited in how much of the resource they can consume. Understanding and optimizing resource requests/limits is crucial both for reducing resource "slack" and ensuring application performance/low-latency. This talk shows our approach to monitoring and optimizing Kubernetes resources for 80+ clusters to achieve cost-efficiency and reducing impact for latency-critical applications. All shown tools are Open Source and can be applied to most Kubernetes deployments.
Redis Cluster is an approach to distributing Redis across multiple nodes. Key-value pairs are partitioned across nodes using consistent hashing on the key's hash slot. Nodes specialize as masters or slaves of data partitions for redundancy. Clients can query any node, which will redirect requests as needed. Nodes continuously monitor each other to detect and address failures, maintaining availability as long as each partition has at least one responsive node. The redis-trib tool is used to setup, check, resize, and repair clusters as needed.
Prometheus is an open-source monitoring system that collects metrics from configured targets, stores time series data, and allows users to query and alert on that data. It is designed for dynamic cloud environments and has built-in service discovery integration. Core features include simplicity, efficiency, a dimensional data model, the PromQL query language, and service discovery.
Prometheus was recently accepted into the Cloud Native Computing Foundation, making it the second project after Kubernetes to be given their blessing and acknowledging that Prometheus and Kubernetes make an awesome combination. In this talk we'll cover common patterns for running Prometheus on Kubernetes, how to monitor services on Kubernetes, and some cool tips and hacks to ensure you get the most out of your Prometheus + Kubernetes deployment.
MoP(MQTT on Pulsar) - a Powerful Tool for Apache Pulsar in IoT - Pulsar Summi...StreamNative
MQTT (Message Queuing Telemetry Transport,) is a message protocol based on the pub/sub model with the advantages of compact message structure, low resource consumption, and high efficiency, which is suitable for IoT applications with low bandwidth and unstable network environments.
This session will introduce MQTT on Pulsar, which allows developers users of MQTT transport protocol to use Apache Pulsar. I will share the architecture, principles and future planning of MoP, to help you understand Apache Pulsar's capabilities and practices in the IoT industry.
Kafka Tutorial - Introduction to Apache Kafka (Part 1)Jean-Paul Azar
Why is Kafka so fast? Why is Kafka so popular? Why Kafka? This slide deck is a tutorial for the Kafka streaming platform. This slide deck covers Kafka Architecture with some small examples from the command line. Then we expand on this with a multi-server example to demonstrate failover of brokers as well as consumers. Then it goes through some simple Java client examples for a Kafka Producer and a Kafka Consumer. We have also expanded on the Kafka design section and added references. The tutorial covers Avro and the Schema Registry as well as advance Kafka Producers.
This document discusses running MySQL on Kubernetes with Percona Kubernetes Operators. It provides an introduction to cloud native applications and Kubernetes. It then discusses the benefits and challenges of running MySQL on Kubernetes compared to database-as-a-service options. It introduces Percona Kubernetes Operators for MySQL, which help manage and configure MySQL deployments on Kubernetes. Finally, it discusses how to deploy MySQL with the Percona Kubernetes Operators, including prerequisites, connectivity, architecture, high availability, and monitoring.
Kubernetes Observability with Prometheus by ExampleThomas Riley
This talk was given at Cloud Native + Kubernetes Manchester, July 2019.
Prometheus is quickly becoming the de factor open-source monitoring and alerting tool for Kubernetes. Through a series of live demos I will explain how to deploy Prometheus into Kubernetes and make use of it for monitoring Kubernetes. I will also demonstrate how to successfully run Prometheus in HA with the Thanos project and how to store years worth of metrics without requiring heaps of CPU, memory and storage for Prometheus.
Docker and Go: why did we decide to write Docker in Go?Jérôme Petazzoni
Docker is currently one of the most popular Go projects. After a (quick) Docker intro, we will discuss why we picked Go, and how it turned out for us.
We tried to list all the drawbacks and minor inconveniences that we met while developing Docker; not to complain about Go, but to give the audience an idea of what to expect. Depending on your project, those drawbacks could be minor inconveniences or showstoppers; we thought you would want to know about them to help you to make the right choice!
A study and practice of OpenStack release Kilo HA deployment. The Kilo document has some errors, and it's hardly find a detailed document to describe how to deploy a HA cloud based on Kilo release. Hope this slides can provide some clues.
Azure Event Hubs - Behind the Scenes With Kasun Indrasiri | Current 2022HostedbyConfluent
Azure Event Hubs - Behind the Scenes With Kasun Indrasiri | Current 2022
Azure Event Hubs is a hyperscale PaaS event stream broker with protocol support for HTTP, AMQP, and Apache Kafka RPC that accepts and forwards several trillion (!) events per day and is available in all global Azure regions. This session is a look behind the curtain where we dive deep into the architecture of Event Hubs and look at the Event Hubs cluster model, resource isolation, and storage strategies and also review some performance figures.
Sharing High-Performance Interconnects Across Multiple Virtual Machinesinside-BigData.com
In this deck from the Stanford HPC Conference, Mohan Potheri from VMware presents: Sharing High-Performance Interconnects Across Multiple Virtual Machines.
"Virtualized devices offer maximum flexibility: sharing of hardware between virtual machines, the use of VMware vMotion to handle migration and take snapshots. However, when performance is the most critical requirement there are other options. VMware Direct Path I/O delivers excellent performance, but only for a single virtual machine. Single root I/O virtualization (SR-IOV), on the other hand, offers the performance of pass-through mode while allowing devices to be shared by multiple virtual machines.
This session introduces SR-IOV, explains how it is enabled in VMware vSphere, and provides details of specific use cases that important for machine learning and high-performance computing. It includes performance comparisons that demonstrate the benefits of SR-IOV and information on how to configure and tune these configurations."
Watch the video: https://youtu.be/-iYYmsBw8SU
Learn more: https://www.vmware.com
and
http://hpcadvisorycouncil.com
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
HPC and cloud distributed computing, as a journeyPeter Clapham
Introducing an internal cloud brings new paradigms, tools and infrastructure management. When placed alongside traditional HPC the new opportunities are significant But getting to the new world with micro-services, autoscaling and autodialing is a journey that cannot be achieved in a single step.
OpenNebulaConf2017EU: Hyper converged infrastructure with OpenNebula and Ceph...OpenNebula Project
Hyperconvergence is one of the big topics in datacenters at the moment. But is it more than an old wine in new bottles? Why we at Runtastic built an hyperconverged datacenter based on Opennebula with Ceph and what we learned.
YouTube: https://youtu.be/50Z4bmevTpg
Tips Tricks and Tactics with Cells and Scaling OpenStack - May, 2015Belmiro Moreira
The document discusses using cells in OpenStack to scale cloud infrastructure across multiple geographic locations. Key points include using cells to distribute OpenStack compute services around Australia, with over 6000 users, 700 hypervisors, and 30,000 cores spread across 8 sites and 14 cells. It also discusses strategies for operating, upgrading, and scheduling across multiple cells.
This document provides an overview of how to create your own cloud using Apache CloudStack. It discusses the key characteristics of clouds, different cloud service and deployment models supported by CloudStack, and the core components that make up a CloudStack deployment including zones, pods, clusters, primary and secondary storage, virtual routers, hypervisors, and the management server. The document also touches on CloudStack's networking, security, high availability, resource allocation, and usage accounting features.
The document provides an agenda and overview of a session on hacking Apache CloudStack. The agenda includes introductions, a session on introducing CloudStack, and a hands-on session with DevCloud. The overview discusses what CloudStack is, how it works as an orchestration platform for IAAS clouds, its architecture and core components, and how users can consume and manage resources through it.
Sanger, upcoming Openstack for Bio-informaticiansPeter Clapham
Delivery of a new Bio-informatics infrastructure at the Wellcome Trust Sanger Center. We include how to programatically create, manage and provide providence for images used both at Sanger and elsewhere using open source tools and continuous integration.
[발표자료] 오픈소스 Pacemaker 활용한 zabbix 이중화 방안(w/ Zabbix Korea Community) 동현 김
Pacemaker is an open source high-availability and load balancing stack for Linux. It provides unified configuration and management of cluster resources through tools like pcs and crmsh. The document discusses installing and configuring the necessary Pacemaker components - Pacemaker, Corosync, SBD and fence agents - on nodes to enable high availability of a Zabbix server through active-passive replication and fencing. A virtual IP will be configured to fail over between nodes when the active node fails.
The document discusses OpenStack high availability (HA), performance tuning, and troubleshooting techniques. It covers HA concepts in OpenStack, including compute and controller node HA. It then discusses performance tuning and analyzing OpenStack logs for troubleshooting. It provides details on HA solutions for various OpenStack components like Nova, Glance, Keystone, Swift, Cinder and Neutron. It also covers techniques for optimizing performance in OpenStack like kernel tuning, huge pages, and KSM. Finally, it lists some common log locations for troubleshooting various OpenStack services.
The document provides information on Juniper SRX platform updates, including:
1) vSRX updates - The virtual firewall platform now supports up to 80G FW throughput on a single server and 100G vSRX was announced. Support for VMware 5.5+SRIOV and features parity with physical SRX firewalls.
2) Physical SRX updates - New SRX3xx and SRX550 series for branches up to 500 users. The SRX1500 provides high performance networking and security for enterprise edge and data center edge. The SRX5400 supports advanced software security services.
3) Software updates - Sky ATP cloud-based malware analysis and SRX User Identity REST API.
CloudStack is an open-source cloud computing platform that provides infrastructure as a service. It supports various hypervisors and storage types, and allows for multi-tenancy and isolation between users/organizations. CloudStack provides tools for provisioning, managing, and monitoring virtual machines and cloud infrastructure resources.
The document provides a technical overview of the CLIMB OpenStack cloud including hardware, software, and configuration details. The key components are IBM servers and storage, xCAT for provisioning, SaltStack for configuration management, OpenStack for cloud services, and IBM Spectrum Scale (formerly GPFS) for parallel file storage. Spectrum Scale is integrated with OpenStack components like Cinder, Glance, and Swift to provide scalable block and object storage.
Apache Kafka is a distributed streaming platform used for building real-time data pipelines and streaming apps. It provides a unified, scalable, and durable platform for handling real-time data feeds. Kafka works by accepting streams of records from one or more producers and organizing them into topics. It allows both storing and forwarding of these streams to consumers. Producers write data to topics which are replicated across clusters for fault tolerance. Consumers can then read the data from the topics in the order it was produced. Major companies like LinkedIn, Yahoo, Twitter, and Netflix use Kafka for applications like metrics, logging, stream processing and more.
This document discusses various topics related to high-performance packet processing, including software and hardware switches/routers, ASICs, kernel bypass techniques like DPDK and netmap, and frameworks like pf_ring and FD.IO. It compares x86 CPUs to ASICs, discusses challenges of packet processing on general-purpose OSes, and explains how techniques like zero-copy, RDMA, and TCP offloading can improve performance.
Secure Your Containers: What Network Admins Should Know When Moving Into Prod...Cynthia Thomas
This session offers techniques for securing Docker containers and hosts using open source network virtualization technologies to implement microsegmentation. Come learn real tips and tricks that you can apply to keep your production environment secure.
Cloud orchestration major tools comparisionRavi Kiran
Cloud Orchestration major tools comparison (including history, installation, market share, integration with other public cloud system for each tool) For any clarification contact kiran79@techgeek.co.in
Mirantis - Continuous Deployment of Infrastructure, Platform, and Application...Jakub Pavlik
Plan to run applications in containers on Kubernetes, but those applications or components are not yet ready to be deployed as micro-services? Then you need a unified platform that can orchestrate and continuously deliver infrastructure, container platform and applications services across bare-metal, containers and VMs so you can easily leverage the benefits of containers incrementally as legacy applications are adapted to micro-services architectures.
Mirantis Cloud Platform (MCP) enables continuous delivery of the infrastructure, and in this demo we will show you how Mirantis Cloud Platform (MCP) enables continuous delivery of application workloads on top of its VM and Bare-Metal IaaS (OpenStack) and Container Services (Kubernetes) resources, backed by a single SDN (OpenContrail) implementation. We will utilize Spinnaker as an open-source multi-cloud continuous delivery platform for releasing software on MCP.
OpenStack Journey in Tieto Elastic CloudJakub Pavlik
This document summarizes Tieto's journey with OpenStack, beginning with initial proof of concept deployments using installer-based distributions like PackStack and Fuel. Tieto found these distributions difficult to maintain and customize. They later implemented tcp cloud using Mirantis OpenStack and OpenContrail, which provided better automation and configuration management. While this worked well initially, managing over 30 management VMs became challenging. Tieto is now moving to Mirantis Cloud Platform, which runs OpenStack services on Kubernetes for improved flexibility, lifecycle management, and reduced management overhead.
Kubernetes SDN performance and architectureJakub Pavlik
The document discusses Kubernetes SDN performance and architecture. It provides an overview of Calico and OpenContrail, two common SDN solutions for Kubernetes. Calico uses standard protocols and has no overhead but lacks L2 capabilities. OpenContrail provides advanced networking features through an overlay but has more overhead and complexity. Both solutions were tested on a 100 node cluster and their performance and production considerations are examined. The presentation concludes with a comparison of Calico and OpenContrail and examples of multi-cloud architectures using them.
1) The document summarizes tcp cloud's work with OpenContrail since the last meetup, including releasing version Mk.20 of their enterprise IT solution based on OpenContrail 2.21 and OpenStack Kilo.
2) It describes tcp cloud's reference architecture and automated deployment of dev clusters using SaltStack formulas and Heat templates.
3) The document discusses tcp cloud's experience upgrading AVG Technologies' OpenContrail deployment from version 2.1 to 2.21 and issues encountered upgrading OpenStack releases from Juno to Kilo and Kilo to Liberty.
OpenStack Ousts vCenter for DevOps and Unites IT Silos at AVG Technologies Jakub Pavlik
tcp cloud & AVG User Story.
Does your IT department’s left hand talk to the right hand? Finally ours does at http://www.avg.com/eu-en/homepage! This is the story of OpenStack as our salvation, and important lessons learned in technology and IT politics.
Our appsdev team’s devops abilities were being held ransom on vCenter, so we wanted public cloud agility for dev/test/staging. With the help of our IT partner…Full session details here: http://awe.sm/r9Ekr
tcp cloud presentation at OpenContrail Meetup in May 2015 Vancouver about OpenStack/OpenContrail implementations, Juno integration and SaltStack announcement.
This document summarizes Jakub Pavlik's experience deploying Contrail virtual networks with OpenStack at tcp cloud. Key points include:
- Contrail 1.05 was deployed with Havana on CentOS using SaltStack instead of Fabric for configuration management.
- The deployment consisted of 3 OpenStack controllers, 2 Contrail controllers, and used HA technologies like Corosync/Pacemaker and Galera for high availability.
- Some issues were encountered with Fabric not providing true HA and missing options for cinder/glance backends. BGP peering also required restoration after control node failures.
Things to Consider When Choosing a Website Developer for your Website | FODUUFODUU
Choosing the right website developer is crucial for your business. This article covers essential factors to consider, including experience, portfolio, technical skills, communication, pricing, reputation & reviews, cost and budget considerations and post-launch support. Make an informed decision to ensure your website meets your business goals.
AI-Powered Food Delivery Transforming App Development in Saudi Arabia.pdfTechgropse Pvt.Ltd.
In this blog post, we'll delve into the intersection of AI and app development in Saudi Arabia, focusing on the food delivery sector. We'll explore how AI is revolutionizing the way Saudi consumers order food, how restaurants manage their operations, and how delivery partners navigate the bustling streets of cities like Riyadh, Jeddah, and Dammam. Through real-world case studies, we'll showcase how leading Saudi food delivery apps are leveraging AI to redefine convenience, personalization, and efficiency.
Taking AI to the Next Level in Manufacturing.pdfssuserfac0301
Read Taking AI to the Next Level in Manufacturing to gain insights on AI adoption in the manufacturing industry, such as:
1. How quickly AI is being implemented in manufacturing.
2. Which barriers stand in the way of AI adoption.
3. How data quality and governance form the backbone of AI.
4. Organizational processes and structures that may inhibit effective AI adoption.
6. Ideas and approaches to help build your organization's AI strategy.
How to Get CNIC Information System with Paksim Ga.pptxdanishmna97
Pakdata Cf is a groundbreaking system designed to streamline and facilitate access to CNIC information. This innovative platform leverages advanced technology to provide users with efficient and secure access to their CNIC details.
Generating privacy-protected synthetic data using Secludy and MilvusZilliz
During this demo, the founders of Secludy will demonstrate how their system utilizes Milvus to store and manipulate embeddings for generating privacy-protected synthetic data. Their approach not only maintains the confidentiality of the original data but also enhances the utility and scalability of LLMs under privacy constraints. Attendees, including machine learning engineers, data scientists, and data managers, will witness first-hand how Secludy's integration with Milvus empowers organizations to harness the power of LLMs securely and efficiently.
Driving Business Innovation: Latest Generative AI Advancements & Success StorySafe Software
Are you ready to revolutionize how you handle data? Join us for a webinar where we’ll bring you up to speed with the latest advancements in Generative AI technology and discover how leveraging FME with tools from giants like Google Gemini, Amazon, and Microsoft OpenAI can supercharge your workflow efficiency.
During the hour, we’ll take you through:
Guest Speaker Segment with Hannah Barrington: Dive into the world of dynamic real estate marketing with Hannah, the Marketing Manager at Workspace Group. Hear firsthand how their team generates engaging descriptions for thousands of office units by integrating diverse data sources—from PDF floorplans to web pages—using FME transformers, like OpenAIVisionConnector and AnthropicVisionConnector. This use case will show you how GenAI can streamline content creation for marketing across the board.
Ollama Use Case: Learn how Scenario Specialist Dmitri Bagh has utilized Ollama within FME to input data, create custom models, and enhance security protocols. This segment will include demos to illustrate the full capabilities of FME in AI-driven processes.
Custom AI Models: Discover how to leverage FME to build personalized AI models using your data. Whether it’s populating a model with local data for added security or integrating public AI tools, find out how FME facilitates a versatile and secure approach to AI.
We’ll wrap up with a live Q&A session where you can engage with our experts on your specific use cases, and learn more about optimizing your data workflows with AI.
This webinar is ideal for professionals seeking to harness the power of AI within their data management systems while ensuring high levels of customization and security. Whether you're a novice or an expert, gain actionable insights and strategies to elevate your data processes. Join us to see how FME and AI can revolutionize how you work with data!
Monitoring and Managing Anomaly Detection on OpenShift.pdfTosin Akinosho
Monitoring and Managing Anomaly Detection on OpenShift
Overview
Dive into the world of anomaly detection on edge devices with our comprehensive hands-on tutorial. This SlideShare presentation will guide you through the entire process, from data collection and model training to edge deployment and real-time monitoring. Perfect for those looking to implement robust anomaly detection systems on resource-constrained IoT/edge devices.
Key Topics Covered
1. Introduction to Anomaly Detection
- Understand the fundamentals of anomaly detection and its importance in identifying unusual behavior or failures in systems.
2. Understanding Edge (IoT)
- Learn about edge computing and IoT, and how they enable real-time data processing and decision-making at the source.
3. What is ArgoCD?
- Discover ArgoCD, a declarative, GitOps continuous delivery tool for Kubernetes, and its role in deploying applications on edge devices.
4. Deployment Using ArgoCD for Edge Devices
- Step-by-step guide on deploying anomaly detection models on edge devices using ArgoCD.
5. Introduction to Apache Kafka and S3
- Explore Apache Kafka for real-time data streaming and Amazon S3 for scalable storage solutions.
6. Viewing Kafka Messages in the Data Lake
- Learn how to view and analyze Kafka messages stored in a data lake for better insights.
7. What is Prometheus?
- Get to know Prometheus, an open-source monitoring and alerting toolkit, and its application in monitoring edge devices.
8. Monitoring Application Metrics with Prometheus
- Detailed instructions on setting up Prometheus to monitor the performance and health of your anomaly detection system.
9. What is Camel K?
- Introduction to Camel K, a lightweight integration framework built on Apache Camel, designed for Kubernetes.
10. Configuring Camel K Integrations for Data Pipelines
- Learn how to configure Camel K for seamless data pipeline integrations in your anomaly detection workflow.
11. What is a Jupyter Notebook?
- Overview of Jupyter Notebooks, an open-source web application for creating and sharing documents with live code, equations, visualizations, and narrative text.
12. Jupyter Notebooks with Code Examples
- Hands-on examples and code snippets in Jupyter Notebooks to help you implement and test anomaly detection models.
Essentials of Automations: The Art of Triggers and Actions in FMESafe Software
In this second installment of our Essentials of Automations webinar series, we’ll explore the landscape of triggers and actions, guiding you through the nuances of authoring and adapting workspaces for seamless automations. Gain an understanding of the full spectrum of triggers and actions available in FME, empowering you to enhance your workspaces for efficient automation.
We’ll kick things off by showcasing the most commonly used event-based triggers, introducing you to various automation workflows like manual triggers, schedules, directory watchers, and more. Plus, see how these elements play out in real scenarios.
Whether you’re tweaking your current setup or building from the ground up, this session will arm you with the tools and insights needed to transform your FME usage into a powerhouse of productivity. Join us to discover effective strategies that simplify complex processes, enhancing your productivity and transforming your data management practices with FME. Let’s turn complexity into clarity and make your workspaces work wonders!
In his public lecture, Christian Timmerer provides insights into the fascinating history of video streaming, starting from its humble beginnings before YouTube to the groundbreaking technologies that now dominate platforms like Netflix and ORF ON. Timmerer also presents provocative contributions of his own that have significantly influenced the industry. He concludes by looking at future challenges and invites the audience to join in a discussion.
Climate Impact of Software Testing at Nordic Testing DaysKari Kakkonen
My slides at Nordic Testing Days 6.6.2024
Climate impact / sustainability of software testing discussed on the talk. ICT and testing must carry their part of global responsibility to help with the climat warming. We can minimize the carbon footprint but we can also have a carbon handprint, a positive impact on the climate. Quality characteristics can be added with sustainability, and then measured continuously. Test environments can be used less, and in smaller scale and on demand. Test techniques can be used in optimizing or minimizing number of tests. Test automation can be used to speed up testing.
Let's Integrate MuleSoft RPA, COMPOSER, APM with AWS IDP along with Slackshyamraj55
Discover the seamless integration of RPA (Robotic Process Automation), COMPOSER, and APM with AWS IDP enhanced with Slack notifications. Explore how these technologies converge to streamline workflows, optimize performance, and ensure secure access, all while leveraging the power of AWS IDP and real-time communication via Slack notifications.
CAKE: Sharing Slices of Confidential Data on BlockchainClaudio Di Ciccio
Presented at the CAiSE 2024 Forum, Intelligent Information Systems, June 6th, Limassol, Cyprus.
Synopsis: Cooperative information systems typically involve various entities in a collaborative process within a distributed environment. Blockchain technology offers a mechanism for automating such processes, even when only partial trust exists among participants. The data stored on the blockchain is replicated across all nodes in the network, ensuring accessibility to all participants. While this aspect facilitates traceability, integrity, and persistence, it poses challenges for adopting public blockchains in enterprise settings due to confidentiality issues. In this paper, we present a software tool named Control Access via Key Encryption (CAKE), designed to ensure data confidentiality in scenarios involving public blockchains. After outlining its core components and functionalities, we showcase the application of CAKE in the context of a real-world cyber-security project within the logistics domain.
Paper: https://doi.org/10.1007/978-3-031-61000-4_16
UiPath Test Automation using UiPath Test Suite series, part 6DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 6. In this session, we will cover Test Automation with generative AI and Open AI.
UiPath Test Automation with generative AI and Open AI webinar offers an in-depth exploration of leveraging cutting-edge technologies for test automation within the UiPath platform. Attendees will delve into the integration of generative AI, a test automation solution, with Open AI advanced natural language processing capabilities.
Throughout the session, participants will discover how this synergy empowers testers to automate repetitive tasks, enhance testing accuracy, and expedite the software testing life cycle. Topics covered include the seamless integration process, practical use cases, and the benefits of harnessing AI-driven automation for UiPath testing initiatives. By attending this webinar, testers, and automation professionals can gain valuable insights into harnessing the power of AI to optimize their test automation workflows within the UiPath ecosystem, ultimately driving efficiency and quality in software development processes.
What will you get from this session?
1. Insights into integrating generative AI.
2. Understanding how this integration enhances test automation within the UiPath platform
3. Practical demonstrations
4. Exploration of real-world use cases illustrating the benefits of AI-driven test automation for UiPath
Topics covered:
What is generative AI
Test Automation with generative AI and Open AI.
UiPath integration with generative AI
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
3. High Availability vs. Disaster Recovery
High Availability = fault detection & correction procedures to maximize
availability of critical services and applications, often in an automated
fashion.
Disaster Recovery = process of preparing for recovery or continuation of
technology infrastructure critical to an organization after a natural or
human-induced disaster.
High Availability ≠ Disaster Recovery!
4. Four types of HA in an OpenStack Cloud
Physical infrastructure
OpenStack
Control services
VMs
OpenStack Compute
Applications
Compute Controller
Network Controller
Database
Message Queue
Storage
....
Physical nodes
Physical network
Physical storage
Hypervisor
Host OS
….
Service Resiliency
QoS Cost
Transparency
Data Integrity
…..
Virtual Machine
Virtual Network
Virtual Storage
VM Mobility
…
9. Stateless services
• There is no dependency between requests
• For example APIs: Nova, Keystone, Glance, Cinder, etc.
Stateful services
• An action typically compromises multiple requests
• For example: MySQL, RabbitMQ, etc.
OpenStack High Availability Concepts
Active/Passive
• Redundant instances of stateless services are load balanced
• For Stateful services a replacement resource can be brought
online
Active/Active
• Redundant instances of stateless services are load balanced
• Stateful services are managed in such a way that services are
redundant, and that all instances have and identical state.
10. Corosync
• Totem single-ring ordering and membership
protocol
• UDP and InfiniBand based messaging, quorum,
and cluster membership to Pacemaker
Pacemaker
• High availability and load balancing stack for the
Linux platform.
• Interacts with applications through Resource
Agents (RA)
HAProxy
• Load Balancing and Proxying for HTTP and TCP
Applications
• Works over multiple connections
• Used to load balance API services
Corosync, Pacemaker and HAProxy
11. • MySQL patched for wsrep
(Write Set REPlication)
• Active/active multi-master
topology
• Read and write to any cluster
node
• True parallel replication, in row
level
• No slave lag or integrity issues
MySQL Galera
Synchronous multi-master cluster technology for MySQL/InnoDB
12. Sample OpenStack HA architecture
Stateful
• Cinder Volume
• Neutron L3, DHCP agents
• Ceilometer central agent
• RabbitMQ
Stateless
• Neutron Server
• OpenStack APIs
• Apache web server
• Nova Scheduler
• Cinder Scheduler
Neutron agents
(Active)
Neutron agents
(Hot Standby)
16. Shared Storage
• Live migration – just RAM memory
• Hypervisor Evacuation – The instance will be booted from
same disk and data will be preserved
• CEPH, Gluster, NFS, Samba, GFS
Non-Shared Storage
• Block Live Migration – disk and RAM
• Hypervisor Evacuation – the instance will be booted from a
new disk, but will preserve the configuration, e.g. id, name,
uuid
• Standard filesystem EXT4, etc.
Non-Shared/Shared Storage filesystem
17. • Instance boots from volume
• iSCSI/FC direct mapping to instance
• Enable Live Migration
• Cinder Backends
• LVM Driver
• Default linux iSCSI server
• Vendor software plugins
• Gluster, CEPH, VMware VMDK driver
• Vendor storage plugins
• EMC VNX, IBM Storwize, Solid Fire, etc.
Block Storage - Cinder
18. Problems
• Routing on Linux server (max. bandwith approximately 3-4
Gbits)
• Limited distribution between more network nodes
• East-West and North-South communication through network
node
High Availability
• Pacemaker&Corosync
• Keepalived VRRP
• DVR + VRRP – should be in Juno release
Networking - Vanilla Neutron L3 agent
19. Examples
• Juniper OpenContrail, VMware NSX, SDN PLUMgrid
Advantages against Neutron L3 agent
• North-South communication on network devices (iBGP,
MLPSoverGRE)
• East-West communication directly between compute nodes
• Higher bandwidth (9.7 Gbits per 10Gbits port)
High Availability
• iBGP peering into two routers
• Native HA implemented inside of network devices
Networking – Vendor SDN Controller plugins
20. OpenStack HA
TCP VPC
MySQL RabbitMQ
Openstack
Controller
GALERA
Zookee
per
Cassandra
Contrail
Database
Contrail Config
with Analytics &
WebUI
Contrail
Control
Zookee
per
Cassandra
Contrail
Database
MySQL RabbitMQ
Openstack
Controller
MySQL RabbitMQ
Openstack
Controller
Zookee
per
Cassandra
Contrail
Database
Contrail
Control
Contrail Config
with Analytics &
WebUI
HAProxy HAProxy HAProxy
VIP
Bond Interface
Pacemaker
Corosync
Contrail Config
with Analytics &
WebUI
Pacemaker
Corosync