The document proposes a reconfiguration service for distributed real-time Java applications. It introduces extensions to support real-time parameters like scheduling, release, memory and network parameters. The service allows adding and removing tasks from distributed Java VMs. It evaluates a priority-based partitioning policy for task allocation. The policy checks feasibility and assigns priorities. Evaluation shows reconfiguration time increases with number of tasks and VMs. Asynchronous events have the highest cost. Ongoing work aims to support different scheduling policies and adaptation using real-time OSGi.
Design and Implementation of Incremental Cooperative Rebalancingconfluent
Watch this talk here: https://www.confluent.io/online-talks/design-and-implementation-of-incremental-cooperative-rebalancing-on-demand
Since its initial release, the Kafka group membership protocol has offered Connect, Streams and Consumer applications an ingenious and robust way to balance resources among distributed processes. The process of rebalancing, as it’s widely known, allows Kafka APIs to define an embedded protocol for load balancing within the group membership protocol itself.
Until now, rebalancing has been working under the simple assumption that every time a new group generation is created, the members join after first releasing all of their resources, getting a whole new load assignment by the time the new group is formed. This allows Kafka APIs to provide task fault-tolerance and elasticity on top of the group membership protocol.
However, due to its side-effects on multi-tenancy and scalability this simple approach in rebalancing, also known as stop-the-world effect, is limiting larger scale deployments. Because of stop-the-world, application tasks get interrupted only for most of them to receive the same resources after rebalancing. In this technical deep dive, we’ll discuss the proposition of Incremental Cooperative Rebalancing as a way to alleviate stop-the-world and optimize rebalancing in Kafka APIs.
This talk will cover:
-The internals of Incremental Cooperative Rebalancing
-Uses cases that benefit from Incremental Cooperative Rebalancing
-Implementation in Kafka Connect
-Performance results in Kafka Connect clusters
Load Balancing In Distributed ComputingRicha Singh
Load Balancing In Distributed Computing
The goal of the load balancing algorithms is to maintain the load to each processing element such that all the processing elements become neither overloaded nor idle that means each processing element ideally has equal load at any moment of time during execution to obtain the maximum performance (minimum execution time) of the system
Pivotal HAWQ is a high performance SQL engine on top of Hadoop. It support SQL 92, multi-way joins and has one of the best query processing engines on top of Hadoop. This presentation explains some of the design principles behind HAWQ HA and offers insight into how it works with Hadoop HA.
A Review of Storage Specific Solutions for Providing Quality of Service in St...Editor IJCATR
Predictable storage performance is a vital requirement for promising performance of the applications utilizing it and it is the
systems administrators’ job to ensure that storage performance meets the requirements of the applications. Most storage solutions are
able to virtualize the amount of storage presented to the host in a flexible way, but the same storage devices have no QOS features
.Storage level agreements provided by storage devices do not provide predictability in service delivery due to the absence of
prioritization (QOS) mechanisms in storage devices. This paper reviews some of the storage specific solutions developed to implement
quality of service in storage area networks.
Design and Implementation of Incremental Cooperative Rebalancingconfluent
Watch this talk here: https://www.confluent.io/online-talks/design-and-implementation-of-incremental-cooperative-rebalancing-on-demand
Since its initial release, the Kafka group membership protocol has offered Connect, Streams and Consumer applications an ingenious and robust way to balance resources among distributed processes. The process of rebalancing, as it’s widely known, allows Kafka APIs to define an embedded protocol for load balancing within the group membership protocol itself.
Until now, rebalancing has been working under the simple assumption that every time a new group generation is created, the members join after first releasing all of their resources, getting a whole new load assignment by the time the new group is formed. This allows Kafka APIs to provide task fault-tolerance and elasticity on top of the group membership protocol.
However, due to its side-effects on multi-tenancy and scalability this simple approach in rebalancing, also known as stop-the-world effect, is limiting larger scale deployments. Because of stop-the-world, application tasks get interrupted only for most of them to receive the same resources after rebalancing. In this technical deep dive, we’ll discuss the proposition of Incremental Cooperative Rebalancing as a way to alleviate stop-the-world and optimize rebalancing in Kafka APIs.
This talk will cover:
-The internals of Incremental Cooperative Rebalancing
-Uses cases that benefit from Incremental Cooperative Rebalancing
-Implementation in Kafka Connect
-Performance results in Kafka Connect clusters
Load Balancing In Distributed ComputingRicha Singh
Load Balancing In Distributed Computing
The goal of the load balancing algorithms is to maintain the load to each processing element such that all the processing elements become neither overloaded nor idle that means each processing element ideally has equal load at any moment of time during execution to obtain the maximum performance (minimum execution time) of the system
Pivotal HAWQ is a high performance SQL engine on top of Hadoop. It support SQL 92, multi-way joins and has one of the best query processing engines on top of Hadoop. This presentation explains some of the design principles behind HAWQ HA and offers insight into how it works with Hadoop HA.
A Review of Storage Specific Solutions for Providing Quality of Service in St...Editor IJCATR
Predictable storage performance is a vital requirement for promising performance of the applications utilizing it and it is the
systems administrators’ job to ensure that storage performance meets the requirements of the applications. Most storage solutions are
able to virtualize the amount of storage presented to the host in a flexible way, but the same storage devices have no QOS features
.Storage level agreements provided by storage devices do not provide predictability in service delivery due to the absence of
prioritization (QOS) mechanisms in storage devices. This paper reviews some of the storage specific solutions developed to implement
quality of service in storage area networks.
Reactive Java: Promises and Streams with Reakt (JavaOne talk 2016)Rick Hightower
see labs at https://github.com/advantageous/j1-talks-2016
Import based on PDF. This is from our JavaOne Talk 2016 on Reakt, reactive Java programming with promises, circuit breakers, and streams. Reakt is a reactive Java lib that provides promises, streams, and a reactor to handle asynchronous call coordination. It was influenced by the design of promises in ES6. You want to async-call serviceA and then serviceB, take the results of serviceA and serviceB, and then call serviceC. Then, based on the results of call C, call D or E and then return the results to the original caller. Calls to A, B, C, D, and E are all async calls, and none should take longer than 10 seconds. If they do, then return a timeout to the original caller. The whole async call sequence should time out in 20 seconds if it does not complete and should also check for circuit breakers and provide back pressure feedback so the system does not have cascading failures. Learn more in this session.
Continuent Tungsten - Scalable Saa S Data Managementguest2e11e8
The key needs of SaaS vendors include:
i) managing multi-tenant architectures with shared DBMS, ii) maintaining customer SLAs for uptime and performance and iii) optimized, efficient operations.
The key benefits Continuent Tungsten offers SaaS vendors are:
i) high availability and protection from data loss, ii) simple, efficient cluster management and iii) enable complex database topologies.
Tungsten offers high-availability, database cluster management and management of complex topologies for multi-tenant architectures.
Tungsten high availability and data protection features include maintaining live copies with data consistency checking and tightly coupled backup/restore integration with cluster management tools.
Tungsten cluster management allows SaaS vendors to migrate customers and perform system upgrades without downtime, thus enabling these maintenance operations during normal business hours.
Tungsten also enables complex replication topologies, including data filtering and data archiving strategies, maintaining extra data copies for data-marts, routing different customers to different DBMS copies, and providing cross-site multi-master replication.
Extending Spark Streaming to Support Complex Event ProcessingOh Chan Kwon
In this talk, we introduce the extensions of Spark Streaming to support (1) SQL-based query processing and (2) elastic-seamless resource allocation. First, we explain the methods of supporting window queries and query chains. As we know, last year, Grace Huang and Jerry Shao introduced the concept of “StreamSQL” that can process streaming data with SQL-like queries by adapting SparkSQL to Spark Streaming. However, we made advances in supporting complex event processing (CEP) based on their efforts. In detail, we implemented the sliding window concept to support a time-based streaming data processing at the SQL level. Here, to reduce the aggregation time of large windows, we generate an efficient query plan that computes the partial results by evaluating only the data entering or leaving the window and then gets the current result by merging the previous one and the partial ones. Next, to support query chains, we made the result of a query over streaming data be a table by adding the “insert into” query. That is, it allows us to apply stream queries to the results of other ones. Second, we explain the methods of allocating resources to streaming applications dynamically, which enable the applications to meet a given deadline. As the rate of incoming events varies over time, resources allocated to applications need to be adjusted for high resource utilization. However, the current Spark's resource allocation features are not suitable for streaming applications. That is, the resources allocated will not be freed when new data are arriving continuously to the streaming applications even though the quantity of the new ones is very small. In order to resolve the problem, we consider their resource utilization. If the utilization is low, we choose victim nodes to be killed. Then, we do not feed new data into the victims to prevent a useless recovery issuing when they are killed. Accordingly, we can scale-in/-out the resources seamlessly.
Some Domestic & Global projects executed by our team.Vasant Bhanushali
We have a team of Enterprise Infrastructure Solutions Delivery professionals , to execute projects across the globe. #Enterpriseinfrastructuresolutions #servicedelivery
Tungsten Connector / Proxy is truly the secret sauce for the Tungsten Clustering solution. Watch this webinar to learn how the Tungsten Connector enables zero-downtime MySQL maintenance via the manual switch operation, and gain an understanding of the various configuration options for doing local reads in remote composite clusters.
AGENDA
- Review the cluster architecture
- Understand the role of the Connector
- Describe Connector deployment best practices (app, dedicated with lb, db with lb)
- Explore zero-downtime MySQL maintenance using the manual role switch procedure
- Learn about Connector routing patterns inside a composite cluster
- Illustrate a manual site switch
- Explain read affinity and the vast performance improvement of local reads
- Examine Connector multi-cluster support
Adaptive Replication for Elastic Data Stream ProcessingZbigniew Jerzak
A major challenge for cloud-based systems is to be fault tolerant so as to cope with an increasing probability of faults in cloud environments. This is especially true for in-memory computing solutions like data stream processing systems, where a single host failure might result in an unrecoverable information loss.
In state of the art data streaming systems either active replication or upstream backup are applied to ensure fault tolerance, which have a high resource overhead or a high recovery time respectively. This paper combines these two fault tolerance mechanisms in one system to minimize the number of violations of a user-defined recovery time threshold and to reduce the overall resource consumption compared to active replication. The system switches for individual operators between both replication techniques dynamically based on the current workload characteristics. Our approach is implemented as an extension of an elastic data stream processing engine, which is able to reduce the number of used hosts due to the smaller replication overhead. Based on a real-world evaluation we show that our system is able to reduce the resource usage by up to 19% compared to an active replication scheme.
Optimization of Continuous Queries in Federated Database and Stream Processin...Zbigniew Jerzak
The constantly increasing number of connected devices and sensors results in increasing volume and velocity of sensor-based streaming data. Traditional approaches for processing high velocity sensor data rely on stream processing engines. However, the increasing complexity of continuous queries executed on top of high velocity data has resulted in growing demand for federated systems composed of data stream processing engines and database engines. One of major challenges for such systems is to devise the optimal query execution plan to maximize the throughput of continuous queries.
In this paper we present a general framework for federated database and stream processing systems, and introduce the design and implementation of a cost-based optimizer for optimizing relational continuous queries in such systems. Our optimizer uses characteristics of continuous queries and source data streams to devise an optimal placement for each operator of a continuous query. This fine level of optimization, combined with the estimation of the feasibility of query plans, allows our optimizer to devise query plans which result in 8 times higher throughput as compared to the baseline approach which uses only stream processing engines. Moreover, our experimental results showed that even for simple queries, a hybrid execution plan can result in 4 times and 1.6 times higher throughput than a pure stream processing engine plan and a pure database engine plan, respectively.
Unified Stream Processing at Scale with Apache Samza by Jake Maes at Big Data...Big Data Spain
The shift to stream processing at LinkedIn has accelerated over the past few years. We now have over 200 Samza applications in production processing more than 260B events per day.
https://www.bigdataspain.org/2017/talk/apache-samza-jake-maes
Big Data Spain 2017
November 16th - 17th Kinépolis Madrid
High-Speed Reactive Microservices - trials and tribulationsRick Hightower
Covers how we built a set of high-speed reactive microservices and maximized cloud/hardware costs while meeting objectives in resilience and scalability. This has more notes attached as it is based on the ppt not the PDF.
GE IOT Predix Time Series & Data Ingestion Service using Apache Apex (Hadoop)Apache Apex
This presentation will introduce usage of Apache Apex for Time Series & Data Ingestion Service by General Electric Internet of things Predix platform. Apache Apex is a native Hadoop data in motion platform that is being used by customers for both streaming as well as batch processing. Common use cases include ingestion into Hadoop, streaming analytics, ETL, database off-loads, alerts and monitoring, machine model scoring, etc.
Abstract: Predix is an General Electric platform for Internet of Things. It helps users develop applications that connect industrial machines with people through data and analytics for better business outcomes. Predix offers a catalog of services that provide core capabilities required by industrial internet applications. We will deep dive into Predix Time Series and Data Ingestion services leveraging fast, scalable, highly performant, and fault tolerant capabilities of Apache Apex.
Speakers:
- Venkatesh Sivasubramanian, Sr Staff Software Engineer, GE Predix & Committer of Apache Apex
- Pramod Immaneni, PPMC member of Apache Apex, and DataTorrent Architect
In the French FedEx company we used Prometheus to monitor the infrastructure. It hosts a CQRS Architecture composed with Kafka, Spark, Cassandra, ElasticSearch, and microservices APIs in scala.
This presentation is about using Prometheus in production, you will see why we choosed Prometheus, how we integrated it, configured it and what kind of insights we extracted from the whole infrastructure.
In addition, you will see how Prometheus changed our way of working, how we implemented self-healing based on Prometheus, how we configured systemd to trigger AlertManager API, integration with slack and other cool stuffs.
VMworld 2013: Part 2: How to Build a Self-Healing Data Center with vCenter Or...VMworld
VMworld 2013
Nicholas Colyer, Catamaran RX
Dan Mitchell, VMware
Learn more about VMworld and register at http://www.vmworld.com/index.jspa?src=socmed-vmworld-slideshare
Reactive Java: Promises and Streams with Reakt (JavaOne talk 2016)Rick Hightower
see labs at https://github.com/advantageous/j1-talks-2016
Import based on PDF. This is from our JavaOne Talk 2016 on Reakt, reactive Java programming with promises, circuit breakers, and streams. Reakt is a reactive Java lib that provides promises, streams, and a reactor to handle asynchronous call coordination. It was influenced by the design of promises in ES6. You want to async-call serviceA and then serviceB, take the results of serviceA and serviceB, and then call serviceC. Then, based on the results of call C, call D or E and then return the results to the original caller. Calls to A, B, C, D, and E are all async calls, and none should take longer than 10 seconds. If they do, then return a timeout to the original caller. The whole async call sequence should time out in 20 seconds if it does not complete and should also check for circuit breakers and provide back pressure feedback so the system does not have cascading failures. Learn more in this session.
Continuent Tungsten - Scalable Saa S Data Managementguest2e11e8
The key needs of SaaS vendors include:
i) managing multi-tenant architectures with shared DBMS, ii) maintaining customer SLAs for uptime and performance and iii) optimized, efficient operations.
The key benefits Continuent Tungsten offers SaaS vendors are:
i) high availability and protection from data loss, ii) simple, efficient cluster management and iii) enable complex database topologies.
Tungsten offers high-availability, database cluster management and management of complex topologies for multi-tenant architectures.
Tungsten high availability and data protection features include maintaining live copies with data consistency checking and tightly coupled backup/restore integration with cluster management tools.
Tungsten cluster management allows SaaS vendors to migrate customers and perform system upgrades without downtime, thus enabling these maintenance operations during normal business hours.
Tungsten also enables complex replication topologies, including data filtering and data archiving strategies, maintaining extra data copies for data-marts, routing different customers to different DBMS copies, and providing cross-site multi-master replication.
Extending Spark Streaming to Support Complex Event ProcessingOh Chan Kwon
In this talk, we introduce the extensions of Spark Streaming to support (1) SQL-based query processing and (2) elastic-seamless resource allocation. First, we explain the methods of supporting window queries and query chains. As we know, last year, Grace Huang and Jerry Shao introduced the concept of “StreamSQL” that can process streaming data with SQL-like queries by adapting SparkSQL to Spark Streaming. However, we made advances in supporting complex event processing (CEP) based on their efforts. In detail, we implemented the sliding window concept to support a time-based streaming data processing at the SQL level. Here, to reduce the aggregation time of large windows, we generate an efficient query plan that computes the partial results by evaluating only the data entering or leaving the window and then gets the current result by merging the previous one and the partial ones. Next, to support query chains, we made the result of a query over streaming data be a table by adding the “insert into” query. That is, it allows us to apply stream queries to the results of other ones. Second, we explain the methods of allocating resources to streaming applications dynamically, which enable the applications to meet a given deadline. As the rate of incoming events varies over time, resources allocated to applications need to be adjusted for high resource utilization. However, the current Spark's resource allocation features are not suitable for streaming applications. That is, the resources allocated will not be freed when new data are arriving continuously to the streaming applications even though the quantity of the new ones is very small. In order to resolve the problem, we consider their resource utilization. If the utilization is low, we choose victim nodes to be killed. Then, we do not feed new data into the victims to prevent a useless recovery issuing when they are killed. Accordingly, we can scale-in/-out the resources seamlessly.
Some Domestic & Global projects executed by our team.Vasant Bhanushali
We have a team of Enterprise Infrastructure Solutions Delivery professionals , to execute projects across the globe. #Enterpriseinfrastructuresolutions #servicedelivery
Tungsten Connector / Proxy is truly the secret sauce for the Tungsten Clustering solution. Watch this webinar to learn how the Tungsten Connector enables zero-downtime MySQL maintenance via the manual switch operation, and gain an understanding of the various configuration options for doing local reads in remote composite clusters.
AGENDA
- Review the cluster architecture
- Understand the role of the Connector
- Describe Connector deployment best practices (app, dedicated with lb, db with lb)
- Explore zero-downtime MySQL maintenance using the manual role switch procedure
- Learn about Connector routing patterns inside a composite cluster
- Illustrate a manual site switch
- Explain read affinity and the vast performance improvement of local reads
- Examine Connector multi-cluster support
Adaptive Replication for Elastic Data Stream ProcessingZbigniew Jerzak
A major challenge for cloud-based systems is to be fault tolerant so as to cope with an increasing probability of faults in cloud environments. This is especially true for in-memory computing solutions like data stream processing systems, where a single host failure might result in an unrecoverable information loss.
In state of the art data streaming systems either active replication or upstream backup are applied to ensure fault tolerance, which have a high resource overhead or a high recovery time respectively. This paper combines these two fault tolerance mechanisms in one system to minimize the number of violations of a user-defined recovery time threshold and to reduce the overall resource consumption compared to active replication. The system switches for individual operators between both replication techniques dynamically based on the current workload characteristics. Our approach is implemented as an extension of an elastic data stream processing engine, which is able to reduce the number of used hosts due to the smaller replication overhead. Based on a real-world evaluation we show that our system is able to reduce the resource usage by up to 19% compared to an active replication scheme.
Optimization of Continuous Queries in Federated Database and Stream Processin...Zbigniew Jerzak
The constantly increasing number of connected devices and sensors results in increasing volume and velocity of sensor-based streaming data. Traditional approaches for processing high velocity sensor data rely on stream processing engines. However, the increasing complexity of continuous queries executed on top of high velocity data has resulted in growing demand for federated systems composed of data stream processing engines and database engines. One of major challenges for such systems is to devise the optimal query execution plan to maximize the throughput of continuous queries.
In this paper we present a general framework for federated database and stream processing systems, and introduce the design and implementation of a cost-based optimizer for optimizing relational continuous queries in such systems. Our optimizer uses characteristics of continuous queries and source data streams to devise an optimal placement for each operator of a continuous query. This fine level of optimization, combined with the estimation of the feasibility of query plans, allows our optimizer to devise query plans which result in 8 times higher throughput as compared to the baseline approach which uses only stream processing engines. Moreover, our experimental results showed that even for simple queries, a hybrid execution plan can result in 4 times and 1.6 times higher throughput than a pure stream processing engine plan and a pure database engine plan, respectively.
Unified Stream Processing at Scale with Apache Samza by Jake Maes at Big Data...Big Data Spain
The shift to stream processing at LinkedIn has accelerated over the past few years. We now have over 200 Samza applications in production processing more than 260B events per day.
https://www.bigdataspain.org/2017/talk/apache-samza-jake-maes
Big Data Spain 2017
November 16th - 17th Kinépolis Madrid
High-Speed Reactive Microservices - trials and tribulationsRick Hightower
Covers how we built a set of high-speed reactive microservices and maximized cloud/hardware costs while meeting objectives in resilience and scalability. This has more notes attached as it is based on the ppt not the PDF.
GE IOT Predix Time Series & Data Ingestion Service using Apache Apex (Hadoop)Apache Apex
This presentation will introduce usage of Apache Apex for Time Series & Data Ingestion Service by General Electric Internet of things Predix platform. Apache Apex is a native Hadoop data in motion platform that is being used by customers for both streaming as well as batch processing. Common use cases include ingestion into Hadoop, streaming analytics, ETL, database off-loads, alerts and monitoring, machine model scoring, etc.
Abstract: Predix is an General Electric platform for Internet of Things. It helps users develop applications that connect industrial machines with people through data and analytics for better business outcomes. Predix offers a catalog of services that provide core capabilities required by industrial internet applications. We will deep dive into Predix Time Series and Data Ingestion services leveraging fast, scalable, highly performant, and fault tolerant capabilities of Apache Apex.
Speakers:
- Venkatesh Sivasubramanian, Sr Staff Software Engineer, GE Predix & Committer of Apache Apex
- Pramod Immaneni, PPMC member of Apache Apex, and DataTorrent Architect
In the French FedEx company we used Prometheus to monitor the infrastructure. It hosts a CQRS Architecture composed with Kafka, Spark, Cassandra, ElasticSearch, and microservices APIs in scala.
This presentation is about using Prometheus in production, you will see why we choosed Prometheus, how we integrated it, configured it and what kind of insights we extracted from the whole infrastructure.
In addition, you will see how Prometheus changed our way of working, how we implemented self-healing based on Prometheus, how we configured systemd to trigger AlertManager API, integration with slack and other cool stuffs.
VMworld 2013: Part 2: How to Build a Self-Healing Data Center with vCenter Or...VMworld
VMworld 2013
Nicholas Colyer, Catamaran RX
Dan Mitchell, VMware
Learn more about VMworld and register at http://www.vmworld.com/index.jspa?src=socmed-vmworld-slideshare
Mike Guthrie - Revamping Your 10 Year Old Nagios InstallationNagios
Mike Guthrie - Revamping Your 10 Year Old Nagios Installation - Mike Merideth from VictorOps talks about the challenges of
sharing responsibility for monitoring in the DevOps world. Learn several strategies for keeping your configuration correct,
consistent, and up-to-date when several people are working on it.
WinOps Conf 2016 - Michael Greene - Release PipelinesWinOps Conf
There are benefits to be gained when patterns and practices from developer techniques are applied to operations. Notably, a fully automated solution where infrastructure is managed as code and all changes are automatically validated before reaching production. This is a process shift that is recognized among industry innovators. For organizations already leveraging these processes, it should be clear how to leverage Microsoft platforms. For organizations that are new to the topic, it should be clear how to bring this process to your environment and what it means to your organizational culture. This presentation explains the components of a Release Pipeline for configuration as code, the value to operations, and solutions that are used when designing a new Release Pipeline architecture.
VMworld 2013: Moving Enterprise Application Dev/Test to VMware’s Internal Pri...VMworld
VMworld 2013
Venkat Gopalakrishnan, VMware
Kurt Milne, VMware
Learn more about VMworld and register at http://www.vmworld.com/index.jspa?src=socmed-vmworld-slideshare
Oracle Drivers configuration for High AvailabilityLudovico Caldara
... is it a developer's job?
UCP, GridLink, TAF, AC, TAC, FAN… The configuration of Oracle Drivers for application high availability is not an easy job. The developers often care about the minimal working configuration, while the DBAs are busy with the operations. In this session I will try to demystify application server’s connectivity to the database and give a direction toward the highest availability, using Real Application Clusters and new Oracle features like TAC and CMAN TDM.
Stop Worrying and Keep Querying, Using Automated Multi-Region Disaster RecoveryDoKC
Stop Worrying and Keep Querying, Using Automated Multi-Region Disaster Recovery - Shivani Gupta, Elotl & Sergey Pronin, Percona
Disaster Recovery(DR) is critical for business continuity in the face of widespread outages taking down entire data centers or cloud provider regions. DR relies on deployment to multiple locations, data replication, monitoring for failure and failover. The process is typically manual involving several moving parts, and, even in the best case, involves some downtime for end-users. A multi-cluster K8s control plane presents the opportunity to automate the DR setup as well as the failure detection and failover. Such automation can dramatically reduce RTO and improve availability for end-users. This talk (and demo) describes one such setup using the open source Percona Operator for PostgreSQL and a multi-cluster K8s orchestrator. The orchestrator will use policy driven placement to replicate the entire workload on multiple clusters (in different regions), detect failure using pluggable logic, and do failover processing by promoting the standby as well as redirecting application traffic
VMworld 2013: Moving Enterprise Application Dev/Test to VMware’s Internal Pri...VMworld
VMworld 2013
Thirumalesh Reddy, VMware
Padmaja Vrudhula, VMware
Learn more about VMworld and register at http://www.vmworld.com/index.jspa?src=socmed-vmworld-slideshare
Continuous Delivery: How RightScale Releases WeeklyRightScale
Continuous delivery may be a natural for greenfield workloads, but how do you take an existing seven-year-old SaaS application and move from multi-month to weekly release cycles? Find out how our team — developers, QA, and ops — worked together to change our process and along the way changed their own ideas of what was possible.
Similar to Towards a real-time reconfiguration service for distributed Java (20)
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
JMeter webinar - integration with InfluxDB and GrafanaRTTS
Watch this recorded webinar about real-time monitoring of application performance. See how to integrate Apache JMeter, the open-source leader in performance testing, with InfluxDB, the open-source time-series database, and Grafana, the open-source analytics and visualization application.
In this webinar, we will review the benefits of leveraging InfluxDB and Grafana when executing load tests and demonstrate how these tools are used to visualize performance metrics.
Length: 30 minutes
Session Overview
-------------------------------------------
During this webinar, we will cover the following topics while demonstrating the integrations of JMeter, InfluxDB and Grafana:
- What out-of-the-box solutions are available for real-time monitoring JMeter tests?
- What are the benefits of integrating InfluxDB and Grafana into the load testing stack?
- Which features are provided by Grafana?
- Demonstration of InfluxDB and Grafana using a practice web application
To view the webinar recording, go to:
https://www.rttsweb.com/jmeter-integration-webinar
UiPath Test Automation using UiPath Test Suite series, part 4DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 4. In this session, we will cover Test Manager overview along with SAP heatmap.
The UiPath Test Manager overview with SAP heatmap webinar offers a concise yet comprehensive exploration of the role of a Test Manager within SAP environments, coupled with the utilization of heatmaps for effective testing strategies.
Participants will gain insights into the responsibilities, challenges, and best practices associated with test management in SAP projects. Additionally, the webinar delves into the significance of heatmaps as a visual aid for identifying testing priorities, areas of risk, and resource allocation within SAP landscapes. Through this session, attendees can expect to enhance their understanding of test management principles while learning practical approaches to optimize testing processes in SAP environments using heatmap visualization techniques
What will you get from this session?
1. Insights into SAP testing best practices
2. Heatmap utilization for testing
3. Optimization of testing processes
4. Demo
Topics covered:
Execution from the test manager
Orchestrator execution result
Defect reporting
SAP heatmap example with demo
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
PHP Frameworks: I want to break free (IPC Berlin 2024)Ralf Eggert
In this presentation, we examine the challenges and limitations of relying too heavily on PHP frameworks in web development. We discuss the history of PHP and its frameworks to understand how this dependence has evolved. The focus will be on providing concrete tips and strategies to reduce reliance on these frameworks, based on real-world examples and practical considerations. The goal is to equip developers with the skills and knowledge to create more flexible and future-proof web applications. We'll explore the importance of maintaining autonomy in a rapidly changing tech landscape and how to make informed decisions in PHP development.
This talk is aimed at encouraging a more independent approach to using PHP frameworks, moving towards a more flexible and future-proof approach to PHP development.
State of ICS and IoT Cyber Threat Landscape Report 2024 previewPrayukth K V
The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development.
The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers:
State of global ICS asset and network exposure
Sectoral targets and attacks as well as the cost of ransom
Global APT activity, AI usage, actor and tactic profiles, and implications
Rise in volumes of AI-powered cyberattacks
Major cyber events in 2024
Malware and malicious payload trends
Cyberattack types and targets
Vulnerability exploit attempts on CVEs
Attacks on counties – USA
Expansion of bot farms – how, where, and why
In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East
Why are attacks on smart factories rising?
Cyber risk predictions
Axis of attacks – Europe
Systemic attacks in the Middle East
Download the full report from here:
https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/
"Impact of front-end architecture on development cost", Viktor TurskyiFwdays
I have heard many times that architecture is not important for the front-end. Also, many times I have seen how developers implement features on the front-end just following the standard rules for a framework and think that this is enough to successfully launch the project, and then the project fails. How to prevent this and what approach to choose? I have launched dozens of complex projects and during the talk we will analyze which approaches have worked for me and which have not.
Accelerate your Kubernetes clusters with Varnish CachingThijs Feryn
A presentation about the usage and availability of Varnish on Kubernetes. This talk explores the capabilities of Varnish caching and shows how to use the Varnish Helm chart to deploy it to Kubernetes.
This presentation was delivered at K8SUG Singapore. See https://feryn.eu/presentations/accelerate-your-kubernetes-clusters-with-varnish-caching-k8sug-singapore-28-2024 for more details.
FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf
Towards a real-time reconfiguration service for distributed Java
1. Towards a Reconfiguration Service
for Distributed Real-Time Java*
Pablo Basanta-Val
Marisol García-Valls
DREQUIEM LAB.
Universidad Carlos III de Madrid
* This work has been presented at REACTION Workshop 2012.
Collocated with IEEE RTSS 2012 Symposium at Puerto Rico
2. Outline
• Context & contribution
▫ Online scheduling & open systems
• The reconfiguration service
▫ Distributed Java approach
Overview of real-time parameters
Reconfiguration points
▫ Contributed Reconfiguration service
Proposed Extensions
Evaluation of a policy
• Conclusion and future work
2
REACTION WORKSHOP
3. Context & Introduction
• Many real-time systems consider admission control as a
offline line feature
▫ Clear advantages like exact algorithms
• However more and more systems are dynamic and
applications may appear and disappear
▫ Real-time CLOUD computing or real-time Open Systems
▫ How to make dynamic admission control efficient !
• Admission control for real-time performance
▫ Exact algorithms tend to be very costly !
3
REACTION
WORKSHOP
4. Context & Introduction
A real-time reconfiguration service in a
nutshell
• A Real-Time Reconfiguration Service for Distributed Real-
Time Java
▫ Idea:
To define a template for reconfiguration in distributed real-
time Java applications
It may customized for different applications
▫ Benefits:
Real-time Java community, and DRTSJ (The Distributed
Real-Time Specification for Java)
▫ Contribution:
Extensions to current APIS
Real-time Reconfiguration Service
Example global partitioner algorithm
with some performance results
REACTION
WORKSHOP
4
5. Distributed Real-Time
Java middleware
• Based on DREQUIEMI
▫ Layered approach for Distributed
Real-time Java
Infrastructure
Distribution
Common Services
Applications
• Currently, it consists of three main
technologies extending RTSJ
▫ Based on Java’s RMI, {RT}-CORBA ,
and TT tenets
5
REACTION
WORKSHOP
6. Distributed RT Java
QoS/RT configuration parameters
• Based on the model proposed by
RTSJ
• They characterize
▫ Application release pattern
▫ Scheduling specific parameters
▫ Memory parameters
▫ Processing group options
▫ Other internal decisions
Thread-pool
Connection Pool
Memory Area pool
QoS parameters in
distribution layer Meaning in DREQUIEMI
Scheduling Parameters
«Priority»
Priority used during the up-call at the
server
Release Parameters
«Release Time, Period,
Deadline, and Cost»
Invocation pattern at the server
according to the real-time scheduling
theory
Processing Group
Parameters
«Release Time, Period,
Deadline and Cost»
Group patterns at the server according
to the real-time scheduling theory
Memory Parameters
Parameters used by the garbage
collector at the server
Thread Pool
Thread pool used at the sever to
manage remote invocations
Connection Pool
Connection pool used at the client to
carry out the remote invocation
Memory Area Pool
Memory area pool used at the server to
accept remote invocations
port
The IP port on which the remote object
is accepting incoming messages
RemoteStub
The stub with the remote reference to
the remove object
EventCommonInt.
A remote object which allows
subscriptions and may be remotely
triggered.
6
REACTION
WORKSHOP
7. Reconfiguration
points
• Three reconfiguration points in
Distribute RT Java
▫ Resources, distribution, and
components
• Contributed reconfiguration applies to
resource management/rescheduling:
▫ Location of tasks in Remote Nodes
▫ Processor, network, and memory
REACTION
WORKSHOP
7
reconf
8. Reconfiguration Service
Introducing Network parameters
• RTSJ does not support the
network
▫ Centralized systems only!
• But it could be extended with it
▫ Adding new classes to the RTSJ
• Network parameters example:
▫ Priority for the IP packet (router)
▫ A maximum bit rate
▫ A complete RSVP stack, etc.
Scheduler
DistributedScheduler
Real-time Java
Distributed Real-time Java
uses MemoryParameters
SchedulingParametersuses
3ResourcesScheduler
NetworkParameters
uses
Serializable
ReconfigurationService
uses
Schedulable
RemoteSchedulable
8
REACTION
WORKSHOP
9. Reconfiguration Service
Distributed Scheduler
• Based on RTSJ’s
scheduler
• It allows invokers to
assign schedulables to
the task set
▫ Allocation
▫ Deallocation
• Each schedulable
defines:
▫ Release parameters,
▫ memory parameters
▫ Processing parameters
▫ Network Parameters
▫ Scheduling Parameters
01: public interface DistributedScheduler
02: extends java.rmi.Remote {
03: boolean addIfFeassible(
04: Schedulable s,
05: ReleaseParameters rs,
06: MemoryParameters mp,
07: ProcessingGroupParameters pgp,
08: NetworkParameter np )
09: throws RemoteException;
10: boolean removeIfFeassible(Schedulable s)
11: throws RemoteException;
12:}
9
REACTION
WORKSHOP
10. Reconfiguration Service
Reconfiguration Service
• To introduce reconfiguration on
the previous system
▫ The reconfiguration process is
modelled as a real-time
schedulable
• Based on that, the
reconfiguration service
▫ reconfiguration service =
DistributedScheduler
+ scheduling info for
DistributedScheduler
▫ The ReconfigurationService executes
on top of a scheduler.
01: public interface ReconfigurationService extends
02 DistributedScheduler, Schedulable{
03: MemoryParameters getMemoryParameters()
04: throws RemoteException;
05: ProcessingGroupParameters
06: getProcessingGroupParameters()
07: throws RemoteException;
08: ReleaseParameters
09: getReleaseParameters()
10: throws RemoteException;
11: Scheduler getScheduler();
12: throws RemoteException;
13: SchedulingParameters getSchedulingParameters();
14: throws RemoteException;
15: NetworkParameters getNetworkParameters();
16: throws RemoteException;
17: void setMemoryParameters(MemoryParameters mp)
18: throws RemoteException;
19: void setProcessingGroupParameters(
10
REACTION
WORKSHOP
11. Tested Scheduling Policy (1/4)
End-2-End Flow-Shop
• Flow-Shop
▫ Defined as a sequence of subtasks that execute in order (i.e. with
precedence constraints).
• Each end-to-end transaction (Γj | j 1..M) is defined as follows:
▫ Global deadline (Dj)
▫ A global period (Tj)
▫ A set of j
n schedulable segments {Sj
1….Sj
n}
A local execution priority (Pj
n) for each end-to-end transaction segment
(Sj
n).
A local worst-case execution time (Cj
n) for each segment (Sj
n).
11
REACTION
WORKSHOP
12. Tested Scheduling Policy (2/4)
Some simplifications
Client1/
2
Net1 serv1 Net2
Client
2/2
client router server
Local Priority (P)
and Cost (C)
Global Deadline (D) and period (T)
Local Priority (P)
and Cost (C)
Local Priority (P)
and Cost (C)
Local Priority (P)
and Cost (C)
Local Priority (P)
and Cost (C)
• Utilization based test
• The evaluation assumes
T=D and fixed priority (RMA)
assignment
▫ Each local deadline is calculated
• Each node runs a periodic
enforcer
▫ It allows analyzing each flux in a
node as local.
REACTION
WORKSHOP
12
13. Tested Scheduling Policy (3/4)
Partitioning tasks on different JVMs (T=D)
1. Global check based on global utilization
To check if it fits or not in advance
2. Sort all your tasks according to utilization
To avoid allocation anomalies
3. Assign a task to a Node (bin packing)
To select a node
4. Calculate your effective priority
To use the RTSJ 28 priorities only
5. Deploy tasks in each different node
Using a distributed scheduler
13
REACTION
WORKSHOP
1.Feasibility
2.Sort
3. Assign.
5. Remote
Deploy
4. Priority
calculus
Fail
Ok
14. Tested Scheduling Policy (4/4)
Additional code to be added to the policy
01: interface FlowShopSchedulables extends Schedulable{
02: Vector getAllShedulable(); //It returns all sch.
03: }
01: interface PriorityBasedReconfigurationService
02: extends ReconfigurationService{
03:}
• The tested policy requires:
▫ A new subclass for a schedulable entity
▫ A new subclass for the reconfiguration system
• In this article we provide:
▫ FlowShopSchedulables for each transaction
▫ PriorityBasedReconfigurationService to encapsulate the scheduler
REACTION
WORKSHOP
14
15. Evaluation
Infrastructure
• One central node to deploy
the system
▫ In charge of the
reconfiguration service
• DREQUIEMI
▫ For execution
• A real-time router
▫ IP priority enforcement
Manager
router
JVM1
JVM2
JVM N
A1 A2
...
796 Mhz
796 Mhz
Node 1
796 Mhz
300 Mhz
100 Mbps
DREQUIEMI
Jtime (TimeSys)
PriorityBased
ReconfigService
DREQUIEMI
Jtime
(TimeSys)
Distributed
Scheduler
addIfFeasible( )
796 Mhz
Node 2
Node N
A1
A2addIfFeasible( )
Tester Node
DREQUIEMI
Jtime (TimeSys)
PriorityBased
ReconfigService (stub)
DREQUIEMI
Jtime
(TimeSys)
Distributed
Scheduler
DREQUIEMI
Jtime
(TimeSys)
Distributed
Scheduler
15
REACTION
WORKSHOP
16. Cost tributaries
Tasks and VMs
• Cost increases with the number of
tasks and the number of processors
• Utilization, sorting, and assignment
(no deployment)
16
REACTION
WORKSHOP
1,E+00
1,E+01
1,E+02
1,E+03
1,E+04
1,E+05
1,E+06
1,E+07
64
32
16
8
4
2
1
Timeconsumed(us)
Processors (JVMs) and
Network available
Feasiblity test algorithm costs
4096Tasks
1024Tasks
64Tasks
1Task
796Mhz-100 Mbps
17. Cost tributaries
Type of schedulable entity
• Schedulable entities
▫ Real-time threads, remote objects, and
async-event handlers
• Each schedulable offers different
performance
▫ Asynchronous event handlers have the
worst allocation time
• Remote allocation adds an extra cost
17
REACTION
WORKSHOP
1,E+03
1,E+04
1,E+05
1,E+06
1,E+07
1,E+08
4096
1024
256
64
8
1
Timeconsumed(us)
Schedulable Segments (tasks) to deploy
Allocation costs depending on the kind
of schedulable
AsyncEvent
Handler
RtRO
Rthread
1,E+03
1,E+04
1,E+05
1,E+06
1,E+07
1,E+08
4096
1024
256
64
8
1
Timeconsumed(us)
Schedulable Segments (tasks) to deploy
Remote allocation of remote entities
AsyncEven
tHandler
RtRO
Rthread
796Mhz-100 Mbps
18. Total reconfiguration time
• Some interesting values
▫ A 1 seconds reconfiguration allows
reconfiguring 64 distributed tasks
• It depends on the number
▫ Of tasks
▫ Number of JVMs
• All schedulables are supposed to be event
handlers
▫ They bound the worst time of a thread and a
remote object
REACTION
WORKSHOP
18
1,E+04
1,E+05
1,E+06
1,E+07
1,E+08
4096
1024
256
64
8
1
Timeconsumed(us)
Schedulable Objects to allocate
Total {re}configuration time
32JVM
16JVM
4JVM
19. Conclusions
• Next generation Java-based distributed real-time
systems may benefit form having an embedded
infrastructure with reconfiguration facilities
▫ Similar to the one described in the article
• However, one may choose the reconfiguration policy to
suit specific application domains
19
REACTION
WORKSHOP
20. Ongoing work
• To address other reconfiguration policies
▫ Different scheduling algorithms
▫ Mode change protocols
As a way to introduce adaptation in DRTJava
Impact of multi-core policies
• Use of real-time OSGi platforms for reconfiguration
▫ Higher level adaptation closer to application
20
REACTION
WORKSHOP