Container orchestration in geo-distributed cloud computing platformsFogGuru MSCA Project
This document discusses research on improving the stability and resource management of geo-distributed Kubernetes federations.
The researchers found that Kubernetes Federation (KubeFed) implementations exhibited instability due to poor network conditions and static configuration parameters. They developed a proportional controller to dynamically adjust health check timeouts and improve stability.
They then addressed resource availability and network traffic awareness by developing mck8s, a platform that can deploy applications across clusters based on resource load and network traffic. Mck8s also allows bursting to public clouds when local clusters are overloaded and retracting those resources when possible. This improves resource management for geo-distributed multi-cluster Kubernetes environments.
From Cloud to Fog: the Tao of IT Infrastructure DecentralizationFogGuru MSCA Project
The document discusses the transition from cloud computing to fog computing. Fog computing decentralizes computing to the edge of networks in order to address issues like high latency and huge data volumes from IoT devices. It describes how Kubernetes works well for containers but is not optimized for proximity-aware routing needed in fog. Algorithms are presented for optimizing pod placement in fog to minimize latency. Models for optimizing stream processing across fog nodes based on computational complexity and network latency are also introduced. The talk concludes by discussing an ongoing research project called FogGuru that is conducting real-world fog computing experiments.
An Experiment-Driven Performance Model of Stream Processing Operators in Fog ...FogGuru MSCA Project
The document presents a performance model for stream processing operators in fog computing environments. The model predicts an operator's processing time based on factors like replication level, network delays, and parallelization efficiency. Experimental tests on Apache Flink show the model predicts processing time with 98% accuracy. The model can also be applied to entire workflows by considering the slowest operator. Parameters like parallelization efficiency may be reused for different operators to reduce calibration tests needed.
The document discusses the FogGuru project, which aims to train the next generation of European experts in fog computing. The 3-year project will provide in-depth technical training to 8 Early Stage Researchers across universities, companies and a living lab testbed in Valencia, Spain. The project addresses key challenges in fog computing like resource management, stream processing and application development through its research objectives and the ESR projects. The training combines technical skills, innovation mindsets, networking opportunities and soft skills to develop recognized specialists and entrepreneurs in fog computing.
Presentation by Steffen Zeuch, Researcher at German Research Center for Artificial Intelligence (DFKI) and Post-Doc at TU Berlin (Germany), at the FogGuru Boot Camp training in September 2018.
Low Energy Task Scheduling based on Work StealingLEGATO project
Abstract: Optimizing energy efficiency of parallel execution on computing systems, ranging from server farms, mobile devices to embedded systems, becomes increasingly one of the first-order concerns. A common way to express a parallel application is as a directed acyclic graph (DAG) in which each node represents a task. The problem of such task scheduling on multiprocessor systems is to find the proper execution processors. Especially nowadays asymmetric multiprocessor systems feature different type of cores with different performance and power consumption, e.g. Arm big.LITTLE and Intel Lakefield. However, naive task assignment without considering core types and task features could result in inefficient resources utilization and detrimentally impacts the overall energy consumption. Dynamic task scheduling is a widely used scheduling strategy, which does not require prior knowledge, e.g. architecture heterogeneity, task DAG structure, before execution but makes the decisions during runtime. Work stealing has been proven to be an effective method among dynamic task scheduling with better scalability in larger systems. DVFS is a common technique to achieve better energy efficiency, however, exploiting it costs reconfiguration overhead ranging from tens of microseconds to one millisecond. With fine-grained tasks as small as milliseconds, as required to expose large parallelism, it is not realistic to use DVFS on a per-task level. Also, it shows that the energy consumed in cores’ under-utilized period is significant.
Based on these problem statements, we come up with a low energy task scheduling work stealing runtime based on XiTAO where the system environment configurations are either fixed or managed by the O/S power governors or system administrators. The runtime contains dynamic performance tracing module, idleness tracing module, power profiling module and a task mapping algorithm. The dynamic performance model is able to give the accurate predictions for future tasks given a set of resources. It is independent of platforms and frequencies and achieves scalability and portability. Power profiling helps runtime systems to understand CPU power consumption trends with respect to number/type of cores and frequencies. Idleness tracing presents the real-time status of cores and contributes to the energy conservation of under-utilized period. It also provides the real-time parallel slackness of active cores, which allows the task mapping algorithm to attribute corresponding power consumption on each concurrent running task. The task mapping algorithm integrates the information from above three modules and outputs the predicted best resources placements for ready tasks.
Poster presented by jing Chen at the LEGaTO Final Event: 'Low-Energy Heterogeneous Computing Workshop'
Architecture and Performance of Runtime Environments for Data Intensive Scala...jaliyae
This document summarizes a student's doctoral research on runtime environments for data-intensive scalable computing. The key points are:
1) The student is investigating cloud runtimes like MapReduce, DryadLINQ, and i-MapReduce for data and compute-intensive applications represented as filter pipelines.
2) The student has applied these runtimes to applications in domains like genomics, phylogenetics, and high energy physics, demonstrating their ability to parallelize tasks.
3) The student has developed i-MapReduce to support iterative MapReduce computations more efficiently than traditional MapReduce systems by caching static data in memory between iterations.
4) Current research directions include evaluating the
Container orchestration in geo-distributed cloud computing platformsFogGuru MSCA Project
This document discusses research on improving the stability and resource management of geo-distributed Kubernetes federations.
The researchers found that Kubernetes Federation (KubeFed) implementations exhibited instability due to poor network conditions and static configuration parameters. They developed a proportional controller to dynamically adjust health check timeouts and improve stability.
They then addressed resource availability and network traffic awareness by developing mck8s, a platform that can deploy applications across clusters based on resource load and network traffic. Mck8s also allows bursting to public clouds when local clusters are overloaded and retracting those resources when possible. This improves resource management for geo-distributed multi-cluster Kubernetes environments.
From Cloud to Fog: the Tao of IT Infrastructure DecentralizationFogGuru MSCA Project
The document discusses the transition from cloud computing to fog computing. Fog computing decentralizes computing to the edge of networks in order to address issues like high latency and huge data volumes from IoT devices. It describes how Kubernetes works well for containers but is not optimized for proximity-aware routing needed in fog. Algorithms are presented for optimizing pod placement in fog to minimize latency. Models for optimizing stream processing across fog nodes based on computational complexity and network latency are also introduced. The talk concludes by discussing an ongoing research project called FogGuru that is conducting real-world fog computing experiments.
An Experiment-Driven Performance Model of Stream Processing Operators in Fog ...FogGuru MSCA Project
The document presents a performance model for stream processing operators in fog computing environments. The model predicts an operator's processing time based on factors like replication level, network delays, and parallelization efficiency. Experimental tests on Apache Flink show the model predicts processing time with 98% accuracy. The model can also be applied to entire workflows by considering the slowest operator. Parameters like parallelization efficiency may be reused for different operators to reduce calibration tests needed.
The document discusses the FogGuru project, which aims to train the next generation of European experts in fog computing. The 3-year project will provide in-depth technical training to 8 Early Stage Researchers across universities, companies and a living lab testbed in Valencia, Spain. The project addresses key challenges in fog computing like resource management, stream processing and application development through its research objectives and the ESR projects. The training combines technical skills, innovation mindsets, networking opportunities and soft skills to develop recognized specialists and entrepreneurs in fog computing.
Presentation by Steffen Zeuch, Researcher at German Research Center for Artificial Intelligence (DFKI) and Post-Doc at TU Berlin (Germany), at the FogGuru Boot Camp training in September 2018.
Low Energy Task Scheduling based on Work StealingLEGATO project
Abstract: Optimizing energy efficiency of parallel execution on computing systems, ranging from server farms, mobile devices to embedded systems, becomes increasingly one of the first-order concerns. A common way to express a parallel application is as a directed acyclic graph (DAG) in which each node represents a task. The problem of such task scheduling on multiprocessor systems is to find the proper execution processors. Especially nowadays asymmetric multiprocessor systems feature different type of cores with different performance and power consumption, e.g. Arm big.LITTLE and Intel Lakefield. However, naive task assignment without considering core types and task features could result in inefficient resources utilization and detrimentally impacts the overall energy consumption. Dynamic task scheduling is a widely used scheduling strategy, which does not require prior knowledge, e.g. architecture heterogeneity, task DAG structure, before execution but makes the decisions during runtime. Work stealing has been proven to be an effective method among dynamic task scheduling with better scalability in larger systems. DVFS is a common technique to achieve better energy efficiency, however, exploiting it costs reconfiguration overhead ranging from tens of microseconds to one millisecond. With fine-grained tasks as small as milliseconds, as required to expose large parallelism, it is not realistic to use DVFS on a per-task level. Also, it shows that the energy consumed in cores’ under-utilized period is significant.
Based on these problem statements, we come up with a low energy task scheduling work stealing runtime based on XiTAO where the system environment configurations are either fixed or managed by the O/S power governors or system administrators. The runtime contains dynamic performance tracing module, idleness tracing module, power profiling module and a task mapping algorithm. The dynamic performance model is able to give the accurate predictions for future tasks given a set of resources. It is independent of platforms and frequencies and achieves scalability and portability. Power profiling helps runtime systems to understand CPU power consumption trends with respect to number/type of cores and frequencies. Idleness tracing presents the real-time status of cores and contributes to the energy conservation of under-utilized period. It also provides the real-time parallel slackness of active cores, which allows the task mapping algorithm to attribute corresponding power consumption on each concurrent running task. The task mapping algorithm integrates the information from above three modules and outputs the predicted best resources placements for ready tasks.
Poster presented by jing Chen at the LEGaTO Final Event: 'Low-Energy Heterogeneous Computing Workshop'
Architecture and Performance of Runtime Environments for Data Intensive Scala...jaliyae
This document summarizes a student's doctoral research on runtime environments for data-intensive scalable computing. The key points are:
1) The student is investigating cloud runtimes like MapReduce, DryadLINQ, and i-MapReduce for data and compute-intensive applications represented as filter pipelines.
2) The student has applied these runtimes to applications in domains like genomics, phylogenetics, and high energy physics, demonstrating their ability to parallelize tasks.
3) The student has developed i-MapReduce to support iterative MapReduce computations more efficiently than traditional MapReduce systems by caching static data in memory between iterations.
4) Current research directions include evaluating the
High Performance & High Throughput Computing - EUDAT Summer School (Giuseppe ...EUDAT
Giuseppe will present the differences between high-performance and high-throughput applications. High-throughput computing (HTC) refers to computations where individual tasks do not need to interact while running. It differs from High-performance (HPC) where frequent and rapid exchanges of intermediate results is required to perform the computations. HPC codes are based on tightly coupled MPI, OpenMP, GPGPU, and hybrid programs and require low latency interconnected nodes. HTC makes use of unreliable components distributing the work out to every node and collecting results at the end of all parallel tasks.
Visit: https://www.eudat.eu/eudat-summer-school
Iterative computations are at the core of the vast majority of data-intensive scientific computations. Recent advancements in data intensive computational fields are fueling a dramatic growth in number as well as usage of such data intensive iterative computations. The utility computing model introduced by cloud computing combined with the rich set of cloud infrastructure services offers a very viable environment for the scientists to perform data intensive computations. However, clouds by nature offer unique reliability and sustained performance challenges to large scale distributed computations necessitating computation frameworks specifically tailored for cloud characteristics to harness the power of clouds easily and effectively. My research focuses on identifying and developing user-friendly distributed parallel computation frameworks to facilitate the optimized efficient execution of iterative as well as non-iterative data-intensive computations in cloud environments, alongside the evaluation of heterogeneous cloud resources offering GPGPU resources in addition to CPU resources, for data-intensive iterative computations.
High Performance Computing in the Cloud is viable in numerous use cases. Common to all successful use cases for cloud-based HPC is the ability embrace latency. Not surprisingly then, early successes were achieved with embarrassingly parallel HPC applications involving minimal amounts of data - in other words, there was little or no latency to be hidden. Over the fulness of time, however, the HPC-cloud community has become increasingly adept in its ability to ‘hide’ latency and, in the process, support increasingly more sophisticated HPC use cases in public and private clouds. Real-world use cases, deemed relevant to remote sensing, will illustrate aspects of these sophistications for hiding latency in accounting for large volumes of data, the need to pass messages between simultaneously executing components of distributed-memory parallel applications, as well as (processing) workflows/pipelines. Finally, the impact of containerizing HPC for the cloud will be considered through the relatively recent creation of the Cloud Native Computing Foundation.
Energy proportionality is the key in order to reduce the Total Cost of Ownership (TCO) of Warehouse Scale Computer (WSC) systems, yet is difficult to achieve in practice. Typical WSC hardware usually does not meet this principle. Furthermore, critical services (e.g. billing) require all servers to remain up regardless the current traffic intensity. These two issues make existing power management technique ineffective at reducing energy use in a WSC dimension. We present Hybrid Performance-aware Power-capping Orchestrator (HyPPO), a distributed Observe Decide Act (ODA) control loop for optimizing energy proportionality of a distribute containerized infrastructures. This first version of HyPPO uses Kubernetes resource metrics (e.g. milli-cpus consumption) in order to dynamically adjust node power consumption, while respecting the Service Level Agreement (SLA) agreement defined by the containerized application owners.
This document outlines a talk on high-performance computing trends. It discusses the need for HPC, parallel architectures and systems, high-throughput computing, distributed computing models like grids and clouds, and examples like the Top500 supercomputers and Amazon Web Services. It also previews a question and answer session at the end.
A Guide to Data Versioning with MapR SnapshotsIan Downard
Experimentation is fundamental to how software is developed for Machine Learning (ML). The procedures used for data preparation, algorithm development, and hyper-parameter tuning are very iterative and frequently depend on trial and error. In order to facilitate this kind of software development you have to track the code, configurations, and data used for ML experiments so you can always answer the question of how a model was trained. However, large training datasets often preclude traditional version control software from being used for this purpose. In these cases, MapR Snapshots provides a highly attractive solution for data versioning.
In this presentation you will learn how to version control data in files, tables, and/or streams with MapR Snapshots, and how to identify cases when MapR Snapshots provide significant advantages versus other data versioning techniques.
Distributed data-intensive systems are increasingly designed to be only eventually consistent. Persistent data is no longer processed with serialized and transactional access, exposing applications to a range of potential consistency and concurrency anomalies that need to be handled by the application itself. Controlling concurrent data access in monolithic systems is already challenging, but the problem is exacerbated in distributed systems. To make it worse, only little systematic engineering guidance is provided by the software architecture community regarding this issue.
Susanne shares her experiences from different case studies with industry clients, and novel design guidelines developed by using action research. You will learn settled and novel approaches to tackle consistency- and concurrency related design challenges.
Deadline Monotonic Scheduling to Reduce Overhead of Superframe in ISA100.11aOka Danil
In this paper, we develop new method to reduce communication overhead in ISA100.11a
Wireless Industrial Networks. Deadline monotonic scheduling scheme was proposed for
the analysis and simulation of the system. The result shows, our proposed approach
require less number of beacon to transmit data in network.
Superframe Scheduling with Beacon Enable Mode in Wireless Industrial NetworksOka Danil
In this paper, we develop a method to check the schedulability superframe with beacon-enabled mode in ISA100.11a Wireless Industrial Networks environment. Maximum length of time slot is added in the superframe to reduce the overhead, and deadline monotonic scheduling approach is proposed to analyze the system scenario. The simulation results indicate that our proposed method required less number of beacon for message scheduling. Therefore, more data can be sent in the network with our proposed method.
Full paper can be find in http://okadanil.com/publications
DSD-INT 2015 - From flood control to energy production - experiences as an in...Deltares
Hydrotec is an intermediary that provides services related to the Delft-FEWS forecasting system. They have 30 years of experience in hydrology and have implemented several Delft-FEWS systems for clients in Germany, Austria, and Turkey. As an intermediary, they provide system design, installation, modeling integration, programming, support and maintenance through a helpdesk system. Two examples of systems they implemented are for waterboards in North Rhine-Westphalia, Germany to process radar and forecast data for flood warnings, and for Verbund AG in Austria to forecast energy production from hydropower plants.
Abstract: In the last few years energy efficiency of large scale infrastructures gained a lot of attention, as power consumption became one of the most impacting factors of the operative costs of a data-center and of its Total Cost of Ownership (TCO). Power consumption can be observed at different layers of the data-center, from the overall power grid, moving to each rack and arriving to each machine and system. Given the rise of application containers both in the cloud computing and High Performance Computing (HPC) scenarios, it becomes more and more important to measure power consumption also at the application level, where power-aware schedulers and orchestrators can optimize the execution of the workloads not only from a performance perspective, but also considering performance/power trade-offs.
What we propose is DEEP-mon, a novel monitoring tool able to measure power consumption and attribute it for each thread and application container running in the system. Moreover, we show how the proposed approach has a negligible impact on the monitored system and on the running workloads, overcoming the limitations of the previous works in the field.
Streaming exa-scale data over 100Gbps networksbalmanme
This document discusses streaming exascale data over 100Gbps networks. It summarizes a demonstration at SC11 where climate simulation data was transferred from NERSC to ANL and ORNL at 83Gbps using a memory-mapped zero-copy network channel called MemzNet. The demonstration showed efficient transfer of large datasets containing many small files is possible over high-bandwidth networks through parallel streams, decoupling I/O and network operations, and dynamic data channel management. High-performance was achieved by keeping the data channel full through concurrent transfers and leveraging high-speed networking testbeds like ANI.
DSD-INT 2015 - The future of Delft-FEWS - Simone van Schijndel, DeltaresDeltares
The document discusses the future of Delft-FEWS, a software platform for hydrological forecasting and water management. It outlines several key points:
1. Delft-FEWS has been in use for over 20 years but the world and technologies are changing rapidly. The software and organization need to adapt.
2. A Community Strategy Board has been established to help guide the future of Delft-FEWS, including support/maintenance, funding, and development priorities.
3. Technical priorities include reviewing the architecture, handling big and open data, cloud computing, web services, and simplifying installation and use. User needs and new types of models/data will also be considered.
IoT Workload Distribution Impact Between Edge and Cloud Computing in a Smart ...Otávio Carvalho
This document summarizes research into distributing the workload of an IoT smart grid application between edge and cloud computing resources. The researchers implemented a three-layer architecture with sensors, edge nodes (Raspberry Pis), and cloud VMs. Their evaluation found that edge processing achieved higher throughput than cloud alone by reducing data sent to the cloud. Moving more workload to edge nodes and aggregating data at the edge also improved scalability. Future work could explore adaptive scheduling and evolving the architecture for general IoT applications.
Low Power High-Performance Computing on the BeagleBoard Platforma3labdsp
The ever increasing energy requirements of supercomputers and server farms is driving the scientific and industrial communities to take in deeper consideration the energy efficiency of computing equipments. This contribution addresses the issue proposing a cluster of ARM processors for high-performance computing. The cluster is composed of five BeagleBoard-xM, with one board managing the cluster, and the other boards executing the actual processing. The software platform is based on the Angstrom GNU/Linux distribution and is equipped with a distributed file system to ease sharing data and code among the nodes of the cluster, and with tools for managing tasks and monitoring the status of each node. The computational capabilities of the cluster have been assessed through High-Performance Linpack and a cluster-wide speaker diarization algorithm, while power consumption has been measured using a clamp meter. Experimental results obtained in the speaker diarization task showed that the energy efficiency of the BeagleBoard-xM cluster is comparable to the one of a laptop computer equipped with a Intel Core2 Duo T8300 running at 2.4 GHz. Furthermore, removing the bottleneck due to the Ethernet interface, the BeagleBoard-xM cluster is able to achieve a superior energy efficiency.
Detecting Lateral Movement with a Compute-Intense Graph KernelData Works MD
Cybersecurity Analytics on a D-Wave Quantum Computer
Effective cybersecurity analysis requires frequent exploration of graphs of many types and sizes, the computational cost of which can be overwhelming if not carefully chosen. After briefly introducing the D-Wave quantum computing system, we describe an analytic for finding “lateral movement” in an enterprise network, i.e., an intruder or insider threat hopping from system to system to gain access to more information. This analytic depends on maximum independent set, an NP-hard graph kernel whose computational cost grows exponentially with the size of the graph and so has not been widely used in cyber analysis. The growing strength of D-Wave’s quantum computers on such NP-hard problems will enable new analytics. We discuss practicalities of the current implementation and implications of this approach.
Steve Reinhardt has built hardware/software systems that deliver new levels of performance usable via conceptually simple interfaces, including Cray Research’s T3E distributed-memory systems, ISC’s Star-P parallel-MATLAB software, and YarcData/Cray’s Urika graph-analytic systems. He now leads D-Wave’s efforts working with customers to map early applications to D-Wave systems.
This document discusses streaming exascale data over 100Gbps networks for climate science applications. It summarizes that:
1) Data volume is increasing exponentially for climate applications, posing challenges for data management.
2) Streaming climate simulation data, which consists of small and large irregularly sized files, efficiently over high-bandwidth networks could benefit climate science.
3) A framework called MemzNet was developed to efficiently move climate files over 100Gbps networks by decoupling I/O and networking operations and dynamically managing data transfer. MemzNet was able to saturate a 100Gbps testbed network.
High performance computing - building blocks, production & perspectiveJason Shih
This document provides an overview of high performance computing (HPC). It defines HPC as using supercomputers and computer clusters to solve advanced computation problems quickly and efficiently through parallel processing. The document discusses the building blocks of HPC systems including CPUs, memory, power consumption, and number of cores. It also outlines some common applications of HPC in fields like physics, engineering, and life sciences. Finally, it traces the evolution of HPC technologies over decades from early mainframes and supercomputers to today's clusters and parallel systems.
CloudOpen 2013: Developing cloud infrastructure: from scratch: the tale of an...Andrey Korolyov
The document summarizes the development of cloud infrastructure from scratch by an ISP. Key points include:
- Ceph was chosen for storage due to its feature set, but initial versions were unstable and had I/O freezes.
- Early versions of libvirt, KVM, and monitoring tools like collectd were used due to their rich feature sets, but had bugs and leaks that caused instability.
- An orchestrator was launched to manage VMs, but performance tuning like NUMA and various features proved unnecessary or unstable for production use.
- Underlying components like XFS and Ceph remained unstable under load. Ongoing work was needed to improve stability and performance.
The document discusses green cloud computing and describes a technical seminar presented by S.Sai Madhuri. It defines cloud computing and discusses types including SaaS, PaaS, and IaaS. It then explains green computing and green cloud computing, describing the core components and architecture of data centers. The document outlines the objective of calculating energy consumption using a green cloud simulator in VMWare Player to analyze existing systems and develop more efficient solutions.
High Performance & High Throughput Computing - EUDAT Summer School (Giuseppe ...EUDAT
Giuseppe will present the differences between high-performance and high-throughput applications. High-throughput computing (HTC) refers to computations where individual tasks do not need to interact while running. It differs from High-performance (HPC) where frequent and rapid exchanges of intermediate results is required to perform the computations. HPC codes are based on tightly coupled MPI, OpenMP, GPGPU, and hybrid programs and require low latency interconnected nodes. HTC makes use of unreliable components distributing the work out to every node and collecting results at the end of all parallel tasks.
Visit: https://www.eudat.eu/eudat-summer-school
Iterative computations are at the core of the vast majority of data-intensive scientific computations. Recent advancements in data intensive computational fields are fueling a dramatic growth in number as well as usage of such data intensive iterative computations. The utility computing model introduced by cloud computing combined with the rich set of cloud infrastructure services offers a very viable environment for the scientists to perform data intensive computations. However, clouds by nature offer unique reliability and sustained performance challenges to large scale distributed computations necessitating computation frameworks specifically tailored for cloud characteristics to harness the power of clouds easily and effectively. My research focuses on identifying and developing user-friendly distributed parallel computation frameworks to facilitate the optimized efficient execution of iterative as well as non-iterative data-intensive computations in cloud environments, alongside the evaluation of heterogeneous cloud resources offering GPGPU resources in addition to CPU resources, for data-intensive iterative computations.
High Performance Computing in the Cloud is viable in numerous use cases. Common to all successful use cases for cloud-based HPC is the ability embrace latency. Not surprisingly then, early successes were achieved with embarrassingly parallel HPC applications involving minimal amounts of data - in other words, there was little or no latency to be hidden. Over the fulness of time, however, the HPC-cloud community has become increasingly adept in its ability to ‘hide’ latency and, in the process, support increasingly more sophisticated HPC use cases in public and private clouds. Real-world use cases, deemed relevant to remote sensing, will illustrate aspects of these sophistications for hiding latency in accounting for large volumes of data, the need to pass messages between simultaneously executing components of distributed-memory parallel applications, as well as (processing) workflows/pipelines. Finally, the impact of containerizing HPC for the cloud will be considered through the relatively recent creation of the Cloud Native Computing Foundation.
Energy proportionality is the key in order to reduce the Total Cost of Ownership (TCO) of Warehouse Scale Computer (WSC) systems, yet is difficult to achieve in practice. Typical WSC hardware usually does not meet this principle. Furthermore, critical services (e.g. billing) require all servers to remain up regardless the current traffic intensity. These two issues make existing power management technique ineffective at reducing energy use in a WSC dimension. We present Hybrid Performance-aware Power-capping Orchestrator (HyPPO), a distributed Observe Decide Act (ODA) control loop for optimizing energy proportionality of a distribute containerized infrastructures. This first version of HyPPO uses Kubernetes resource metrics (e.g. milli-cpus consumption) in order to dynamically adjust node power consumption, while respecting the Service Level Agreement (SLA) agreement defined by the containerized application owners.
This document outlines a talk on high-performance computing trends. It discusses the need for HPC, parallel architectures and systems, high-throughput computing, distributed computing models like grids and clouds, and examples like the Top500 supercomputers and Amazon Web Services. It also previews a question and answer session at the end.
A Guide to Data Versioning with MapR SnapshotsIan Downard
Experimentation is fundamental to how software is developed for Machine Learning (ML). The procedures used for data preparation, algorithm development, and hyper-parameter tuning are very iterative and frequently depend on trial and error. In order to facilitate this kind of software development you have to track the code, configurations, and data used for ML experiments so you can always answer the question of how a model was trained. However, large training datasets often preclude traditional version control software from being used for this purpose. In these cases, MapR Snapshots provides a highly attractive solution for data versioning.
In this presentation you will learn how to version control data in files, tables, and/or streams with MapR Snapshots, and how to identify cases when MapR Snapshots provide significant advantages versus other data versioning techniques.
Distributed data-intensive systems are increasingly designed to be only eventually consistent. Persistent data is no longer processed with serialized and transactional access, exposing applications to a range of potential consistency and concurrency anomalies that need to be handled by the application itself. Controlling concurrent data access in monolithic systems is already challenging, but the problem is exacerbated in distributed systems. To make it worse, only little systematic engineering guidance is provided by the software architecture community regarding this issue.
Susanne shares her experiences from different case studies with industry clients, and novel design guidelines developed by using action research. You will learn settled and novel approaches to tackle consistency- and concurrency related design challenges.
Deadline Monotonic Scheduling to Reduce Overhead of Superframe in ISA100.11aOka Danil
In this paper, we develop new method to reduce communication overhead in ISA100.11a
Wireless Industrial Networks. Deadline monotonic scheduling scheme was proposed for
the analysis and simulation of the system. The result shows, our proposed approach
require less number of beacon to transmit data in network.
Superframe Scheduling with Beacon Enable Mode in Wireless Industrial NetworksOka Danil
In this paper, we develop a method to check the schedulability superframe with beacon-enabled mode in ISA100.11a Wireless Industrial Networks environment. Maximum length of time slot is added in the superframe to reduce the overhead, and deadline monotonic scheduling approach is proposed to analyze the system scenario. The simulation results indicate that our proposed method required less number of beacon for message scheduling. Therefore, more data can be sent in the network with our proposed method.
Full paper can be find in http://okadanil.com/publications
DSD-INT 2015 - From flood control to energy production - experiences as an in...Deltares
Hydrotec is an intermediary that provides services related to the Delft-FEWS forecasting system. They have 30 years of experience in hydrology and have implemented several Delft-FEWS systems for clients in Germany, Austria, and Turkey. As an intermediary, they provide system design, installation, modeling integration, programming, support and maintenance through a helpdesk system. Two examples of systems they implemented are for waterboards in North Rhine-Westphalia, Germany to process radar and forecast data for flood warnings, and for Verbund AG in Austria to forecast energy production from hydropower plants.
Abstract: In the last few years energy efficiency of large scale infrastructures gained a lot of attention, as power consumption became one of the most impacting factors of the operative costs of a data-center and of its Total Cost of Ownership (TCO). Power consumption can be observed at different layers of the data-center, from the overall power grid, moving to each rack and arriving to each machine and system. Given the rise of application containers both in the cloud computing and High Performance Computing (HPC) scenarios, it becomes more and more important to measure power consumption also at the application level, where power-aware schedulers and orchestrators can optimize the execution of the workloads not only from a performance perspective, but also considering performance/power trade-offs.
What we propose is DEEP-mon, a novel monitoring tool able to measure power consumption and attribute it for each thread and application container running in the system. Moreover, we show how the proposed approach has a negligible impact on the monitored system and on the running workloads, overcoming the limitations of the previous works in the field.
Streaming exa-scale data over 100Gbps networksbalmanme
This document discusses streaming exascale data over 100Gbps networks. It summarizes a demonstration at SC11 where climate simulation data was transferred from NERSC to ANL and ORNL at 83Gbps using a memory-mapped zero-copy network channel called MemzNet. The demonstration showed efficient transfer of large datasets containing many small files is possible over high-bandwidth networks through parallel streams, decoupling I/O and network operations, and dynamic data channel management. High-performance was achieved by keeping the data channel full through concurrent transfers and leveraging high-speed networking testbeds like ANI.
DSD-INT 2015 - The future of Delft-FEWS - Simone van Schijndel, DeltaresDeltares
The document discusses the future of Delft-FEWS, a software platform for hydrological forecasting and water management. It outlines several key points:
1. Delft-FEWS has been in use for over 20 years but the world and technologies are changing rapidly. The software and organization need to adapt.
2. A Community Strategy Board has been established to help guide the future of Delft-FEWS, including support/maintenance, funding, and development priorities.
3. Technical priorities include reviewing the architecture, handling big and open data, cloud computing, web services, and simplifying installation and use. User needs and new types of models/data will also be considered.
IoT Workload Distribution Impact Between Edge and Cloud Computing in a Smart ...Otávio Carvalho
This document summarizes research into distributing the workload of an IoT smart grid application between edge and cloud computing resources. The researchers implemented a three-layer architecture with sensors, edge nodes (Raspberry Pis), and cloud VMs. Their evaluation found that edge processing achieved higher throughput than cloud alone by reducing data sent to the cloud. Moving more workload to edge nodes and aggregating data at the edge also improved scalability. Future work could explore adaptive scheduling and evolving the architecture for general IoT applications.
Low Power High-Performance Computing on the BeagleBoard Platforma3labdsp
The ever increasing energy requirements of supercomputers and server farms is driving the scientific and industrial communities to take in deeper consideration the energy efficiency of computing equipments. This contribution addresses the issue proposing a cluster of ARM processors for high-performance computing. The cluster is composed of five BeagleBoard-xM, with one board managing the cluster, and the other boards executing the actual processing. The software platform is based on the Angstrom GNU/Linux distribution and is equipped with a distributed file system to ease sharing data and code among the nodes of the cluster, and with tools for managing tasks and monitoring the status of each node. The computational capabilities of the cluster have been assessed through High-Performance Linpack and a cluster-wide speaker diarization algorithm, while power consumption has been measured using a clamp meter. Experimental results obtained in the speaker diarization task showed that the energy efficiency of the BeagleBoard-xM cluster is comparable to the one of a laptop computer equipped with a Intel Core2 Duo T8300 running at 2.4 GHz. Furthermore, removing the bottleneck due to the Ethernet interface, the BeagleBoard-xM cluster is able to achieve a superior energy efficiency.
Detecting Lateral Movement with a Compute-Intense Graph KernelData Works MD
Cybersecurity Analytics on a D-Wave Quantum Computer
Effective cybersecurity analysis requires frequent exploration of graphs of many types and sizes, the computational cost of which can be overwhelming if not carefully chosen. After briefly introducing the D-Wave quantum computing system, we describe an analytic for finding “lateral movement” in an enterprise network, i.e., an intruder or insider threat hopping from system to system to gain access to more information. This analytic depends on maximum independent set, an NP-hard graph kernel whose computational cost grows exponentially with the size of the graph and so has not been widely used in cyber analysis. The growing strength of D-Wave’s quantum computers on such NP-hard problems will enable new analytics. We discuss practicalities of the current implementation and implications of this approach.
Steve Reinhardt has built hardware/software systems that deliver new levels of performance usable via conceptually simple interfaces, including Cray Research’s T3E distributed-memory systems, ISC’s Star-P parallel-MATLAB software, and YarcData/Cray’s Urika graph-analytic systems. He now leads D-Wave’s efforts working with customers to map early applications to D-Wave systems.
This document discusses streaming exascale data over 100Gbps networks for climate science applications. It summarizes that:
1) Data volume is increasing exponentially for climate applications, posing challenges for data management.
2) Streaming climate simulation data, which consists of small and large irregularly sized files, efficiently over high-bandwidth networks could benefit climate science.
3) A framework called MemzNet was developed to efficiently move climate files over 100Gbps networks by decoupling I/O and networking operations and dynamically managing data transfer. MemzNet was able to saturate a 100Gbps testbed network.
High performance computing - building blocks, production & perspectiveJason Shih
This document provides an overview of high performance computing (HPC). It defines HPC as using supercomputers and computer clusters to solve advanced computation problems quickly and efficiently through parallel processing. The document discusses the building blocks of HPC systems including CPUs, memory, power consumption, and number of cores. It also outlines some common applications of HPC in fields like physics, engineering, and life sciences. Finally, it traces the evolution of HPC technologies over decades from early mainframes and supercomputers to today's clusters and parallel systems.
CloudOpen 2013: Developing cloud infrastructure: from scratch: the tale of an...Andrey Korolyov
The document summarizes the development of cloud infrastructure from scratch by an ISP. Key points include:
- Ceph was chosen for storage due to its feature set, but initial versions were unstable and had I/O freezes.
- Early versions of libvirt, KVM, and monitoring tools like collectd were used due to their rich feature sets, but had bugs and leaks that caused instability.
- An orchestrator was launched to manage VMs, but performance tuning like NUMA and various features proved unnecessary or unstable for production use.
- Underlying components like XFS and Ceph remained unstable under load. Ongoing work was needed to improve stability and performance.
The document discusses green cloud computing and describes a technical seminar presented by S.Sai Madhuri. It defines cloud computing and discusses types including SaaS, PaaS, and IaaS. It then explains green computing and green cloud computing, describing the core components and architecture of data centers. The document outlines the objective of calculating energy consumption using a green cloud simulator in VMWare Player to analyze existing systems and develop more efficient solutions.
This is the presentation on clusters computing which includes information from other sources too including my own research and edition. I hope this will help everyone who required to know on this topic.
XPDS13: Enabling Fast, Dynamic Network Processing with ClickOS - Joao Martins...The Linux Foundation
While virtualization technologies like Xen have been around for a long time, it is only in recent years that they have started to be targeted as viable systems for implementing middlebox processing (e.g., firewalls, NATs). But can they provide this functionality while yielding the high performance expected from hardware-based middlebox offerings? In this talk Joao Martins will introduce ClickOS, a tiny, MiniOS-based virtual machine tailored for network processing. In addition to the vm itself, Joao Martins will describe performance improvements done to the entire Xen I/O pipe. Finally, Joao Martins will discuss an evaluation showing that ClickOS can be instantiated in 30 msecs, can process traffic at 10Gb/s for almost all packet sizes, introduces delay of 40 microseconds and can run middleboxes at rates of 5 Mp/s.
The World of Internet
History of cloud computing
What is Cloud Computing?
Types of Cloud Computing
i. Software as a Service(SaaS)
ii. Platform as aService(PaaS)
iii. Infrastructure as a Service(IaaS)
Characteristics of Cloud Computing
Deployment model of Cloud Computing
In today’s world the growing demand for knowledge has made cloud computing a center of attraction. Cloud computing is providing utility based services to all the users worldwide. It enables presentation of applications from consumers, scientific and business domains. However, data centers created for cloud computing applications consume huge amounts of energy, contributing to high operational costs and a large amount of carbon dioxide emission to the environment. With enhancement of data center, the power consumption is increasing at such a rate that it has become a key concern these days because it is ultimately leading to energy shortcomings and global climatic change. Therefore, we need green cloud computing solutions that can not only save energy, but also reduce operational costs.
Dependable Storage and Computing using Multiple Cloud ProvidersAlysson Bessani
Presentation at the International Industry-Academia Workshop on Cloud Reliability and Resilience. 7-8 November 2016, Berlin, Germany.
Organized by EIT Digital and Huawei GRC, Germany.
Twitter: @CloudRR2016
We demonstrated how at Criteo we have introduced on our Mesos clusters:
* network isolation between our containers
* a network bandwidth custom resource patching all our frameworks (marathon and aurora).
This talk has been presented at MesosCon18 in SF.
This document discusses the benefits of 10 Gigabit Ethernet (10GE) for reducing latency. It states that 10GE allows for lower CPU utilization and reduced latency within servers compared to Gigabit Ethernet. It also discusses that while Infiniband promised low latency, it required application rewrites and added complexity due to needing translation to Ethernet outside local networks. The document explores 10GE cabling options like SFP+ which provide the lowest latency, and network interface cards that support technologies like RDMA for further reducing latency within servers. With the right hardware and software, organizations can see over an 80% reduction in overall end-to-end latency by moving from Gigabit Ethernet to 10GE.
Cloud computing allows users to access shared computing resources over the Internet. It provides advantages like reduced costs, increased mobility and scalability. There are three main service models - Infrastructure as a Service (IaaS), Platform as a Service (PaaS) and Software as a Service (SaaS). Cloud environments can be private, public or hybrid. While cloud computing provides benefits, it also has disadvantages relating to security, vendor dependence and internet connectivity requirements.
The document discusses performance characterization across a computing continuum from the edge to the cloud. It evaluates the performance of video encoding and machine learning tasks on different devices. For video encoding, older single-board computers had significantly higher encoding times than other resources but provided lower data transfer times. For machine learning, training a convolutional neural network took much longer than a simpler model. Cloud and fog resources generally outperformed edge devices for more complex tasks. The document recommends offloading large or complex tasks to more powerful resources when possible.
1. Building exascale computers requires moving to sub-nanometer scales and steering individual electrons to solve problems more efficiently.
2. Moving data is a major challenge, as moving data off-chip uses 200x more energy than computing with it on-chip.
3. Future computers should optimize for data movement at all levels, from system design to microarchitecture, to minimize energy usage.
Webinar: OpenEBS - Still Free and now FASTEST Kubernetes storageMayaData Inc
Webinar Session - https://youtu.be/_5MfGMf8PG4
In this webinar, we share how the Container Attached Storage pattern makes performance tuning more tractable, by giving each workload its own storage system, thereby decreasing the variables needed to understand and tune performance.
We then introduce MayaStor, a breakthrough in the use of containers and Kubernetes as a data plane. MayaStor is the first containerized data engine available that delivers near the theoretical maximum performance of underlying systems. MayaStor performance scales with the underlying hardware and has been shown, for example, to deliver in excess of 10 million IOPS in a particular environment.
Here is a draft proposal for migrating the Windows XP machines in the new LSDG research group to Linux:
Proposal to Migrate LSDG Desktops from Windows XP to Linux
Introduction
The new LSDG research group at Linx LLC will be using desktop operating systems. Currently, some machines in the larger Linx LLC organization run Windows XP and Windows 7. As LSDG will be a separate research group, we need to consider the best desktop OS choice for their needs and the longevity of the machines.
Analysis
Windows XP is no longer supported by Microsoft, so continuing to use it poses major security risks. Without updates and patches, XP machines are vulnerable to exploits. Support for Windows 7 will also end
automation in it's next level,applications of fog computing,need of fog computing,fog vs cloud, Internet of things,fog vs cloud vs IOT ,existing cloud system, proposed system presentation conclusion
A computer cluster is a group of tightly coupled computers that work together as a single computer. Clusters provide increased processing power at lower costs compared to single computers. They improve availability by eliminating single points of failure. Additional nodes can be added to a cluster to increase its overall capacity as processing demands grow. Key components of clusters include processors, memory, fast networking components, and specialized cluster software.
High Performance Communication for Oracle using InfiniBandwebhostingguy
The document discusses how InfiniBand provides benefits for Oracle databases by enabling higher performance communication within Oracle Real Application Clusters (RAC). InfiniBand allows for faster block transfers, lower CPU utilization, and higher throughput compared to Gigabit Ethernet. It also supports features like remote direct memory access that improve performance of Oracle RAC operations like locking and parallel queries.
The Impact of Software-based Virtual Network in the Public CloudChunghan Lee
Today’s cloud network consists of sophisticated virtual
networks, and a virtual switch is a key element of these
networks. Although there is tremendous interest in measuring
cloud network performance, little is known about the impact
of software-based virtual network on latency. In this paper, we
conduct the impact of virtual network on latency in the public
cloud based on OpenStack. We measured the throughput of VMs
and simultaneously captured their packets on hosts. We analyzed
the traces by using well-known metrics, such as throughput and
RTT, and investigated the abrupt fluctuation of latency called as
‘the burstiness of latency’. We quantitatively clarify the impact of
software-based virtual network on latency. In our public cloud,
the latency is approximately 35.2% of RTT and 10% of burstiness
mainly contributes to the increased RTT. The total latency was
increased by the receiving side regardless of data and ACK
paths. Our analysis results, discussions, and implications can
not only help cloud researchers and developers design the next
generation of software-based virtual network but can also help
cloud operators improve the performance of virtual network.
We leave in the era where the atomic building elements of silicon computers, e.g., transistors and wires, are no longer visible using traditional optical microscopes and their sizes are measured in just tens of Angstroms. In addition, power dissipation per unit volume is bounded by the laws of Physics that all resulted among others in stagnating processor clock frequencies. Adding more and more processor cores that perform simpler and simpler tasks in an attempt to efficiently fill the available on-chip area seems to be the current trend taken by the Industry.
Similar to From data centers to fog computing: the evaporating cloud (20)
This document provides tips and guidelines for effective public speaking from Véronique Trüb. It outlines 12 key points for crafting an engaging speech, including making an eye-catching introduction and conclusion, using examples and stories, incorporating data and analogies, and engaging multiple senses. It also lists the fundamental rules for optimizing speaking skills, such as thorough preparation, practice, breathing techniques, simplifying content, maintaining energy and enthusiasm, and using body language like smiling and gestures. The overall message is that following Trüb's guidelines can help avoid boring speeches and improve one's public speaking abilities.
Presentation by Diego Useche, Associated Professor at the University of Rennes 1 (France), at the FogGuru training Business Modeling and Development in November 2019.
Presentation by Pooya Hedaytinia, PhD at the University of Rennes 1 (France), at the FogGuru training Business Modeling and Development in November 2019.
Presentation by Diego Useche, Associated Professor at the University of Rennes 1 (France), at the FogGuru training Business Modeling and Development in November 2019.
The document discusses resources and capabilities as internal abilities that allow firms to succeed. It defines resources as tangible and intangible assets that firms can use to implement strategies, and capabilities as operating routines and dynamic processes for adapting resources. Examples include McDonald's efficient operating capabilities and Toyota's continuous improvement processes. Priorities are guided by values and influence resource allocation. The VRIO model assesses if resources provide competitive advantage by being valuable, rare, inimitable, and exploitable. Firms must analyze their resources and capabilities using multiple data sources to understand strengths and competitive positions.
The document discusses the bankruptcy filing of Seven Dreamers Laboratories Inc., a Tokyo-based startup that had developed a prototype laundry-folding robot called Laundroid. The startup had received over 10 billion yen in funding but struggled to commercialize the robot, missing two sales targets. It accumulated 2.2 billion yen in debt from heavy R&D investments. Unable to secure more funds for further development to improve the robot, the startup filed for bankruptcy, ending development of the Laundroid robot for now. The concept of a laundry-folding robot had generated interest, but technical and financial challenges prevented Seven Dreamers from successfully bringing their prototype to market.
Presentation by Tommy Lofstedt, Associated Professor at Umeå University (Sweden), at the FogGuru Workshop on linking with other disciplines in October 2019.
Presentation by Tommy Lofstedt, Associated Professor at Umeå University (Sweden), at the FogGuru Workshop on linking with other disciplines in October 2019.
Presentation by Cedric Tedeshi, Associate Professor of Computer Science at the University of Rennes 1 (France), at the FogGuru Workshop on research methods in January 2019.
Presentation by Guillaume Pierre, Professor of Computer Science at the University of Rennes 1 (France), at the FogGuru Workshop on research methods in January 2019.
Rennes Métropole is working to promote soft mobility solutions to reduce reliance on personal vehicles. Key issues include reducing traffic congestion caused by single-occupancy vehicles, improving communication about existing transportation options, and increasing accessibility for all users. New mobility options being explored include electric scooters, electric bikes, and improved integration between transportation modes like public transit, carpooling services, and micro-mobility services. Data sharing and user experience are also important considerations for developing an effective and equitable soft mobility network.
For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2024/06/building-and-scaling-ai-applications-with-the-nx-ai-manager-a-presentation-from-network-optix/
Robin van Emden, Senior Director of Data Science at Network Optix, presents the “Building and Scaling AI Applications with the Nx AI Manager,” tutorial at the May 2024 Embedded Vision Summit.
In this presentation, van Emden covers the basics of scaling edge AI solutions using the Nx tool kit. He emphasizes the process of developing AI models and deploying them globally. He also showcases the conversion of AI models and the creation of effective edge AI pipelines, with a focus on pre-processing, model conversion, selecting the appropriate inference engine for the target hardware and post-processing.
van Emden shows how Nx can simplify the developer’s life and facilitate a rapid transition from concept to production-ready applications.He provides valuable insights into developing scalable and efficient edge AI solutions, with a strong focus on practical implementation.
Generating privacy-protected synthetic data using Secludy and MilvusZilliz
During this demo, the founders of Secludy will demonstrate how their system utilizes Milvus to store and manipulate embeddings for generating privacy-protected synthetic data. Their approach not only maintains the confidentiality of the original data but also enhances the utility and scalability of LLMs under privacy constraints. Attendees, including machine learning engineers, data scientists, and data managers, will witness first-hand how Secludy's integration with Milvus empowers organizations to harness the power of LLMs securely and efficiently.
Fueling AI with Great Data with Airbyte WebinarZilliz
This talk will focus on how to collect data from a variety of sources, leveraging this data for RAG and other GenAI use cases, and finally charting your course to productionalization.
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?Speck&Tech
ABSTRACT: A prima vista, un mattoncino Lego e la backdoor XZ potrebbero avere in comune il fatto di essere entrambi blocchi di costruzione, o dipendenze di progetti creativi e software. La realtà è che un mattoncino Lego e il caso della backdoor XZ hanno molto di più di tutto ciò in comune.
Partecipate alla presentazione per immergervi in una storia di interoperabilità, standard e formati aperti, per poi discutere del ruolo importante che i contributori hanno in una comunità open source sostenibile.
BIO: Sostenitrice del software libero e dei formati standard e aperti. È stata un membro attivo dei progetti Fedora e openSUSE e ha co-fondato l'Associazione LibreItalia dove è stata coinvolta in diversi eventi, migrazioni e formazione relativi a LibreOffice. In precedenza ha lavorato a migrazioni e corsi di formazione su LibreOffice per diverse amministrazioni pubbliche e privati. Da gennaio 2020 lavora in SUSE come Software Release Engineer per Uyuni e SUSE Manager e quando non segue la sua passione per i computer e per Geeko coltiva la sua curiosità per l'astronomia (da cui deriva il suo nickname deneb_alpha).
Introduction of Cybersecurity with OSS at Code Europe 2024Hiroshi SHIBATA
I develop the Ruby programming language, RubyGems, and Bundler, which are package managers for Ruby. Today, I will introduce how to enhance the security of your application using open-source software (OSS) examples from Ruby and RubyGems.
The first topic is CVE (Common Vulnerabilities and Exposures). I have published CVEs many times. But what exactly is a CVE? I'll provide a basic understanding of CVEs and explain how to detect and handle vulnerabilities in OSS.
Next, let's discuss package managers. Package managers play a critical role in the OSS ecosystem. I'll explain how to manage library dependencies in your application.
I'll share insights into how the Ruby and RubyGems core team works to keep our ecosystem safe. By the end of this talk, you'll have a better understanding of how to safeguard your code.
5th LF Energy Power Grid Model Meet-up SlidesDanBrown980551
5th Power Grid Model Meet-up
It is with great pleasure that we extend to you an invitation to the 5th Power Grid Model Meet-up, scheduled for 6th June 2024. This event will adopt a hybrid format, allowing participants to join us either through an online Mircosoft Teams session or in person at TU/e located at Den Dolech 2, Eindhoven, Netherlands. The meet-up will be hosted by Eindhoven University of Technology (TU/e), a research university specializing in engineering science & technology.
Power Grid Model
The global energy transition is placing new and unprecedented demands on Distribution System Operators (DSOs). Alongside upgrades to grid capacity, processes such as digitization, capacity optimization, and congestion management are becoming vital for delivering reliable services.
Power Grid Model is an open source project from Linux Foundation Energy and provides a calculation engine that is increasingly essential for DSOs. It offers a standards-based foundation enabling real-time power systems analysis, simulations of electrical power grids, and sophisticated what-if analysis. In addition, it enables in-depth studies and analysis of the electrical power grid’s behavior and performance. This comprehensive model incorporates essential factors such as power generation capacity, electrical losses, voltage levels, power flows, and system stability.
Power Grid Model is currently being applied in a wide variety of use cases, including grid planning, expansion, reliability, and congestion studies. It can also help in analyzing the impact of renewable energy integration, assessing the effects of disturbances or faults, and developing strategies for grid control and optimization.
What to expect
For the upcoming meetup we are organizing, we have an exciting lineup of activities planned:
-Insightful presentations covering two practical applications of the Power Grid Model.
-An update on the latest advancements in Power Grid -Model technology during the first and second quarters of 2024.
-An interactive brainstorming session to discuss and propose new feature requests.
-An opportunity to connect with fellow Power Grid Model enthusiasts and users.
Programming Foundation Models with DSPy - Meetup SlidesZilliz
Prompting language models is hard, while programming language models is easy. In this talk, I will discuss the state-of-the-art framework DSPy for programming foundation models with its powerful optimizers and runtime constraint system.
Monitoring and Managing Anomaly Detection on OpenShift.pdfTosin Akinosho
Monitoring and Managing Anomaly Detection on OpenShift
Overview
Dive into the world of anomaly detection on edge devices with our comprehensive hands-on tutorial. This SlideShare presentation will guide you through the entire process, from data collection and model training to edge deployment and real-time monitoring. Perfect for those looking to implement robust anomaly detection systems on resource-constrained IoT/edge devices.
Key Topics Covered
1. Introduction to Anomaly Detection
- Understand the fundamentals of anomaly detection and its importance in identifying unusual behavior or failures in systems.
2. Understanding Edge (IoT)
- Learn about edge computing and IoT, and how they enable real-time data processing and decision-making at the source.
3. What is ArgoCD?
- Discover ArgoCD, a declarative, GitOps continuous delivery tool for Kubernetes, and its role in deploying applications on edge devices.
4. Deployment Using ArgoCD for Edge Devices
- Step-by-step guide on deploying anomaly detection models on edge devices using ArgoCD.
5. Introduction to Apache Kafka and S3
- Explore Apache Kafka for real-time data streaming and Amazon S3 for scalable storage solutions.
6. Viewing Kafka Messages in the Data Lake
- Learn how to view and analyze Kafka messages stored in a data lake for better insights.
7. What is Prometheus?
- Get to know Prometheus, an open-source monitoring and alerting toolkit, and its application in monitoring edge devices.
8. Monitoring Application Metrics with Prometheus
- Detailed instructions on setting up Prometheus to monitor the performance and health of your anomaly detection system.
9. What is Camel K?
- Introduction to Camel K, a lightweight integration framework built on Apache Camel, designed for Kubernetes.
10. Configuring Camel K Integrations for Data Pipelines
- Learn how to configure Camel K for seamless data pipeline integrations in your anomaly detection workflow.
11. What is a Jupyter Notebook?
- Overview of Jupyter Notebooks, an open-source web application for creating and sharing documents with live code, equations, visualizations, and narrative text.
12. Jupyter Notebooks with Code Examples
- Hands-on examples and code snippets in Jupyter Notebooks to help you implement and test anomaly detection models.
Ocean lotus Threat actors project by John Sitima 2024 (1).pptxSitimaJohn
Ocean Lotus cyber threat actors represent a sophisticated, persistent, and politically motivated group that poses a significant risk to organizations and individuals in the Southeast Asian region. Their continuous evolution and adaptability underscore the need for robust cybersecurity measures and international cooperation to identify and mitigate the threats posed by such advanced persistent threat groups.
Building Production Ready Search Pipelines with Spark and MilvusZilliz
Spark is the widely used ETL tool for processing, indexing and ingesting data to serving stack for search. Milvus is the production-ready open-source vector database. In this talk we will show how to use Spark to process unstructured data to extract vector representations, and push the vectors to Milvus vector database for search serving.
Ivanti’s Patch Tuesday breakdown goes beyond patching your applications and brings you the intelligence and guidance needed to prioritize where to focus your attention first. Catch early analysis on our Ivanti blog, then join industry expert Chris Goettl for the Patch Tuesday Webinar Event. There we’ll do a deep dive into each of the bulletins and give guidance on the risks associated with the newly-identified vulnerabilities.
Webinar: Designing a schema for a Data WarehouseFederico Razzoli
Are you new to data warehouses (DWH)? Do you need to check whether your data warehouse follows the best practices for a good design? In both cases, this webinar is for you.
A data warehouse is a central relational database that contains all measurements about a business or an organisation. This data comes from a variety of heterogeneous data sources, which includes databases of any type that back the applications used by the company, data files exported by some applications, or APIs provided by internal or external services.
But designing a data warehouse correctly is a hard task, which requires gathering information about the business processes that need to be analysed in the first place. These processes must be translated into so-called star schemas, which means, denormalised databases where each table represents a dimension or facts.
We will discuss these topics:
- How to gather information about a business;
- Understanding dictionaries and how to identify business entities;
- Dimensions and facts;
- Setting a table granularity;
- Types of facts;
- Types of dimensions;
- Snowflakes and how to avoid them;
- Expanding existing dimensions and facts.
HCL Notes and Domino License Cost Reduction in the World of DLAUpanagenda
Webinar Recording: https://www.panagenda.com/webinars/hcl-notes-and-domino-license-cost-reduction-in-the-world-of-dlau/
The introduction of DLAU and the CCB & CCX licensing model caused quite a stir in the HCL community. As a Notes and Domino customer, you may have faced challenges with unexpected user counts and license costs. You probably have questions on how this new licensing approach works and how to benefit from it. Most importantly, you likely have budget constraints and want to save money where possible. Don’t worry, we can help with all of this!
We’ll show you how to fix common misconfigurations that cause higher-than-expected user counts, and how to identify accounts which you can deactivate to save money. There are also frequent patterns that can cause unnecessary cost, like using a person document instead of a mail-in for shared mailboxes. We’ll provide examples and solutions for those as well. And naturally we’ll explain the new licensing model.
Join HCL Ambassador Marc Thomas in this webinar with a special guest appearance from Franz Walder. It will give you the tools and know-how to stay on top of what is going on with Domino licensing. You will be able lower your cost through an optimized configuration and keep it low going forward.
These topics will be covered
- Reducing license cost by finding and fixing misconfigurations and superfluous accounts
- How do CCB and CCX licenses really work?
- Understanding the DLAU tool and how to best utilize it
- Tips for common problem areas, like team mailboxes, functional/test users, etc
- Practical examples and best practices to implement right away
OpenID AuthZEN Interop Read Out - AuthorizationDavid Brossard
During Identiverse 2024 and EIC 2024, members of the OpenID AuthZEN WG got together and demoed their authorization endpoints conforming to the AuthZEN API
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdfChart Kalyan
A Mix Chart displays historical data of numbers in a graphical or tabular form. The Kalyan Rajdhani Mix Chart specifically shows the results of a sequence of numbers over different periods.
From data centers to fog computing: the evaporating cloud
1. From data centers to fog computing:
the evaporating cloud
Guillaume Pierre
Journées Cloud 2018
From data centers to fog computing 1 / 37
2. Table of Contents
1 Introduction
2 Fog computing
3 Making fog servers REALLY small
4 Toward proximity-aware Kubernetes
5 Conclusion
From data centers to fog computing Introduction 2 / 37
3. The new mobile computing landscape
Since 2016: mobile network trac xed trac
From data centers to fog computing Introduction 3 / 37
4. New types of mobile applications
Interactive applications require ultra-low network latencies
E.g., augmented reality require end-to-end delays under 20 ms
But latencies to the closest data center are 20-30 ms using wired
networks, up to 50-150 ms on 4G mobile networks!!!
From data centers to fog computing Introduction 4 / 37
5. The new IoT landscape
Fact: IoT-generated trac grows faster than the Internet backbone
capacity
From data centers to fog computing Introduction 5 / 37
6. New types of mobile applications
Throughput-oriented applications require local computations
E.g., distributed videosurveillance is relevant only close to the cameras
Why waste long-distance network resources?
From data centers to fog computing Introduction 6 / 37
7. Question
Who owns computing resources located
closest to the mobile end users
and the IoT devices?
Answer: Mobile network operators
From data centers to fog computing Introduction 7 / 37
8. Table of Contents
1 Introduction
2 Fog computing
3 Making fog servers REALLY small
4 Toward proximity-aware Kubernetes
5 Conclusion
From data centers to fog computing Fog computing 8 / 37
10. What is fog computing?
Fog computing = cloud
+ edge
+ end-user devices
as a single execution platform
Low latency
Localized trac
Less global trac
Better reliability
Privacy
. . .
From data centers to fog computing Fog computing 10 / 37
11. Fog = Cloud
We need cloud servers close to the users, but the users are everywhere
(and they are mobile)
Let's place one cloud server within short range of any end-user device
⇒ Fog computing resources will need to be distributed in thousands of
locations
Fog nodes will be connected with commodity networks
Forget your multi-Gbps data center networks!
Fog nodes must be small, cheap, easy to deploy and to replace
From data centers to fog computing Fog computing 11 / 37
12. Fog = Cloud
Typical Cloud platform
Few data centers
Lots of resources per data
center
High-performance networks
Cloud resource location is
(mostly) irrelevant
Typical Fog platform
Huge number of
points-of-presence
Few resources per PoP
Commodity networks
Fog resource location is
extremely important
From data centers to fog computing Fog computing 12 / 37
13. The FogGuru project
No general-purpose reference fog platform available
Mostly cloud systems which pretend to address fog concerns
But with very little solutions to the specic challenges of fog computing
FogGuru will investigate some of the most challenging
management issues in fog platforms: resource scheduling, service
migration, autonomous management, anomaly detection. . .
No fog-specic middleware
FogGuru will investigate how to adapt Apache Flink for the fog
No guidelines on how to develop new fog applications
FogGuru will develop blueprints for latency-sensitive and
throughput-intensive applications
No highly-trained specialist in fog computing
FogGuru will train 8 PhD students on various aspects of fog
computing
www.fogguru.eu
From data centers to fog computing Fog computing 13 / 37
14. Table of Contents
1 Introduction
2 Fog computing
3 Making fog servers REALLY small
4 Toward proximity-aware Kubernetes
5 Conclusion
From data centers to fog computing Making fog servers REALLY small 14 / 37
15. Our powerful server machines
From data centers to fog computing Making fog servers REALLY small 15 / 37
16. Our powerful server machines
From data centers to fog computing Making fog servers REALLY small 15 / 37
19. Getting the best out of our RPIs: le system tuning
SD Cards were designed to store photos, videos, etc.
Small number of large les
⇒ Large physical block sizes
But Linux le systems expect small blocks by default
21. Align partitions on the erase block boundaries (multiples of 4 MB)
http://3gfp.com/wp/2014/07/formatting-sd-cards-for-speed-and-lifetime/
From data centers to fog computing Making fog servers REALLY small 18 / 37
22. Getting the best out of our RPIs: swappiness tuning
Swappiness is the kernel parameter that denes how much (and how often)
your Linux kernel will copy RAM contents to swap. This parameter's
default value is 60 and it can take anything from 0 to 100. The higher
the value of the swappiness parameter, the more aggressively your kernel
will swap. https://www.howtoforge.com/tutorial/linux-swappiness/
What's the right swappiness value for us?
RPI-specic linux distributions set it to a very low value (e.g., 1)
We changed it to the maximum value: 100
If your machine is going to swap anyway (because of small memory),
then better do it out of the critical path
Also the le system cache makes use of all the unused RAM
Actually this results in less I/O. . .
https://hal.inria.fr/hal-01446483v1
From data centers to fog computing Making fog servers REALLY small 19 / 37
23. Container deployment in a RPI
Let's deploy a very simple Docker container on the RPI3
Standard ubuntu container (∼45 MB) + one extra 51-MB layer
0
500
1000
1500
2000
2500
3000
50 100 150 200 250
0
2000
4000
6000
8000
10000
NetworkThroughput(KB/s)
DiskI/O(KB/s)
Time(s)
Network and Disk activities during container deployment
Received(KB/s)
Disk Write(KB/s)
CPU usage(%)
Total deployment time: about 4 minutes!!!
24. Can we make that faster?
From data centers to fog computing Making fog servers REALLY small 20 / 37
25. How Docker deploys a new container image
A container image is composed of multiple layers
Each layer complements or modies the previous one
Layer 0 with the base OS, layer 1 with extra cong les, layer 2 with
the necessary middleware, layer 3 with the application, layer 4 with a
quick-and-dirty x for some wrong cong le, etc.
What takes a lot of time is bringing the image from the external
repository to the raspberry pi
Starting the container itself is much faster
Deployment process:
1 Download all layers simultaneously
2 Decompress and extract each layer to disk sequentially
3 Start the container
Question: What's wrong with this?
From data centers to fog computing Making fog servers REALLY small 21 / 37
26. Sequential layer download
Observation: parallel downloading delays the time when the rst
download has completed
Idea: let's download layers sequentially. This should allow the
deployment process to use the bandwidth and disk I/O simultaneously
Time
Download layer 1
Download layer 2
Download layer 3
Extract layer 1
Extract layer 2Download layer 2
Extract layer 3
Standard Docker deployment
Time
Download layer 1
Download layer 2
Download layer 3
Extract layer 1
Extract layer 2Download layer 2
Extract layer 3
Our proposal
Performance gains:
∼ 3 6% on fast networks
∼ 6 12% on slow (256 kbps) networks
Best gains with several big layers + slow networks
From data centers to fog computing Making fog servers REALLY small 22 / 37
27. Let's speed up the decompression part
Image layers are always delivered in compressed form, usually with gzip
Docker calls gunzip to decompress the images
But gunzip is single-threaded while RPis have 4 CPU cores!
Let's use a multithreaded gunzip implementation instead
Performance improvement: ∼15 18% on the entire deployment
process
Much more if we look just at the decompression part of the process. . .
From data centers to fog computing Making fog servers REALLY small 23 / 37
28. Let's pipeline the download, decompression and extraction
processes
Idea: let's split the download/decompress/extract process into three
threads per layer
And pipeline the three parts: start the decompression and extract
processes as soon as the rst bytes are downloaded
29. Intersting thread synchronization exercise. . .
Download Decompress Extract
Download Decompress Extract
Download Decompress Extract
Layer 0
Layer 1
Layer 2
Performance improvement: ∼ 20 55%
From data centers to fog computing Making fog servers REALLY small 24 / 37
30. Let's combine all three techniques
0
200
400
600
800
1000
50 100 150 200 250 300 350 400 450
0
2000
4000
6000
8000
10000
NetworkThroughput(KB/s)
DiskI/O(KB/s)
Time(s)
Recieved(KB/s) Disk Write(KB/s) CPU usage(%)
Before
0
200
400
600
800
1000
50 100 150 200 250 300 350 400 450
0
2000
4000
6000
8000
10000
NetworkThroughput(KB/s)
DiskI/O(KB/s)
Time(s)
Received(KB/s) Disk Write(KB/s) CPU usage(%)
After
Combined performance gain: 17 73%
Lowest gains obtained with low (256 kbps) networks. The network is
the bottleneck, not many local optimizations are possible
When using a decent network connection: ∼ 50 73% gains
From data centers to fog computing Making fog servers REALLY small 25 / 37
31. What about real server machines?
We reproduced the same experiments on a real server machine
2x 10-core CPU (Intel Xeon E5-2660v2)
128 GB RAM
2x 10 Gbps network
Performance gains: 29-36%
Most eective for images with multiple large layers
These optimizations are also relevant
in cloud-like environments
Docker Container Deployment in Fog Computing Infrastructures. A. Ahmed and G. Pierre. In Proc.
IEEE EDGE conference, July 2018.
From data centers to fog computing Making fog servers REALLY small 26 / 37
32. Table of Contents
1 Introduction
2 Fog computing
3 Making fog servers REALLY small
4 Toward proximity-aware Kubernetes
5 Conclusion
From data centers to fog computing Toward proximity-aware Kubernetes 27 / 37
33. The fog according to us
INTERNET
(commodity networks,
very heterogeneous)
eth0
Wifi hotspot
+ fog server
From data centers to fog computing Toward proximity-aware Kubernetes 28 / 37
34. Kubernetes is a great platform. . .
Container-based ⇒ lightweight, can run on a Raspberry Pi
Pods and services ⇒ easy to deploy and (re-)scale applications
Integrated autoscaling ⇒ ready for uctuating workloads
Designed around feedback control loops ⇒ simple and robust
Large community of users and developers ⇒ we can nd help
whenever necessary
From data centers to fog computing Toward proximity-aware Kubernetes 29 / 37
35. . . . but it is not ready for the fog
From data centers to fog computing Toward proximity-aware Kubernetes 30 / 37
36. . . . but it is not ready for the fog
From data centers to fog computing Toward proximity-aware Kubernetes 30 / 37
37. . . . but it is not ready for the fog
From data centers to fog computing Toward proximity-aware Kubernetes 30 / 37
38. . . . but it is not ready for the fog
From data centers to fog computing Toward proximity-aware Kubernetes 30 / 37
39. . . . but it is not ready for the fog
Scheduling policies:
Random
Round-Robin
Least loaded node
Hand-picked
. . . but no location-aware policy!
From data centers to fog computing Toward proximity-aware Kubernetes 30 / 37
40. . . . but it is not ready for the fog
From data centers to fog computing Toward proximity-aware Kubernetes 30 / 37
41. . . . but it is not ready for the fog
Load-balancing policies:
Random
Round-Robin
Least loaded node
Hand-picked
. . . but no location-aware policy!
From data centers to fog computing Toward proximity-aware Kubernetes 30 / 37
42. How Kubernetes routes user requests
(not necessary in our use cases): expose a public IP address to
external users
Route trac from any Kubernetes cluster node to one of the pods
1 The central kube-proxy controller detects a change in the set of pods
2 It requests all cluster nodes to update their routes
3 Routes are implemented using iptables
By default: requests are randomly distributed according to weights
(which also seem random)
From data centers to fog computing Toward proximity-aware Kubernetes 31 / 37
43. Making request redirection location-aware
Let's change kube-proxy:
Detect a change in the set of pods
Request each cluster node to update its routes to nearby pods
Where should each node route its trac?
To the closest pod according to GPS locations
To the closest pod according to Vivaldi coordinates
Load-balance to the n closest pods according to their load
. . .
We need to deploy additional monitoring functionality
Vivaldi. . .
From data centers to fog computing Toward proximity-aware Kubernetes 32 / 37
44. From data centers to fog computing Toward proximity-aware Kubernetes 33 / 37
45. Table of Contents
1 Introduction
2 Fog computing
3 Making fog servers REALLY small
4 Toward proximity-aware Kubernetes
5 Conclusion
From data centers to fog computing Conclusion 34 / 37
46. Conclusion
Cloud data centers are very powerful and exible
But not all applications can use them (latency, trac locality)
If we evaporate a cloud, then we get a fog
Extremely distributed infrastructure: there must be a server node close
to every end user
Server nodes must be small, cheap, easy to add and replace
Server nodes are very far from each other
This is only the beginning
No satisfactory edge/fog platforms are available today
(we are not even close)
There remains thousands of potential PhD research topics in this
domain
www.fogguru.eu
From data centers to fog computing Conclusion 35 / 37
48. This work is part of a project that has received funding from the European
Union's Horizon 2020 research and innovation programme under the Marie
Skªodowska-Curie grant agreement No 765452. The information and views
set out in this publication are those of the author(s) and do not necessarily
reect the ocial opinion of the European Union. Neither the European
Union institutions and bodies nor any person acting on their behalf may be
held responsible for the use which may be made of the information
contained therein.
From data centers to fog computing Conclusion 37 / 37