Super Infrastructure for Large-Scale Experimental Computer Science, (Almost) everything you wanted to know about SILECS/SLICES but didn't dare to ask. Presentation at "journées du GDR RSD", Nantes, Jan. 23, 2020/.
SILECS/SLICES - Super Infrastructure for Large-Scale Experimental Computer Sc...Frederic Desprez
The aim of the SILECS and SLICES projects is to design and build a large infrastructure for experimental research on various aspects of distributed computing, from small connected objects to the large data centres of tomorrow. This infrastructure will allow end-to-end experimentation with software and applications at all levels of the software layers, from event capture (sensors, actuators) to data processing and storage, to radio transmission management and dynamic deployment of edge computing services, enabling reproducible research on all-point programmable networks, ... SILECS is the french node of a european infrastructure called SLICES.
Interoperability and scalability with microservices in scienceOla Spjuth
Microservices have emerged as a modern interpretation of service-oriented architectures where processes are small and communicate over a network using lightweight protocols to fulfill a goal.
In this talk I will present our work on microservices, and how they can be used to empower interoperable and scalable analysis services and pipelines in virtual infrastructures on cloud computing resources.
I will also give examples and experiences from the PhenoMeNal H2020 project where a developer community in metabolomics is moving to such architecture.
Un cloud pour comparer nos gènes aux images du cerveau" Le pionnier des bases de données, aujourd'hui disparu, Jim Gray avait annoncé en 2007 l'emergence d'un 4eme paradigme scientifique: celui d'une recherche scientifique numérique entierement guidée par l'exploration de données massives. Cette vision est aujourd'hui la réalité de tous les jours dans les laboratoire de recherche scientifique, et elle va bien au delà de ce que l'on appelle communément "BIG DATA". Microsoft Research et Inria on démarré en 2010 un projet intitulé Azure-Brain (ou A-Brain) dont l'originalité consiste à a la fois construire au dessus de Windows Azure une nouvelle plateforme d'acces aux données massives pour les applications scientifiques, et de se confronter à la réalité de la recherche scientifique. Dans cette session nous vous proposons dans une premiere partie de resituer les enjeux recherche concernant la gestion de données massives dans le cloud, et ensuite de vous presenter la plateforme "TOMUS Blob" cloud storage optimisé sur Azure. Enfin nous vous presenterons le projet A-Brain et les résultats que nous avons obtenus: La neuro-imagerie contribue au diagnostic de certaines maladies du système nerveux. Mais nos cerveaux s'avèrent tous un peu différents les uns des autres. Cette variabilité complique l'interprétation médicale. D'où l'idée de corréler ldes images IRM du cerveaux et le patrimoine génétique de chaque patient afin de mieux délimiter les régions cérébrales qui présentent un intérêt symptomatique. Les images IRM haute définition de ce projet sont produites par la plate-forme Neurospin du CEA (Saclay). Problème pour Les chercheurs : la masse d'informations à traiter. Le CV génétique d'un individu comporte environ un million de données. À cela s'ajoutent des volumes tout aussi colossaux de pixel 3D pour décrire les images. Un data deluge: des peta octets de donnés et potentiellement des années de calcul. C'est donc ici qu'entre en jeu le cloud et une plateforme optimisée sur Azure pour traiter des applications massivement parallèles sur des données massives... Comme l'explique Gabriel Antoniu, son responsable, cette équipe de recherche rennaise a développé “des mécanismes de stockage efficaces pour améliorer l'accès à ces données massives et optimiser leur traitement. Nos développements permettent de répondre aux besoins applicatifs de nos collègues de Saclay.
IDB-Cloud Providing Bioinformatics Services on Cloudstratuslab
A presentation of IDB (Infrastructure Distributed for Biology) using StratusLab technology by Christophe Blanchet and Clément Gauthey at Lille, France, May 2013.
SILECS: Super Infrastructure for Large-scale Experimental Computer ScienceFrederic Desprez
SILECS, based on two existing infrastructure (FIT and Grid'5000), aims to provide a large robust, trustable and scalable instrument for research in
distributed computing and networks. Experiments from the Internet of Things, data centers, cloud computing, security services, and the networks
connecting them will be possible, in a reproducible way, on various hardware and software. This instrument will offer a multi-platform experimental
infrastructure (HPC, Cloud, Big Data, Software Defined Storage, IoT, wireless, Software Defined Network / Radio) capable of exploring the
infrastructures that will be deployed tomorrow and assist researchers and industrial about how to design, build and operate a multi-scale, robust and
safe computer system. Diverse digital resources (compute, storage, link, IO devices) are be assembled to support a “playground” at scale.
Grid'5000: Running a Large Instrument for Parallel and Distributed Computing ...Frederic Desprez
The increasing complexity of available infrastructures (hierarchical, parallel, distributed, etc.) with specific features (caches, hyper-threading, dual core, etc.) makes it extremely difficult to build analytical models that allow for a satisfying prediction. Hence, it raises the question on how to validate algorithms and software systems if a realistic analytic study is not possible. As for many other sciences, the one answer is experimental validation. However, such experimentations rely on the availability of an instrument able to validate every level of the software stack and offering different hardware and software facilities about compute, storage, and network resources.
Almost ten years after its premises, the Grid'5000 testbed has become one of the most complete testbed for designing or evaluating large-scale distributed systems. Initially dedicated to the study of large HPC facilities, Grid’5000 has evolved in order to address wider concerns related to Desktop Computing, the Internet of Services and more recently the Cloud Computing paradigm. We now target new processors features such as hyperthreading, turbo boost, and power management or large applications managing big data. In this keynote we will both address the issue of experiments in HPC and computer science and the design and usage of the Grid'5000 platform for various kind of applications.
SILECS/SLICES - Super Infrastructure for Large-Scale Experimental Computer Sc...Frederic Desprez
The aim of the SILECS and SLICES projects is to design and build a large infrastructure for experimental research on various aspects of distributed computing, from small connected objects to the large data centres of tomorrow. This infrastructure will allow end-to-end experimentation with software and applications at all levels of the software layers, from event capture (sensors, actuators) to data processing and storage, to radio transmission management and dynamic deployment of edge computing services, enabling reproducible research on all-point programmable networks, ... SILECS is the french node of a european infrastructure called SLICES.
Interoperability and scalability with microservices in scienceOla Spjuth
Microservices have emerged as a modern interpretation of service-oriented architectures where processes are small and communicate over a network using lightweight protocols to fulfill a goal.
In this talk I will present our work on microservices, and how they can be used to empower interoperable and scalable analysis services and pipelines in virtual infrastructures on cloud computing resources.
I will also give examples and experiences from the PhenoMeNal H2020 project where a developer community in metabolomics is moving to such architecture.
Un cloud pour comparer nos gènes aux images du cerveau" Le pionnier des bases de données, aujourd'hui disparu, Jim Gray avait annoncé en 2007 l'emergence d'un 4eme paradigme scientifique: celui d'une recherche scientifique numérique entierement guidée par l'exploration de données massives. Cette vision est aujourd'hui la réalité de tous les jours dans les laboratoire de recherche scientifique, et elle va bien au delà de ce que l'on appelle communément "BIG DATA". Microsoft Research et Inria on démarré en 2010 un projet intitulé Azure-Brain (ou A-Brain) dont l'originalité consiste à a la fois construire au dessus de Windows Azure une nouvelle plateforme d'acces aux données massives pour les applications scientifiques, et de se confronter à la réalité de la recherche scientifique. Dans cette session nous vous proposons dans une premiere partie de resituer les enjeux recherche concernant la gestion de données massives dans le cloud, et ensuite de vous presenter la plateforme "TOMUS Blob" cloud storage optimisé sur Azure. Enfin nous vous presenterons le projet A-Brain et les résultats que nous avons obtenus: La neuro-imagerie contribue au diagnostic de certaines maladies du système nerveux. Mais nos cerveaux s'avèrent tous un peu différents les uns des autres. Cette variabilité complique l'interprétation médicale. D'où l'idée de corréler ldes images IRM du cerveaux et le patrimoine génétique de chaque patient afin de mieux délimiter les régions cérébrales qui présentent un intérêt symptomatique. Les images IRM haute définition de ce projet sont produites par la plate-forme Neurospin du CEA (Saclay). Problème pour Les chercheurs : la masse d'informations à traiter. Le CV génétique d'un individu comporte environ un million de données. À cela s'ajoutent des volumes tout aussi colossaux de pixel 3D pour décrire les images. Un data deluge: des peta octets de donnés et potentiellement des années de calcul. C'est donc ici qu'entre en jeu le cloud et une plateforme optimisée sur Azure pour traiter des applications massivement parallèles sur des données massives... Comme l'explique Gabriel Antoniu, son responsable, cette équipe de recherche rennaise a développé “des mécanismes de stockage efficaces pour améliorer l'accès à ces données massives et optimiser leur traitement. Nos développements permettent de répondre aux besoins applicatifs de nos collègues de Saclay.
IDB-Cloud Providing Bioinformatics Services on Cloudstratuslab
A presentation of IDB (Infrastructure Distributed for Biology) using StratusLab technology by Christophe Blanchet and Clément Gauthey at Lille, France, May 2013.
SILECS: Super Infrastructure for Large-scale Experimental Computer ScienceFrederic Desprez
SILECS, based on two existing infrastructure (FIT and Grid'5000), aims to provide a large robust, trustable and scalable instrument for research in
distributed computing and networks. Experiments from the Internet of Things, data centers, cloud computing, security services, and the networks
connecting them will be possible, in a reproducible way, on various hardware and software. This instrument will offer a multi-platform experimental
infrastructure (HPC, Cloud, Big Data, Software Defined Storage, IoT, wireless, Software Defined Network / Radio) capable of exploring the
infrastructures that will be deployed tomorrow and assist researchers and industrial about how to design, build and operate a multi-scale, robust and
safe computer system. Diverse digital resources (compute, storage, link, IO devices) are be assembled to support a “playground” at scale.
Grid'5000: Running a Large Instrument for Parallel and Distributed Computing ...Frederic Desprez
The increasing complexity of available infrastructures (hierarchical, parallel, distributed, etc.) with specific features (caches, hyper-threading, dual core, etc.) makes it extremely difficult to build analytical models that allow for a satisfying prediction. Hence, it raises the question on how to validate algorithms and software systems if a realistic analytic study is not possible. As for many other sciences, the one answer is experimental validation. However, such experimentations rely on the availability of an instrument able to validate every level of the software stack and offering different hardware and software facilities about compute, storage, and network resources.
Almost ten years after its premises, the Grid'5000 testbed has become one of the most complete testbed for designing or evaluating large-scale distributed systems. Initially dedicated to the study of large HPC facilities, Grid’5000 has evolved in order to address wider concerns related to Desktop Computing, the Internet of Services and more recently the Cloud Computing paradigm. We now target new processors features such as hyperthreading, turbo boost, and power management or large applications managing big data. In this keynote we will both address the issue of experiments in HPC and computer science and the design and usage of the Grid'5000 platform for various kind of applications.
EDF2014: Franck Cotton & Kamel Gadouche, France: TeraLab - A Secure Big Data...European Data Forum
Selected Talk of Franck Cotton, Technology Advisor, Institut National de la Statistique et des Etudes Economiques, France & Kamel Gadouche, Director, Centre d'Accès Sécurisé aux Données / Groupe des Ecoles Nationales d'Economie et Statistique, France at the European Data Forum 2014, 19 March 2014 in Athens, Greece: TeraLab - A Secure Big Data Platform, Description And Use Cases
David Loureiro - Presentation at HP's HPC & OSL TESSysFera
David Loureiro, SysFera CEO, talks about "Managing large-scale, heterogeneous infrastructures: from DIET to SysFera-DS" at HP's High Performance Computing and Open Source & Linux Technical Excellence Symposium that took place on the 19-23 March, 2012, in Grenoble, France.
To foster greater and more consistent use of the new 100 Gbps connections that is being deployed in the national RNP backbone, the e-Cyber project aims at delivering high-performing services to the most infrastructure-demanding research centers in Brazil. To do this, the project is getting inspired by the “superfacility” concept, which is adopted by initiatives like GRP (Global Research Platform) and EOSC (European Open Science Cloud). However, one of our biggest challenges is to engage the client institutions and bring them to co-create solutions and participate in the project governance.
Adoption of Cloud Computing in Scientific ResearchYehia El-khatib
Some might say the scientific research community is somewhat behind the curve of adopting the cloud. In this talk, I present a few examples of adopting the cloud from the wider research community. I also highlight some of the aspects by which cloud computing could affect scientific research in the near future and the associated challenges.
40 Powers of 10 - Simulating the Universe with the DiRAC HPC Facilityinside-BigData.com
In this deck from the Swiss HPC Conference, Mark Wilkinson presents: 40 Powers of 10 - Simulating the Universe with the DiRAC HPC Facility.
"DiRAC is the integrated supercomputing facility for theoretical modeling and HPC-based research in particle physics, and astrophysics, cosmology, and nuclear physics, all areas in which the UK is world-leading. DiRAC provides a variety of compute resources, matching machine architecture to the algorithm design and requirements of the research problems to be solved. As a single federated Facility, DiRAC allows more effective and efficient use of computing resources, supporting the delivery of the science programs across the STFC research communities. It provides a common training and consultation framework and, crucially, provides critical mass and a coordinating structure for both small- and large-scale cross-discipline science projects, the technical support needed to run and develop a distributed HPC service, and a pool of expertise to support knowledge transfer and industrial partnership projects. The on-going development and sharing of best-practice for the delivery of productive, national HPC services with DiRAC enables STFC researchers to produce world-leading science across the entire STFC science theory program."
Watch the video: https://wp.me/p3RLHQ-k94
Learn more: https://dirac.ac.uk/
and
http://hpcadvisorycouncil.com/events/2019/swiss-workshop/agenda.php
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
From Jisc's campus network engineering for data-intensive science workshop on 19 October 2016.
https://www.jisc.ac.uk/events/campus-network-engineering-for-data-intensive-science-workshop-19-oct-2016
(R)evolution of the computing continuum - A few challengesFrederic Desprez
Initially proposed to interconnect computers worldwide, the Internet has significantly evolved to become in two decades a key element in almost all our activities. This (r)evolution mainly relies on the progress that has been achieved in computation and communication fields and that has led to the well-known and widely spread Cloud Computing paradigm.
With the emergence of the Internet of Things (IoT), stakeholders expect a new revolution that will push, once again, the limits of the Internet, in particular by favouring the convergence between physical and virtual worlds. This convergence is about to be made possible thanks to the development of minimalist sensors as well as complex industrial physical machines that can be connected to the Internet through edge computing infrastructures.
Among the obstacles to this new generation of Internet services is the development of a convenient and powerful framework that should allow operators, and devops, to manage the life-cycle of both the digital infrastructures and the applications deployed on top of these infrastructures, throughout the cloud to IoT continuum.
In this keynote, Frédéric Desprez and his colleague Adrien Lebre presented research issues and provide preliminary answers to identify whether the challenges brought by this new paradigm is an evolution or a revolution for our community.
EDF2014: Franck Cotton & Kamel Gadouche, France: TeraLab - A Secure Big Data...European Data Forum
Selected Talk of Franck Cotton, Technology Advisor, Institut National de la Statistique et des Etudes Economiques, France & Kamel Gadouche, Director, Centre d'Accès Sécurisé aux Données / Groupe des Ecoles Nationales d'Economie et Statistique, France at the European Data Forum 2014, 19 March 2014 in Athens, Greece: TeraLab - A Secure Big Data Platform, Description And Use Cases
David Loureiro - Presentation at HP's HPC & OSL TESSysFera
David Loureiro, SysFera CEO, talks about "Managing large-scale, heterogeneous infrastructures: from DIET to SysFera-DS" at HP's High Performance Computing and Open Source & Linux Technical Excellence Symposium that took place on the 19-23 March, 2012, in Grenoble, France.
To foster greater and more consistent use of the new 100 Gbps connections that is being deployed in the national RNP backbone, the e-Cyber project aims at delivering high-performing services to the most infrastructure-demanding research centers in Brazil. To do this, the project is getting inspired by the “superfacility” concept, which is adopted by initiatives like GRP (Global Research Platform) and EOSC (European Open Science Cloud). However, one of our biggest challenges is to engage the client institutions and bring them to co-create solutions and participate in the project governance.
Adoption of Cloud Computing in Scientific ResearchYehia El-khatib
Some might say the scientific research community is somewhat behind the curve of adopting the cloud. In this talk, I present a few examples of adopting the cloud from the wider research community. I also highlight some of the aspects by which cloud computing could affect scientific research in the near future and the associated challenges.
40 Powers of 10 - Simulating the Universe with the DiRAC HPC Facilityinside-BigData.com
In this deck from the Swiss HPC Conference, Mark Wilkinson presents: 40 Powers of 10 - Simulating the Universe with the DiRAC HPC Facility.
"DiRAC is the integrated supercomputing facility for theoretical modeling and HPC-based research in particle physics, and astrophysics, cosmology, and nuclear physics, all areas in which the UK is world-leading. DiRAC provides a variety of compute resources, matching machine architecture to the algorithm design and requirements of the research problems to be solved. As a single federated Facility, DiRAC allows more effective and efficient use of computing resources, supporting the delivery of the science programs across the STFC research communities. It provides a common training and consultation framework and, crucially, provides critical mass and a coordinating structure for both small- and large-scale cross-discipline science projects, the technical support needed to run and develop a distributed HPC service, and a pool of expertise to support knowledge transfer and industrial partnership projects. The on-going development and sharing of best-practice for the delivery of productive, national HPC services with DiRAC enables STFC researchers to produce world-leading science across the entire STFC science theory program."
Watch the video: https://wp.me/p3RLHQ-k94
Learn more: https://dirac.ac.uk/
and
http://hpcadvisorycouncil.com/events/2019/swiss-workshop/agenda.php
Sign up for our insideHPC Newsletter: http://insidehpc.com/newsletter
From Jisc's campus network engineering for data-intensive science workshop on 19 October 2016.
https://www.jisc.ac.uk/events/campus-network-engineering-for-data-intensive-science-workshop-19-oct-2016
(R)evolution of the computing continuum - A few challengesFrederic Desprez
Initially proposed to interconnect computers worldwide, the Internet has significantly evolved to become in two decades a key element in almost all our activities. This (r)evolution mainly relies on the progress that has been achieved in computation and communication fields and that has led to the well-known and widely spread Cloud Computing paradigm.
With the emergence of the Internet of Things (IoT), stakeholders expect a new revolution that will push, once again, the limits of the Internet, in particular by favouring the convergence between physical and virtual worlds. This convergence is about to be made possible thanks to the development of minimalist sensors as well as complex industrial physical machines that can be connected to the Internet through edge computing infrastructures.
Among the obstacles to this new generation of Internet services is the development of a convenient and powerful framework that should allow operators, and devops, to manage the life-cycle of both the digital infrastructures and the applications deployed on top of these infrastructures, throughout the cloud to IoT continuum.
In this keynote, Frédéric Desprez and his colleague Adrien Lebre presented research issues and provide preliminary answers to identify whether the challenges brought by this new paradigm is an evolution or a revolution for our community.
Challenges and Issues of Next Cloud Computing PlatformsFrederic Desprez
Cloud computing has now crossed the frontiers of research to reach industry. It is used every day , whether to exchange emails or make
reservations on web sites. However, many research works remain to be done to improve the performance and functionality of these platforms of tomorrow. In this talk, I will do an overview of some these theoretical and appliead researches done at INRIA and particularly around Clouds distribution, energy monitoring and management, massive data processing and exchange, and resource management.
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...Ramesh Iyer
In today's fast-changing business world, Companies that adapt and embrace new ideas often need help to keep up with the competition. However, fostering a culture of innovation takes much work. It takes vision, leadership and willingness to take risks in the right proportion. Sachin Dev Duggal, co-founder of Builder.ai, has perfected the art of this balance, creating a company culture where creativity and growth are nurtured at each stage.
UiPath Test Automation using UiPath Test Suite series, part 4DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 4. In this session, we will cover Test Manager overview along with SAP heatmap.
The UiPath Test Manager overview with SAP heatmap webinar offers a concise yet comprehensive exploration of the role of a Test Manager within SAP environments, coupled with the utilization of heatmaps for effective testing strategies.
Participants will gain insights into the responsibilities, challenges, and best practices associated with test management in SAP projects. Additionally, the webinar delves into the significance of heatmaps as a visual aid for identifying testing priorities, areas of risk, and resource allocation within SAP landscapes. Through this session, attendees can expect to enhance their understanding of test management principles while learning practical approaches to optimize testing processes in SAP environments using heatmap visualization techniques
What will you get from this session?
1. Insights into SAP testing best practices
2. Heatmap utilization for testing
3. Optimization of testing processes
4. Demo
Topics covered:
Execution from the test manager
Orchestrator execution result
Defect reporting
SAP heatmap example with demo
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
JMeter webinar - integration with InfluxDB and GrafanaRTTS
Watch this recorded webinar about real-time monitoring of application performance. See how to integrate Apache JMeter, the open-source leader in performance testing, with InfluxDB, the open-source time-series database, and Grafana, the open-source analytics and visualization application.
In this webinar, we will review the benefits of leveraging InfluxDB and Grafana when executing load tests and demonstrate how these tools are used to visualize performance metrics.
Length: 30 minutes
Session Overview
-------------------------------------------
During this webinar, we will cover the following topics while demonstrating the integrations of JMeter, InfluxDB and Grafana:
- What out-of-the-box solutions are available for real-time monitoring JMeter tests?
- What are the benefits of integrating InfluxDB and Grafana into the load testing stack?
- Which features are provided by Grafana?
- Demonstration of InfluxDB and Grafana using a practice web application
To view the webinar recording, go to:
https://www.rttsweb.com/jmeter-integration-webinar
Transcript: Selling digital books in 2024: Insights from industry leaders - T...BookNet Canada
The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more.
Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/
Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...UiPathCommunity
💥 Speed, accuracy, and scaling – discover the superpowers of GenAI in action with UiPath Document Understanding and Communications Mining™:
See how to accelerate model training and optimize model performance with active learning
Learn about the latest enhancements to out-of-the-box document processing – with little to no training required
Get an exclusive demo of the new family of UiPath LLMs – GenAI models specialized for processing different types of documents and messages
This is a hands-on session specifically designed for automation developers and AI enthusiasts seeking to enhance their knowledge in leveraging the latest intelligent document processing capabilities offered by UiPath.
Speakers:
👨🏫 Andras Palfi, Senior Product Manager, UiPath
👩🏫 Lenka Dulovicova, Product Program Manager, UiPath
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf91mobiles
91mobiles recently conducted a Smart TV Buyer Insights Survey in which we asked over 3,000 respondents about the TV they own, aspects they look at on a new TV, and their TV buying preferences.
Generating a custom Ruby SDK for your web service or Rails API using Smithyg2nightmarescribd
Have you ever wanted a Ruby client API to communicate with your web service? Smithy is a protocol-agnostic language for defining services and SDKs. Smithy Ruby is an implementation of Smithy that generates a Ruby SDK using a Smithy model. In this talk, we will explore Smithy and Smithy Ruby to learn how to generate custom feature-rich SDKs that can communicate with any web service, such as a Rails JSON API.
Essentials of Automations: Optimizing FME Workflows with ParametersSafe Software
Are you looking to streamline your workflows and boost your projects’ efficiency? Do you find yourself searching for ways to add flexibility and control over your FME workflows? If so, you’re in the right place.
Join us for an insightful dive into the world of FME parameters, a critical element in optimizing workflow efficiency. This webinar marks the beginning of our three-part “Essentials of Automation” series. This first webinar is designed to equip you with the knowledge and skills to utilize parameters effectively: enhancing the flexibility, maintainability, and user control of your FME projects.
Here’s what you’ll gain:
- Essentials of FME Parameters: Understand the pivotal role of parameters, including Reader/Writer, Transformer, User, and FME Flow categories. Discover how they are the key to unlocking automation and optimization within your workflows.
- Practical Applications in FME Form: Delve into key user parameter types including choice, connections, and file URLs. Allow users to control how a workflow runs, making your workflows more reusable. Learn to import values and deliver the best user experience for your workflows while enhancing accuracy.
- Optimization Strategies in FME Flow: Explore the creation and strategic deployment of parameters in FME Flow, including the use of deployment and geometry parameters, to maximize workflow efficiency.
- Pro Tips for Success: Gain insights on parameterizing connections and leveraging new features like Conditional Visibility for clarity and simplicity.
We’ll wrap up with a glimpse into future webinars, followed by a Q&A session to address your specific questions surrounding this topic.
Don’t miss this opportunity to elevate your FME expertise and drive your projects to new heights of efficiency.
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...
SILECS/SLICES
1. SILECS/SLICES
Super Infrastructure for Large-Scale Experimental Computer
Science
(Almost) everything you wanted to know about SILECS/SLICES but didn't dare to ask
F. Desprez – Inria/LIG,
S. Fdida – Sorbonne University
F. Desprez - SILECS/SLICES - Frederic.Desprez@inria.fr
INRIA, CNRS, RENATER, CEA, CPU, CDEFI, IMT, Sorbonne Université, Université Strasbourg, Université Lorraine, Université Grenoble Alpes,
Université Lille 1, Université Rennes 1, Université Toulouse, ENS Lyon, INSA Lyon, …
Journées GDR RSD Nantes hOp://www.silecs.net/
2. Previous Presenta2ons (at least some of them)
• Journées non-théma/ques RESCOM, Grenoble, 17-18 janvier 2019
• F. Desprez, SILECS
• Ecole RESCOM, Anglet, 24-28 juin 2019
• Serge Fdida, Large scale experimentaTon plaUorms
Journées GDR RSD Nantes F. Desprez - SILECS/SLICES - Frederic.Desprez@inria.fr
3. The Discipline of Computing: An Experimental Science
The reality of computer science
- InformaTon
- Computers, networks, algorithms, programs, etc.
Studied objects (hardware, programs, data, protocols, algorithms, networks)
are more and more complex
Modern infrastructures
• Processors have very nice features: caches, hyperthreading, mulT-core, …
• OperaTng system impacts the performance (process scheduling, socket
implementaTon, etc.)
• The runTme environment plays a role (MPICH ≠ OPENMPI)
• Middleware have an impact
• Various parallel architectures that can be heterogeneous, hierarchical,
distributed, dynamic
Journées GDR RSD Nantes F. Desprez - SILECS/SLICES - Frederic.Desprez@inria.fr
4. Good Experiments
A good experiment should fulfill the following properties
– Reproducibility: must give the same result with the same input
– Extensibility: must target possible comparisons with other works and extensions
(more/other processors, larger data sets, different architectures)
– Applicability: must define realistic parameters and must allow for an easy calibration
– “Revisability”: when an implementation does not perform as expected, must help to
identify the reasons
F. Desprez - SILECS/SLICES - Frederic.Desprez@inria.frJournées GDR RSD Nantes
5. Analytic Modeling
Purely analy/cal (mathema/cal)
models
• DemonstraTon of properTes (theorem)
• Models need to be tractable: over-
simplificaTon?
• Good to understand the basic of the
problem
• Most of the Tme ones sTll perform a
experiments (at least for comparison)
For a prac/cal impact (especially in distributed compu/ng): analy/c study not always possible or not
sufficient
Journées GDR RSD Nantes F. Desprez - SILECS/SLICES - Frederic.Desprez@inria.fr
6. Experimental Valida2on
A good alterna/ve to analy/cal valida/on
• Provides a comparison between algorithms and programs
• Provides a validaTon of the model or helps to define the validity domain of
the model
Several methodologies
• Simula/on (SimGrid, NS, …)
• Emula/on (MicroGrid, Distem, …)
• Benchmarking (NAS, SPEC, LINPACK, ….)
• Real-scale (Grid’5000, FIT, FED4Fire, Chameleon, OpenCirrus, PlanetLab, …)
Journées GDR RSD Nantes F. Desprez - SILECS/SLICES - Frederic.Desprez@inria.fr
7. SILECS/SLICES Mo2va2on
• Exponential improvement of
– Electronics (energy consumption, size, cost)
– Capacity of networks (WAN, wireless, new technologies)
• Exponential growth of applications near users
– Smartphones, tablets, connected devices, sensors, …
– Large variety of applications and large community
• Large number of Cloud facilities to cope with generated data
– Many platforms and infrastructures available around the world
– Several offers for IaaS, PaaS, and SaaS platforms
– Public, private, community, and hybrid clouds
– Going toward distributed Clouds (FOG, Edge, extreme Edge)
F. Desprez - SILECS/SLICES - Frederic.Desprez@inria.frJournées GDR RSD Nantes
8. SLICES
• Core partners
• Cyprus
• France
• Greece
• Italy
• Luxembourg
• Spain
• Switzerland
Journées GDR RSD Nantes F. Desprez - SILECS/SLICES - Frederic.Desprez@inria.fr
• In discussion
• Germany
• Hungary
• Norway
• Romania
• …
• Almost core partners
• Belgium
• Netherland
• Poland
• Sweden
• GIANT and naTonal
NRENs
10. SILECS/GRID’5000
• Testbed for research on distributed systems
• Born in 2003 from the observation that we need a better and larger testbed
• HPC, Grids, P2P, and now Cloud computing, and BigData systems
• A complete access to the nodes’ hardware in an exclusive mode
(from one node to the whole infrastructure)
• Dedicated network (RENATER)
• Reconfigurable: nodes with Kadeploy and network with KaVLAN
• Current status
• 8 sites, 36 clusters, 838 nodes, 15116 cores
• Diverse technologies/resources
(Intel, AMD, Myrinet, Infiniband, two GPU clusters, energy probes)
• Some Experiments examples
• In Situ analytics
• Big Data Management
• HPC Programming approaches
• Network modeling and simulation
• Energy consumption evaluation
• Batch scheduler optimization
• Large virtual machines deployments
F. Desprez - SILECS/SLICES - Frederic.Desprez@inria.frJournées GDR RSD Nantes
hOps://www.grid5000.fr/
11. SILECS/ FIT
F. Desprez - SILECS/SLICES - Frederic.Desprez@inria.fr
FIT-IoT-LAB
• 2700 wireless sensor nodes spread across six different sites in France
• Nodes are either fixed or mobile and can be allocated in various topologies throughout all sites
Sophia
Lyon
FIT-CorteXlab: Cognitive Radio Testbed
40 Software Defined Radio Nodes
(SOCRATE)
FIT-R2Lab: WiFi mesh testbed
(DIANA)
Journées GDR RSD Nantes
hOps://fit-equipex.fr/
hOps://www.iot-lab.info/hardware/
Providing Internet players access
to a variety of fixed and mobile
technologies and services, thus
accelerating the design of
advanced technologies for the
Future Internet
12. Data Center PorLolio
Targets
● Performance, resilience, energy-efficiency, security in the context of data-center design, Big Data
processing, Exascale computing, AI, etc.
Hardware
● Servers: x86, ARM64, POWER, accelerators (GPU, FPGA), …
● AI dedicated servers
● Edge computing micro datacenters
● Networking: Ethernet (10G, 40G), HPC networks (InfiniBand, Omni-Path), …
● Storage: HDD, SSD, NVMe, both in storage arrays and clusters of servers, …
Experimental support
● Bare-metal reconfiguration
● Large clusters
● Integrated monitoring (performance, energy, temperature, network traffic)
F. Desprez - SILECS/SLICES - Frederic.Desprez@inria.frJournées GDR RSD Nantes
13. Wireless PorLolio
Targets
• Performance, security, safety and privacy-preservaTon in complex sensing environment,
• Performance understanding and enhancement in wireless networking,
• Target applicaTons: smart ciTes/manufacturing, building automaTon, standard and interoperability,
security, energy harvesTng, health care
Hardware
• Sojware Defined Radio (SDR), NB-IoT, 5G, BLE, Thread
• Wireless Sensor Network (IEEE 802.15.4),
• LoRa/LoRaWAN, …
Experimental support
• Bare-metal reconfiguraTon
• Large-scale deployment (both in terms of densiTes and network diameter)
• Different topologies with indoor/outdoor locaTons
• Mobility-enabled with customized trajectories
• Anechoic chamber
• Integrated monitoring (power consumpTon, radio signal, network traffic)
F. Desprez - SILECS/SLICES - Frederic.Desprez@inria.frJournées GDR RSD Nantes
14. Outdoor IOT testbed
• IoT is not limited to smart objects or indoor wireless sensors (smart
building, industry 4.0, ….)
• Smart cities need outdoor IoT solutions
• Outdoor smart metering
• Outdoor metering at the scale of a neighborhood (air, noise smart sensing, ….)
• Citizens and local authorities are more and more interested by outdoor metering
• Controlled outdoor testbed
• (Reproducible) polymorphic IoT: support of multiple IoT technologies (long, middle
and short range IoT wireless solutions) at the same time on a large scale testbed
• Agreement and support of local authorities
• Deployment in Strasbourg city (500000 citizens, 384 km2) and Lyon campus
F. Desprez - SILECS/SLICES - Frederic.Desprez@inria.frJournées GDR RSD Nantes
15. An experiment outline
• Discovering resources from their descripTon
• Reconfiguring the testbed to meet experimental needs
• Monitoring experiments, extracTng and analyzing data
• Controlling experiments: API
Journées GDR RSD Nantes F. Desprez - SILECS/SLICES - Frederic.Desprez@inria.fr
16. Plans for SILECS/SLICES: Testbed Services
● Provide a unified framework that (really) meets all needs
○ Make it easier for experimenters to move for one testbed to another
○ Make it easy to create simultaneous reservations on several testbeds (for cross-
testbeds experiments)
○ Make it easy to extend SILECS/SLICES with additional kinds of resources
● Factor testbed services
○ Services that can exist at a higher level, e.g. open data service, for storage and
preservation of experiments data
○ In collaboration with Open Data repositories such as OpenAIRE/Zenodo
○ Services that are required to operate such infrastructures, but add no scientific
value
○ Users management, usage tracking
F. Desprez - SILECS/SLICES - Frederic.Desprez@inria.frJournées GDR RSD Nantes
17. Services & SoQware Stack
F. Desprez - SILECS/SLICES - Frederic.Desprez@inria.fr
Built from already funcTonal soluTons
Journées GDR RSD Nantes
18. Some recent experiments examples
• QoS differentiation in data collection for smart Grids, J. Nassar, M. Berthomé, J. Dubrulle, N. Gouvy, N.
Mitton, B. Quoitin
• Damaris: Scalable I/O and In-situ Big Data Processing, G. Antoniu, H. Salimi, M. Dorier
• Frequency Selection Approach for Energy Aware Cloud Database, C. Guo, J.-M. Pierson
• Distributed Storage for a Fog/Edge infrastructure based on a P2P and a Scale-Out NAS, B. Confais, B.
Parrein, A. Lebre
• FogIoT Orchestrator: an Orchestration System for IoT Applications in Fog Environment, B. Donassolo, I.
Fajjari, A. Legrand, P. Mertikopoulos
F. Desprez - SILECS/SLICES - Frederic.Desprez@inria.frJournées GDR RSD Nantes
19. QoS differen2a2on in data collec2on for smart Grids
• Data collecTon with different QoS requirements for Smart Grid applicaTons
• TradiTonal approach
• Use of standard RPL protocol which offers overall good performance but no QoS
differenTaTon based on applicaTon
• SoluTon
• Use a dynamic objecTve funcTon
• FIT IoT LAB as a validaTon testbed
• Access to 67 sensor nodes with IoT features remotely
• Customizable environment and tools (data size and rate, consumpTon measure, clock, etc)
• Repeat the experiments and compare to alternate approaches with the same environment
• The results show that based on the service requested, data from different
applicaTons follow different paths, each meeTng expected requirements
• FIT IoT LAB helped validate the approach to go further with standardizaTon
F. Desprez - SILECS/SLICES - Frederic.Desprez@inria.fr
Mul/ple Instances QoS Rou/ng In RPL: Applica/on To Smart Grids – J. Nassar, M. Berthomé, J. Dubrulle, N. Gouvy, N. MiOon, B. QuoiTn –
MDPI Sensors, July 2018
Journées GDR RSD Nantes
20. Damaris
• Scalable, asynchronous data storage for large-scale simulaTons using the HDF5 format (HDF5 blog at
hOps://goo.gl/7A4cZh)
• TradiTonal approach
• All simulaTon processes (10K+) write on disk at the
same Tme synchronously
• Problems: 1) I/O jiOer, 2) long I/O phase, 3) Blocked
simulaTon during data wriTng
• SoluTon
• Aggregate data in dedicated cores using shared memory and write
asynchronously
• Grid’5000 used as a testbed
– Access to many (1024) homogeneous cores
– Customizable environment and tools
– Repeat the experiments later with the same environment saved as an image
• The results show that Damaris can provide a jiOer-free and wait-free data storage mechanism
• G5K helped prepare Damaris for deployment on top supercomputers (Titan, Pangea (Total), Jaguar,
Kraken, etc.)
F. Desprez - SILECS/SLICES - Frederic.Desprez@inria.fr
…
hOps://project.inria.fr/damaris/Journées GDR RSD Nantes
21. Frequency Selec2on Approach for Energy Aware Cloud Database
• Objec/ve: Study the energy efficiency of cloud database systems and propose a
frequency selecTon approach and corresponding algorithms to cope with resource
proposing problem
F. Desprez - SILECS/SLICES - Frederic.Desprez@inria.fr
Frequency Selec/on Approach for Energy Aware Cloud Database, C. Guo, J.-M. Pierson. In Proc. SBAC-PAD, 2018.
Relationship between Request Amount and Throughput
• Contribu/on: Propose frequency selecTon model
and algorithms.
• Propose a GeneTc Based Algorithm and a Monte Carlo
Tree Based Algorithm to produce the frequencies
according to workload predicTons
• Propose a model simplificaTon method to improve the
performance of the algorithms
• Grid5000 usage
• A cloud database system, Cassandra, was deployed within a Grid’5000 cluster using 10 nodes of Nancy side
to study the relaTonship between system throughput and energy efficiency of the system
• By another benchmark experiment, the migraTon cost parameters of the model were obtained
Journées GDR RSD Nantes
22. Distributed Storage for a Fog/Edge infrastructure based
on a P2P and a Scale-Out NAS
• ObjecTve
• Design of a storage infrastructure taking locality into account
• ProperTes a distributed storage system should have: data locality, network
containment, mobility support, disconnected mode, scalability
• ContribuTons
• Improving locality when accessing an object stored locally coupling IPFS and a Scale-
Out NAS
• Improving locality when accessing an object stored on a remote site using a tree
inspired by the DNS
• Experiments
• Deployment of a Fog Site on the Grid’5000 testbed and the clients on the IoTLab
plaUorm
• Coupling a Scale-Out NAS to IPFS limits the inter-sites network traffic and improves
locality of local accesses
• Replacing the DHT by a tree mapped on the physical topology improves locality to
find the locaTon of objects
• Experiments using IoTlab and Grid’5000 are (currently) not easy to perform
F. Desprez - SILECS/SLICES - Frederic.Desprez@inria.frJournées GDR RSD Nantes
An Object Store Service for a Fog/Edge Compu/ng Infrastructure based on IPFS and Scale-out NAS, B. Confais, A. Lebre, and B. Parrein
(May 2017). In: 1st IEEE InternaTonal Conference on Fog and Edge CompuTng - ICFEC’2017.
23. FogIoT Orchestrator: an Orchestra2on System for IoT
Applica2ons in Fog Environment
F. Desprez - SILECS/SLICES - Frederic.Desprez@inria.fr
• Objective
• Design a Optimized Fog Service Provisioning strategy (O-FSP) and
validate it on a real infrastructure
• Contributions
• Design and implementation of FITOR, an orchestration framework for
the automation of the deployment, the scalability management, and
migration of micro-service based IoT applications
• Design of a provisioning solution for IoT applications that optimizes the
placement and the composition of IoT components, while dealing with
the heterogeneity of the underlying Fog infrastructure
• Experiments
• Fog layer is composed of 20 servers from Grid’5000 which are part of the
genepi cluster, Mist layer is composed of 50 A8 nodes
• Use of a software stack made of open-source components (Calvin,
Prometheus, Cadvisor, Blackbox exporter, Netdata)
• Experiments show that the O-FSP strategy makes the provisioning more
effective and outperforms classical strategies in terms of: i) acceptance
rate, ii) provisioning cost, and iii) resource usage
Journées GDR RSD Nantes
FogIoT Orchestrator: an Orchestra/on System for IoT Applica/ons in Fog Environment, B. Donassolo, I.
Fajjari, A. Legrand, P. MerTkopoulos.. 1st Grid’5000-FIT school, Apr 2018, Sophia AnTpolis, France. 2018.
24. Journées GDR RSD Nantes F. Desprez - SILECS/SLICES - Frederic.Desprez@inria.fr
Need of exchanges with the community
• TILECS Workshop
• July 3-4, Grenoble, LIG/IMAG
• 101 attendees (academics and some from the industry)
• https://www.silecs.net/tilecs-2019/
• Request for input
• 1/2 page(s) document describing which kind of experiment
you would like to perform in the next 4 years and what will
be you dream infrastructure (hardware/software/services)
• Dead-line Feb, 24 !
• Docs (latex, Word): https://frama.link/QzneeS7A
• Envoi à silecs-pia3-contrib@inria.fr
• Next workshop in March/early April
• More information coming soon
25. Conclusions
• SLICES: new infrastructure for experimental computer science and future services in Europe
• SILECS: new infrastructure in France based on two existing instruments (FIT and Grid’5000)
• Big challenges !
• Design a software stack that will allow experiments mixing both kinds of resources at the European level while keeping
reproducibility level high
• Keep the existing infrastructures up while designing and deploying the new one
• Keep the aim of previous platforms (their core scientific issues addressed)
– Scalability issues, energy management, …
– IoT, wireless networks, future Internet
– HPC, big data, clouds, virtualization, deep learning, ...
• Address new challenges
– IoT and Clouds
– New generation Cloud platforms and software stacks (Edge, FOG)
– Data streaming applications
– Locality aware resource management
– Big data management and analysis from sensors to the (distributed) cloud
– Mobility
– Next generation wireless
– …
• Next steps
– PIA-3 (Equipements structurants pour la recherche/EQUIPEX+) and ESFRI in May (5 and 18)
F. Desprez - SILECS/SLICES - Frederic.Desprez@inria.frJournées GDR RSD Nantes
26. Thanks, any ques2ons ?
F. Desprez - SILECS/SLICES - Frederic.Desprez@inria.frJournées GDR RSD Nantes
hOp://www.silecs.net/
hOps://www.grid5000.fr/
hOps://fit-equipex.fr/