The statistical assessment of the empirical comparison of algorithms is an essential step in heuristic optimization. Classically, researchers have relied on the use of statistical tests. However, recently, concerns about their use have arisen and, in many fields, other (Bayesian) alternatives are being considered. For a proper analysis, different aspects should be considered. In this talk, we focus on the question: what is the probability of a given algorithm being the best (among the compared)? To tackle this question, we propose a Bayesian analysis based on the Plackett-Luce model over rankings that allows several algorithms to be considered at the same time. In order to illustrate the proposed Bayesian alternative, explicitly, we examine performance data of 11 evolutionary algorithms (EAs) over a set of 23 discrete optimization problems in several dimensions. Using this data, and following a brief introduction to the relevant Bayesian inference practice, we demonstrate how to draw the algorithms’ probabilities of winning.
Interpretable machine learning in endocrinology, M. Biehl, APPIS 2024University of Groningen
An introduction to interpretable machine learning in endocrinology.
In particular, the application of Generalized Matrix Relevance LVQ to the classification of andrenocortical tumors and the differential diagnosis of primary aldosteronism is given.
Forecasting using data workshop slides for the Deliver conference in Winnipeg October 2016. This session introduces practical exercises for probabilistic forecasting. http://www.prdcdeliver.com
A meta-analysis of computational biology benchmarks reveals predictors of pro...Paul Gardner
This meta-analysis of computational biology benchmarks found no significant predictors of algorithm accuracy. The author analyzed 84 benchmarks comparing the accuracy and speed of 203 bioinformatics methods. Metrics like journal impact factor, author citation counts, and method age showed weak or no correlation with accuracy rank. While fast methods underwent more development iterations, speed was also not predictive of accuracy. The author concludes that factors like author/journal prestige do not guarantee software quality, and heuristic approaches can perform as well as mathematically rigorous ones. Overall, the study found no clear predictors of a method's programming accuracy based on existing benchmarks.
We propose a microphysical theory of the triboelectric effect by which mechanical rubbing separates charges across the interface between two materials. Surface electrons are treated as an open system coupled to two baths, corresponding to the bulks. Extending Zel'dovich's theory of bosonic superradiance, we show that motion-induced population inversion can generate an electromotive force. We argue that this is consistent with the basic phenomenology of triboelectrification as an irreversible process, and we suggest how to carry out more price experimental tests. This work has been published as: R. Alicki and A. Jenkins, Phys. Rev. Lett. 125, 186101 (2020).
Slides from a project the 2018 Brains, Minds, & Machines summer school, on understanding adversarial examples in deep convolutional neural networks, using attention maps.
"El álgebra lineal es una herramienta fundamental en muchos campos de la ciencia y la tecnología. Es particularmente importante en la física, la ingeniería, la informática y la estadística. La capacidad de manipular eficientemente grandes cantidades de datos y matrices complejas es esencial en estas áreas para la resolución de problemas y la toma de decisiones.
A priori, puede dar la sensación de que estamos muy lejos del uso del álgebra lineal en nuestro día a día. Sin embargo, algunas técnicas como la descomposición en valores singulares y la regresión lineal para entrenar modelos y hacer predicciones precisas están detrás de la inteligencia artificial y el aprendizaje automático. ¿Te suena ChatGPT? Puede no parecerlo, pero el álgebra lineal también está detrás en algunos de sus procesos. Por este motivo, debemos seguir trabajando en este campo, ya que su importancia seguirá creciendo a medida que se generen y analicen grandes cantidades de datos en el mundo actual.
"
La pandemia de COVID-19 ha supuesto una proliferación de mapas y contramapas. Por ello, organizaciones de la sociedad civil y movimientos sociales han generado sus propias interpretaciones y representaciones de los datos sobre la crisis. Estos también han contribuido a visibilizar aspectos, sujetos y temas que han sido desatendidos o infrarrepresentados en las visualizaciones hegemónicas y dominantes. En este contexto, la presente ponencia se centra en el análisis de los imaginarios sociales relacionados con la elaboración de mapas durante la pandemia. Es decir, trata de indagar en la importancia de los mapas para el activismo digital, las potencialidades que se extraen de esta tecnología y los valores asociados a las visualizaciones creadas con ellos. El objetivo último es reflexionar sobre la vía emergente del activismo de datos, así como sobre la intersección entre los imaginarios sociales y la geografía digital.
Interpretable machine learning in endocrinology, M. Biehl, APPIS 2024University of Groningen
An introduction to interpretable machine learning in endocrinology.
In particular, the application of Generalized Matrix Relevance LVQ to the classification of andrenocortical tumors and the differential diagnosis of primary aldosteronism is given.
Forecasting using data workshop slides for the Deliver conference in Winnipeg October 2016. This session introduces practical exercises for probabilistic forecasting. http://www.prdcdeliver.com
A meta-analysis of computational biology benchmarks reveals predictors of pro...Paul Gardner
This meta-analysis of computational biology benchmarks found no significant predictors of algorithm accuracy. The author analyzed 84 benchmarks comparing the accuracy and speed of 203 bioinformatics methods. Metrics like journal impact factor, author citation counts, and method age showed weak or no correlation with accuracy rank. While fast methods underwent more development iterations, speed was also not predictive of accuracy. The author concludes that factors like author/journal prestige do not guarantee software quality, and heuristic approaches can perform as well as mathematically rigorous ones. Overall, the study found no clear predictors of a method's programming accuracy based on existing benchmarks.
We propose a microphysical theory of the triboelectric effect by which mechanical rubbing separates charges across the interface between two materials. Surface electrons are treated as an open system coupled to two baths, corresponding to the bulks. Extending Zel'dovich's theory of bosonic superradiance, we show that motion-induced population inversion can generate an electromotive force. We argue that this is consistent with the basic phenomenology of triboelectrification as an irreversible process, and we suggest how to carry out more price experimental tests. This work has been published as: R. Alicki and A. Jenkins, Phys. Rev. Lett. 125, 186101 (2020).
Slides from a project the 2018 Brains, Minds, & Machines summer school, on understanding adversarial examples in deep convolutional neural networks, using attention maps.
"El álgebra lineal es una herramienta fundamental en muchos campos de la ciencia y la tecnología. Es particularmente importante en la física, la ingeniería, la informática y la estadística. La capacidad de manipular eficientemente grandes cantidades de datos y matrices complejas es esencial en estas áreas para la resolución de problemas y la toma de decisiones.
A priori, puede dar la sensación de que estamos muy lejos del uso del álgebra lineal en nuestro día a día. Sin embargo, algunas técnicas como la descomposición en valores singulares y la regresión lineal para entrenar modelos y hacer predicciones precisas están detrás de la inteligencia artificial y el aprendizaje automático. ¿Te suena ChatGPT? Puede no parecerlo, pero el álgebra lineal también está detrás en algunos de sus procesos. Por este motivo, debemos seguir trabajando en este campo, ya que su importancia seguirá creciendo a medida que se generen y analicen grandes cantidades de datos en el mundo actual.
"
La pandemia de COVID-19 ha supuesto una proliferación de mapas y contramapas. Por ello, organizaciones de la sociedad civil y movimientos sociales han generado sus propias interpretaciones y representaciones de los datos sobre la crisis. Estos también han contribuido a visibilizar aspectos, sujetos y temas que han sido desatendidos o infrarrepresentados en las visualizaciones hegemónicas y dominantes. En este contexto, la presente ponencia se centra en el análisis de los imaginarios sociales relacionados con la elaboración de mapas durante la pandemia. Es decir, trata de indagar en la importancia de los mapas para el activismo digital, las potencialidades que se extraen de esta tecnología y los valores asociados a las visualizaciones creadas con ellos. El objetivo último es reflexionar sobre la vía emergente del activismo de datos, así como sobre la intersección entre los imaginarios sociales y la geografía digital.
Designing RISC-V-based Accelerators for next generation Computers (DRAC) is a 3-year project (2019-2022) funded by the ERDF Operational Program of Catalonia 2014-2020. DRAC will design, verify, implement and fabricate a high performance general purpose processor that will incorporate different accelerators based on the RISC-V technology, with specific applications in the field of post-quantum security, genomics and autonomous navigation. In this talk, we will provide an overview of the main achievements in the DRAC project, including the fabrication of Lagarto, the first RISC-V processor developed in Spain.
This talk will begin introducing the uElectronics section of ESA at ESTEC and the general activities the group is responsible for. Then, it will go through some of the R+D on-going activities that the group is involved with, hand in hand with universities and/or companies. One of the major ones is related to the European rad-hard FPGAs that have been partially founded by ESA for several years and that will be playing a major role in the sector in the upcoming years. It´s also worth talking about the RTL soft IPs that are currently under development and that will allow us to keep on providing the European ecosystem with some key capabilities. The latter will be an overview of RISC-V space hardened on-going activities that might be replacing the current SPARC based processors available for our missions.
El objetivo de esta charla es presentar las últimas novedades incorporadas en la arquitectura ARM y describir las tendencias en la microarquitectura de los procesadores con arquitectura ARM. ARM es una empresa relativamente pequeña en comparación con otros gigantes del sector tecnológico. Sin embargo, la amplia implantación de su arquitectura, siendo ampliamente dominante en algunos sectores, y sus microarquitecturas, hacen que la tecnología ARM ocupe un lugar central en el desarrollo tecnológico del mundo actual. La tecnología ARM está presente prácticamente en todo el espectro tecnológico, desde los dispositivos más sencillos hasta el HPC y Cloud computing, pasando por los smartphones, automoción electrónica de consumo, etc
"Formal verification has been used by computer scientists for decades to prevent
software bugs. However, with a few exceptions, it has not been used by researchers
working in most areas of mathematics (geometry, algebra, analysis, etc.). In this
talk, we will discuss how this has changed in the past few years, and the possible
implications to the future of mathematical research, teaching and communication.
We will focus on the theorem prover Lean and its mathematical library
mathlib, since this is currently the system most widely used by mathematicians.
Lean is a functional programming language and interactive theorem prover based
on dependent type theory, with proof irrelevance and non-cumulative universes.
The mathlib library, open-source and designed as a basis for research level
mathematics, is one of the largest collections of formalized mathematics. It allows
classical reasoning, uses large- and small-scale automation, and is characterized
by its decentralized nature with over 200 contributors, including both computer
scientists and mathematicians."
"Part of the research community thinks that it is still early to tackle the development of quantum software engineering techniques. The reason is that how the quantum computers of the future will look like is still unknown. However, there are some facts that we can affirm today: 1) quantum and classical computers will coexist, each dedicated to the tasks at which they are most efficient. 2) quantum computers will be part of the cloud infrastructure and will be accessible through the Internet. 3) complex software systems will be made up of smaller pieces that will collaborate with each other. 4) some of those pieces will be quantum, therefore the systems of the future will be hybrid. 5) the coexistence and interaction between the components of said hybrid systems will be supported by service composition: quantum services.
This talk analyzes the challenges that the integration of quantum services poses to Service Oriented Computing."
In this talk, after a brief overview of AI concepts in particular Machine Learning (ML) techniques, some of the well-known computer design concepts for high performance and power efficiency are presented. Subsequently, those techniques that have had a promising impact for computing ML algorithms are discussed. Deep learning has emerged as a game changer for many applications in various fields of engineering and medical sciences. Although the primary computation function is matrix vector multiplication, many competing efficient implementations of this primary function have been proposed and put into practice. This talk will review and compare some of those techniques that are used for ML computer design.
Tras una breve introducción a la informática médica y unas pinceladas sobre conceptos prácticos de Inteligencia Artificial (posible definición consensuada, strong VS weak AI y técnicas y métodos comúnmente empleados), el bloque central de la charla muestra ejemplos prácticos (en forma de casos de éxito) de distintos desarrollos llevados a cabo por el grupo de Sistemas Informáticos de Nueva Generación (SING: http//sing-group.org/) en los ámbitos de (i) Informática clínica (InNoCBR, PolyDeep), (ii) Informática para investigación clínica (PathJam, WhichGenes), (iii) bioinformática traslacional (Genómica: ALTER, Proteómica: DPD, BI, BS, Mlibrary, Mass-Up, e integración de datos ÓMICOS: PunDrugs) y (iv) Informática en salud pública (CURMIS4th). Finalmente, se comenta brevemente la importancia que se espera tenga en un futuro inmediato la IA interpretable (XAI, Explainable Artificial Intelligence) y la participación humana (HITL. Human-In-The-Loop). La charla termina con una breve reflexión sobre las lecciones aprendidas por el ponente después de más de 16 años de desarrollo de sistemas inteligentes en el ámbito de la informática médica.
Many emerging applications require methods tailored towards high-speed data acquisition and filtering of streaming data followed by offline event reconstruction and analysis. In this case, the main objective is to relieve the immense pressure on the storage and communication resources within the experimental infrastructure. In other applications, ultra low latency real time analysis is required for autonomous experimental systems and anomaly detection in acquired scientific data in the absence of any prior data model for unknown events. At these data rates, traditional computing approaches cannot carry out even cursory analyses in a time frame necessary to guide experimentation. In this talk, Prof. Ogrenci will present some examples of AI hardware architectures. She will discuss the concept of co-design, which makes the unique needs of an application domain transparent to the hardware design process and present examples from three applications: (1) An in-pixel AI chip built using the HLS methodology; (2) A radiation hardened ASIC chip for quantum systems; (3) An FPGA-based edge computing controller for real-time control of a High Energy Physics experiment.
En esta conferencia se presentará una revisión del concepto de autonomía para robots móviles de campo y la identificación de desafíos para lograr un verdadero sistema autónomo, además de sugerir posibles direcciones de investigación. Los sistemas robóticos inteligentes, por lo general, obtienen conocimiento de sus funciones y del entorno de trabajo en etapa de diseño y desarrollo. Este enfoque no siempre es eficiente, especialmente en entornos semiestructurados y complejos como puede ser el campo de cultivo. Un sistema robótico verdaderamente autónomo debería desarrollar habilidades que le permitan tener éxito en tales entornos sin la necesidad de tener a-priori un conocimiento ontológico del área de trabajo y la definición de un conjunto de tareas o comportamientos predefinidos. Por lo que en esta conferencia se presentarán posibles estrategias basadas en Inteligencia Artificial que permitan perfeccionar las capacidades de navegación de robots móviles y que sean capaces de ofrecer un nivel de autonomía lo suficientemente elevado para poder ejecutar todas las tareas dentro de una misión casa-a-casa (home-to-home).
Quantum computing has become a noteworthy topic in academia and industry. The multinational companies in the world have been obtaining impressive advances in all areas of quantum technology during the last two decades. These companies try to construct real quantum computers in order to exploit their theoretical preferences over today’s classical computers in practical applications. However, they are challenging to build a full-scale quantum computer because of their increased susceptibility to errors due to decoherence and other quantum noise. Therefore, quantum error correction (QEC) and fault-tolerance protocol will be essential for running quantum algorithms on large-scale quantum computers.
The overall effect of noise is modeled in terms of a set of Pauli operators and the identity acting on the physical qubits (bit flip, phase flip and a combination of bit and phase flips). In addition to Pauli errors, there is another error named leakage errors that occur when a qubit leaves the defined computational subspace. As the location of leakage errors is unknown, these can damage even more the quantum computations. Thus, this talk will briefly provide quantum error models.
Los chatbots son un elemento clave en la transformación digital de nuestra sociedad. Están por todas partes: eCommerce, salud digital, asistencia a clientes, turismo,... Pero si habéis usado alguno, probablemente os habrá decepcionado. Lo confieso, la mayoría de los chatbots que existen son muy malos. Y es que no es nada fácil hacer un chatbot que sea realmente útil e inteligente. Un chatbot combina toda la complejidad de la ingeniería de software con la del procesamiento de lenguaje natural. Pensad que muchos chatbots hay que desplegarlos en varios canales (web, telegram, slack,...) y a menudo tienen que utilizar APIs y servicios externos, acceder a bases de datos internas o integrar modelos de lenguaje preentrenados (por ej. detectores de toxicidad), etc. Y el problema no es sólo crear el bot, si no también probarlo y evolucionarlo. En esta charla veremos los mayores desafíos a los que hay que enfrentarse cuando nos encargan un proyecto de desarrollo que incluye un chatbot y qué técnicas y estrategias podemos ir aplicando en función de las necesidades del proyecto, para conseguir, esta vez sí un chatbot que sepa de lo que habla.
Many HPC applications are massively parallel and can benefit from the spatial parallelism offered by reconfigurable logic. While modern memory technologies can offer high bandwidth, designers must craft advanced communication and memory architectures for efficient data movement and on-chip storage. Addressing these challenges requires to combine compiler optimizations, high-level synthesis, and hardware design.
In this talk, I will present challenges, solutions, and trends for generating massively parallel accelerators on FPGA for high-performance computing. These architectures can provide performance comparable to software implementations on high-end processors, and much higher energy efficiency thanks to logic customization.
The main challenge of concurrent software verification has always been in achieving modularity, i.e., the ability to divide and conquer the correctness proofs with the goal of scaling the verification effort. Types are a formal method well-known for its ability to modularize programs, and in the case of dependent types, the ability to modularize and scale complex mathematical proofs.
In this talk I will present our recent work towards reconciling dependent types with shared memory concurrency, with the goal of achieving modular proofs for the latter. Applying the type-theoretic paradigm to concurrency has lead us to view separation logic as a type theory of state, and has motivated novel abstractions for expressing concurrency proofs based on the algebraic structure of a resource and on structure-preserving functions (i.e., morphisms) between resources.
Microarchitectural attacks, such as Spectre and Meltdown, are a class of
security threats that affect almost all modern processors. These attacks exploit the side-effects resulting from processor optimizations to leak sensitive information and compromise a system’s security.
Over the years, a large number of hardware and software mechanisms for
preventing microarchitectural leaks have been proposed. Intuitively, more
defensive mechanisms are less efficient, while more permissive mechanisms may offer more performance but require more defensive programming. Unfortunately, there are no
hardware-software contracts that would turn this intuition into a basis for
principled co-design.
In this talk, we present a framework for specifying hardware/software security
contracts, an abstraction that captures a processor’s security guarantees in a
simple, mechanism-independent manner by specifying which program executions a
microarchitectural attacker can distinguish.
La aparición de vulnerabilidades por la falta de controles de seguridad es una de las causas por las que se demandan nuevos marcos de trabajo que produzcan software seguro de forma predeterminada. En la conferencia se abordará cómo transformar el proceso de desarrollo de software dando la importancia que merece la seguridad desde el inicio del ciclo de vida. Para ello se propone un nuevo modelo de desarrollo – modelo Viewnext-UEx – que incorpora prácticas de seguridad de forma preventiva y sistemática en todas las fases del proceso de ciclo de vida del software. El propósito de este nuevo modelo es anticipar la detección de vulnerabilidades aplicando la seguridad desde las fases más tempranas, a la vez que se optimizan los procesos de construcción del software. Se exponen los resultados de un escenario preventivo, tras la aplicación del modelo Viewnext-UEx, frente al escenario reactivo tradicional de aplicar la seguridad a partir de la fase de testing.
This document discusses trusting artificial intelligence systems. It begins with an overview of trust in social and computing contexts. It then discusses artificial intelligence, including machine learning, deep learning, and natural language processing. It details how AI systems can be attacked, including adversarial inputs, data poisoning, and model stealing. It raises important discussions around using AI in contexts like cybersecurity, medicine, transportation, and sentiment analysis, and the challenges of ensuring systems can be trusted.
El uso de energías renovables es clave para cumplir los objetivos de desarrollo sostenible de la Agenda 2030. Entre estas energías, la eólica es la segunda más utilizada debido a su alta eficiencia. Algunos estudios sugieren que la energía eólica será la principal fuente de generación en 2050. Por ello es conveniente seguir investigando en la aplicación de técnicas de control avanzadas en estos sistemas.
Entre estas técnicas avanzadas cabe destacar las redes neuronales y el aprendizaje por refuerzo combinadas con estrategias clásicas de control. Estas técnicas ya se han empleado con éxito en el modelado y el control de sistemas complejos.
Esta conferencia presentará la aplicación de redes neuronales y aprendizaje por refuerzo al control de aerogeneradores, centrándolo especialmente en el control de pitch. Se detallarán diferentes configuraciones con redes neuronales y otras técnicas aplicadas al control de pitch. Finalmente se propondrán algunas técnicas híbridas que combinen lógica difusa, tablas de búsqueda y redes neuronales, mostrando resultados que han permitido probar su utilidad para mejorar la eficiencia de las turbinas eólicas.
As the world's energy demand rises, so does the amount of renewable energy, particularly wind energy, in the supply. The life cycle of wind farms starting from manufacturing the components to decommission stage involve significant involvement of cost and the application of AI and data analytics are on reducing these costs are limited. With this conference talk, the audience expected to know some of the interesting applications of AI and data analytics on offshore wind. And, also highlight the future challenges and opportunities. This conference could be useful for students, academics and researcher who want to make next career in offshore wind but yet know where to start.
This document discusses the evolution of edge AI systems and architectures for the Internet of Things (IoT) era. It describes how IoT has transitioned from simple wireless sensor networks to complex systems that converge digitized enterprise data with edge AI sensors and deep learning analytics. Edge AI moves intelligence closer to IoT devices by enabling real-time data processing and filtering at the network edge. This reduces data transmission costs and latency. The document outlines several examples of edge AI applications in healthcare, smart homes, and industry that analyze sensor data in real-time to provide personalized and energy efficient services. It also discusses how new edge AI hardware platforms and open-source systems are enabling more customized and affordable IoT solutions.
Embedded real-time software construction has usually posed interesting challenges due to the complexity of the tasks these systems have to execute. Most methods for developing these systems are either hard to scale up for large systems, or require a difficult testing effort with no guarantee for bug-free software products. Construction of system models and their analysis through simulation reduces both end costs and risks, while enhancing system capabilities and improving the quality of the final products. This is a useful approach, moreover considering that testing under actual operating conditions may be impractical and in some cases impossible. In this talk, we will present a Modeling and Simulation-based framework to develop embedded systems based on the DEVS (Discrete Event systems Specification) formalism. This approach combines the advantages of a simulation-based approach with the rigor of a formal methodology. We will discuss how to use this framework to incrementally develop embedded applications, and to integrate simulation models with hardware components seamlessly.
Open Channel Flow: fluid flow with a free surfaceIndrajeet sahu
Open Channel Flow: This topic focuses on fluid flow with a free surface, such as in rivers, canals, and drainage ditches. Key concepts include the classification of flow types (steady vs. unsteady, uniform vs. non-uniform), hydraulic radius, flow resistance, Manning's equation, critical flow conditions, and energy and momentum principles. It also covers flow measurement techniques, gradually varied flow analysis, and the design of open channels. Understanding these principles is vital for effective water resource management and engineering applications.
Designing RISC-V-based Accelerators for next generation Computers (DRAC) is a 3-year project (2019-2022) funded by the ERDF Operational Program of Catalonia 2014-2020. DRAC will design, verify, implement and fabricate a high performance general purpose processor that will incorporate different accelerators based on the RISC-V technology, with specific applications in the field of post-quantum security, genomics and autonomous navigation. In this talk, we will provide an overview of the main achievements in the DRAC project, including the fabrication of Lagarto, the first RISC-V processor developed in Spain.
This talk will begin introducing the uElectronics section of ESA at ESTEC and the general activities the group is responsible for. Then, it will go through some of the R+D on-going activities that the group is involved with, hand in hand with universities and/or companies. One of the major ones is related to the European rad-hard FPGAs that have been partially founded by ESA for several years and that will be playing a major role in the sector in the upcoming years. It´s also worth talking about the RTL soft IPs that are currently under development and that will allow us to keep on providing the European ecosystem with some key capabilities. The latter will be an overview of RISC-V space hardened on-going activities that might be replacing the current SPARC based processors available for our missions.
El objetivo de esta charla es presentar las últimas novedades incorporadas en la arquitectura ARM y describir las tendencias en la microarquitectura de los procesadores con arquitectura ARM. ARM es una empresa relativamente pequeña en comparación con otros gigantes del sector tecnológico. Sin embargo, la amplia implantación de su arquitectura, siendo ampliamente dominante en algunos sectores, y sus microarquitecturas, hacen que la tecnología ARM ocupe un lugar central en el desarrollo tecnológico del mundo actual. La tecnología ARM está presente prácticamente en todo el espectro tecnológico, desde los dispositivos más sencillos hasta el HPC y Cloud computing, pasando por los smartphones, automoción electrónica de consumo, etc
"Formal verification has been used by computer scientists for decades to prevent
software bugs. However, with a few exceptions, it has not been used by researchers
working in most areas of mathematics (geometry, algebra, analysis, etc.). In this
talk, we will discuss how this has changed in the past few years, and the possible
implications to the future of mathematical research, teaching and communication.
We will focus on the theorem prover Lean and its mathematical library
mathlib, since this is currently the system most widely used by mathematicians.
Lean is a functional programming language and interactive theorem prover based
on dependent type theory, with proof irrelevance and non-cumulative universes.
The mathlib library, open-source and designed as a basis for research level
mathematics, is one of the largest collections of formalized mathematics. It allows
classical reasoning, uses large- and small-scale automation, and is characterized
by its decentralized nature with over 200 contributors, including both computer
scientists and mathematicians."
"Part of the research community thinks that it is still early to tackle the development of quantum software engineering techniques. The reason is that how the quantum computers of the future will look like is still unknown. However, there are some facts that we can affirm today: 1) quantum and classical computers will coexist, each dedicated to the tasks at which they are most efficient. 2) quantum computers will be part of the cloud infrastructure and will be accessible through the Internet. 3) complex software systems will be made up of smaller pieces that will collaborate with each other. 4) some of those pieces will be quantum, therefore the systems of the future will be hybrid. 5) the coexistence and interaction between the components of said hybrid systems will be supported by service composition: quantum services.
This talk analyzes the challenges that the integration of quantum services poses to Service Oriented Computing."
In this talk, after a brief overview of AI concepts in particular Machine Learning (ML) techniques, some of the well-known computer design concepts for high performance and power efficiency are presented. Subsequently, those techniques that have had a promising impact for computing ML algorithms are discussed. Deep learning has emerged as a game changer for many applications in various fields of engineering and medical sciences. Although the primary computation function is matrix vector multiplication, many competing efficient implementations of this primary function have been proposed and put into practice. This talk will review and compare some of those techniques that are used for ML computer design.
Tras una breve introducción a la informática médica y unas pinceladas sobre conceptos prácticos de Inteligencia Artificial (posible definición consensuada, strong VS weak AI y técnicas y métodos comúnmente empleados), el bloque central de la charla muestra ejemplos prácticos (en forma de casos de éxito) de distintos desarrollos llevados a cabo por el grupo de Sistemas Informáticos de Nueva Generación (SING: http//sing-group.org/) en los ámbitos de (i) Informática clínica (InNoCBR, PolyDeep), (ii) Informática para investigación clínica (PathJam, WhichGenes), (iii) bioinformática traslacional (Genómica: ALTER, Proteómica: DPD, BI, BS, Mlibrary, Mass-Up, e integración de datos ÓMICOS: PunDrugs) y (iv) Informática en salud pública (CURMIS4th). Finalmente, se comenta brevemente la importancia que se espera tenga en un futuro inmediato la IA interpretable (XAI, Explainable Artificial Intelligence) y la participación humana (HITL. Human-In-The-Loop). La charla termina con una breve reflexión sobre las lecciones aprendidas por el ponente después de más de 16 años de desarrollo de sistemas inteligentes en el ámbito de la informática médica.
Many emerging applications require methods tailored towards high-speed data acquisition and filtering of streaming data followed by offline event reconstruction and analysis. In this case, the main objective is to relieve the immense pressure on the storage and communication resources within the experimental infrastructure. In other applications, ultra low latency real time analysis is required for autonomous experimental systems and anomaly detection in acquired scientific data in the absence of any prior data model for unknown events. At these data rates, traditional computing approaches cannot carry out even cursory analyses in a time frame necessary to guide experimentation. In this talk, Prof. Ogrenci will present some examples of AI hardware architectures. She will discuss the concept of co-design, which makes the unique needs of an application domain transparent to the hardware design process and present examples from three applications: (1) An in-pixel AI chip built using the HLS methodology; (2) A radiation hardened ASIC chip for quantum systems; (3) An FPGA-based edge computing controller for real-time control of a High Energy Physics experiment.
En esta conferencia se presentará una revisión del concepto de autonomía para robots móviles de campo y la identificación de desafíos para lograr un verdadero sistema autónomo, además de sugerir posibles direcciones de investigación. Los sistemas robóticos inteligentes, por lo general, obtienen conocimiento de sus funciones y del entorno de trabajo en etapa de diseño y desarrollo. Este enfoque no siempre es eficiente, especialmente en entornos semiestructurados y complejos como puede ser el campo de cultivo. Un sistema robótico verdaderamente autónomo debería desarrollar habilidades que le permitan tener éxito en tales entornos sin la necesidad de tener a-priori un conocimiento ontológico del área de trabajo y la definición de un conjunto de tareas o comportamientos predefinidos. Por lo que en esta conferencia se presentarán posibles estrategias basadas en Inteligencia Artificial que permitan perfeccionar las capacidades de navegación de robots móviles y que sean capaces de ofrecer un nivel de autonomía lo suficientemente elevado para poder ejecutar todas las tareas dentro de una misión casa-a-casa (home-to-home).
Quantum computing has become a noteworthy topic in academia and industry. The multinational companies in the world have been obtaining impressive advances in all areas of quantum technology during the last two decades. These companies try to construct real quantum computers in order to exploit their theoretical preferences over today’s classical computers in practical applications. However, they are challenging to build a full-scale quantum computer because of their increased susceptibility to errors due to decoherence and other quantum noise. Therefore, quantum error correction (QEC) and fault-tolerance protocol will be essential for running quantum algorithms on large-scale quantum computers.
The overall effect of noise is modeled in terms of a set of Pauli operators and the identity acting on the physical qubits (bit flip, phase flip and a combination of bit and phase flips). In addition to Pauli errors, there is another error named leakage errors that occur when a qubit leaves the defined computational subspace. As the location of leakage errors is unknown, these can damage even more the quantum computations. Thus, this talk will briefly provide quantum error models.
Los chatbots son un elemento clave en la transformación digital de nuestra sociedad. Están por todas partes: eCommerce, salud digital, asistencia a clientes, turismo,... Pero si habéis usado alguno, probablemente os habrá decepcionado. Lo confieso, la mayoría de los chatbots que existen son muy malos. Y es que no es nada fácil hacer un chatbot que sea realmente útil e inteligente. Un chatbot combina toda la complejidad de la ingeniería de software con la del procesamiento de lenguaje natural. Pensad que muchos chatbots hay que desplegarlos en varios canales (web, telegram, slack,...) y a menudo tienen que utilizar APIs y servicios externos, acceder a bases de datos internas o integrar modelos de lenguaje preentrenados (por ej. detectores de toxicidad), etc. Y el problema no es sólo crear el bot, si no también probarlo y evolucionarlo. En esta charla veremos los mayores desafíos a los que hay que enfrentarse cuando nos encargan un proyecto de desarrollo que incluye un chatbot y qué técnicas y estrategias podemos ir aplicando en función de las necesidades del proyecto, para conseguir, esta vez sí un chatbot que sepa de lo que habla.
Many HPC applications are massively parallel and can benefit from the spatial parallelism offered by reconfigurable logic. While modern memory technologies can offer high bandwidth, designers must craft advanced communication and memory architectures for efficient data movement and on-chip storage. Addressing these challenges requires to combine compiler optimizations, high-level synthesis, and hardware design.
In this talk, I will present challenges, solutions, and trends for generating massively parallel accelerators on FPGA for high-performance computing. These architectures can provide performance comparable to software implementations on high-end processors, and much higher energy efficiency thanks to logic customization.
The main challenge of concurrent software verification has always been in achieving modularity, i.e., the ability to divide and conquer the correctness proofs with the goal of scaling the verification effort. Types are a formal method well-known for its ability to modularize programs, and in the case of dependent types, the ability to modularize and scale complex mathematical proofs.
In this talk I will present our recent work towards reconciling dependent types with shared memory concurrency, with the goal of achieving modular proofs for the latter. Applying the type-theoretic paradigm to concurrency has lead us to view separation logic as a type theory of state, and has motivated novel abstractions for expressing concurrency proofs based on the algebraic structure of a resource and on structure-preserving functions (i.e., morphisms) between resources.
Microarchitectural attacks, such as Spectre and Meltdown, are a class of
security threats that affect almost all modern processors. These attacks exploit the side-effects resulting from processor optimizations to leak sensitive information and compromise a system’s security.
Over the years, a large number of hardware and software mechanisms for
preventing microarchitectural leaks have been proposed. Intuitively, more
defensive mechanisms are less efficient, while more permissive mechanisms may offer more performance but require more defensive programming. Unfortunately, there are no
hardware-software contracts that would turn this intuition into a basis for
principled co-design.
In this talk, we present a framework for specifying hardware/software security
contracts, an abstraction that captures a processor’s security guarantees in a
simple, mechanism-independent manner by specifying which program executions a
microarchitectural attacker can distinguish.
La aparición de vulnerabilidades por la falta de controles de seguridad es una de las causas por las que se demandan nuevos marcos de trabajo que produzcan software seguro de forma predeterminada. En la conferencia se abordará cómo transformar el proceso de desarrollo de software dando la importancia que merece la seguridad desde el inicio del ciclo de vida. Para ello se propone un nuevo modelo de desarrollo – modelo Viewnext-UEx – que incorpora prácticas de seguridad de forma preventiva y sistemática en todas las fases del proceso de ciclo de vida del software. El propósito de este nuevo modelo es anticipar la detección de vulnerabilidades aplicando la seguridad desde las fases más tempranas, a la vez que se optimizan los procesos de construcción del software. Se exponen los resultados de un escenario preventivo, tras la aplicación del modelo Viewnext-UEx, frente al escenario reactivo tradicional de aplicar la seguridad a partir de la fase de testing.
This document discusses trusting artificial intelligence systems. It begins with an overview of trust in social and computing contexts. It then discusses artificial intelligence, including machine learning, deep learning, and natural language processing. It details how AI systems can be attacked, including adversarial inputs, data poisoning, and model stealing. It raises important discussions around using AI in contexts like cybersecurity, medicine, transportation, and sentiment analysis, and the challenges of ensuring systems can be trusted.
El uso de energías renovables es clave para cumplir los objetivos de desarrollo sostenible de la Agenda 2030. Entre estas energías, la eólica es la segunda más utilizada debido a su alta eficiencia. Algunos estudios sugieren que la energía eólica será la principal fuente de generación en 2050. Por ello es conveniente seguir investigando en la aplicación de técnicas de control avanzadas en estos sistemas.
Entre estas técnicas avanzadas cabe destacar las redes neuronales y el aprendizaje por refuerzo combinadas con estrategias clásicas de control. Estas técnicas ya se han empleado con éxito en el modelado y el control de sistemas complejos.
Esta conferencia presentará la aplicación de redes neuronales y aprendizaje por refuerzo al control de aerogeneradores, centrándolo especialmente en el control de pitch. Se detallarán diferentes configuraciones con redes neuronales y otras técnicas aplicadas al control de pitch. Finalmente se propondrán algunas técnicas híbridas que combinen lógica difusa, tablas de búsqueda y redes neuronales, mostrando resultados que han permitido probar su utilidad para mejorar la eficiencia de las turbinas eólicas.
As the world's energy demand rises, so does the amount of renewable energy, particularly wind energy, in the supply. The life cycle of wind farms starting from manufacturing the components to decommission stage involve significant involvement of cost and the application of AI and data analytics are on reducing these costs are limited. With this conference talk, the audience expected to know some of the interesting applications of AI and data analytics on offshore wind. And, also highlight the future challenges and opportunities. This conference could be useful for students, academics and researcher who want to make next career in offshore wind but yet know where to start.
This document discusses the evolution of edge AI systems and architectures for the Internet of Things (IoT) era. It describes how IoT has transitioned from simple wireless sensor networks to complex systems that converge digitized enterprise data with edge AI sensors and deep learning analytics. Edge AI moves intelligence closer to IoT devices by enabling real-time data processing and filtering at the network edge. This reduces data transmission costs and latency. The document outlines several examples of edge AI applications in healthcare, smart homes, and industry that analyze sensor data in real-time to provide personalized and energy efficient services. It also discusses how new edge AI hardware platforms and open-source systems are enabling more customized and affordable IoT solutions.
Embedded real-time software construction has usually posed interesting challenges due to the complexity of the tasks these systems have to execute. Most methods for developing these systems are either hard to scale up for large systems, or require a difficult testing effort with no guarantee for bug-free software products. Construction of system models and their analysis through simulation reduces both end costs and risks, while enhancing system capabilities and improving the quality of the final products. This is a useful approach, moreover considering that testing under actual operating conditions may be impractical and in some cases impossible. In this talk, we will present a Modeling and Simulation-based framework to develop embedded systems based on the DEVS (Discrete Event systems Specification) formalism. This approach combines the advantages of a simulation-based approach with the rigor of a formal methodology. We will discuss how to use this framework to incrementally develop embedded applications, and to integrate simulation models with hardware components seamlessly.
Open Channel Flow: fluid flow with a free surfaceIndrajeet sahu
Open Channel Flow: This topic focuses on fluid flow with a free surface, such as in rivers, canals, and drainage ditches. Key concepts include the classification of flow types (steady vs. unsteady, uniform vs. non-uniform), hydraulic radius, flow resistance, Manning's equation, critical flow conditions, and energy and momentum principles. It also covers flow measurement techniques, gradually varied flow analysis, and the design of open channels. Understanding these principles is vital for effective water resource management and engineering applications.
Applications of artificial Intelligence in Mechanical Engineering.pdfAtif Razi
Historically, mechanical engineering has relied heavily on human expertise and empirical methods to solve complex problems. With the introduction of computer-aided design (CAD) and finite element analysis (FEA), the field took its first steps towards digitization. These tools allowed engineers to simulate and analyze mechanical systems with greater accuracy and efficiency. However, the sheer volume of data generated by modern engineering systems and the increasing complexity of these systems have necessitated more advanced analytical tools, paving the way for AI.
AI offers the capability to process vast amounts of data, identify patterns, and make predictions with a level of speed and accuracy unattainable by traditional methods. This has profound implications for mechanical engineering, enabling more efficient design processes, predictive maintenance strategies, and optimized manufacturing operations. AI-driven tools can learn from historical data, adapt to new information, and continuously improve their performance, making them invaluable in tackling the multifaceted challenges of modern mechanical engineering.
Impartiality as per ISO /IEC 17025:2017 StandardMuhammadJazib15
This document provides basic guidelines for imparitallity requirement of ISO 17025. It defines in detial how it is met and wiudhwdih jdhsjdhwudjwkdbjwkdddddddddddkkkkkkkkkkkkkkkkkkkkkkkwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwioiiiiiiiiiiiii uwwwwwwwwwwwwwwwwhe wiqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq gbbbbbbbbbbbbb owdjjjjjjjjjjjjjjjjjjjj widhi owqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq uwdhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhhwqiiiiiiiiiiiiiiiiiiiiiiiiiiiiw0pooooojjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjjj whhhhhhhhhhh wheeeeeeee wihieiiiiii wihe
e qqqqqqqqqqeuwiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiqw dddddddddd cccccccccccccccv s w c r
cdf cb bicbsad ishd d qwkbdwiur e wetwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwww w
dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddfffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffw
uuuuhhhhhhhhhhhhhhhhhhhhhhhhe qiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiii iqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqq eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee qqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqqccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc ccccccccccccccccccccccccccccccccccc bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbu uuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuuum
m
m mmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmmm m i
g i dijsd sjdnsjd ndjajsdnnsa adjdnawddddddddddddd uw
This presentation is about Food Delivery Systems and how they are developed using the Software Development Life Cycle (SDLC) and other methods. It explains the steps involved in creating a food delivery app, from planning and designing to testing and launching. The slide also covers different tools and technologies used to make these systems work efficiently.
We have designed & manufacture the Lubi Valves LBF series type of Butterfly Valves for General Utility Water applications as well as for HVAC applications.
Null Bangalore | Pentesters Approach to AWS IAMDivyanshu
#Abstract:
- Learn more about the real-world methods for auditing AWS IAM (Identity and Access Management) as a pentester. So let us proceed with a brief discussion of IAM as well as some typical misconfigurations and their potential exploits in order to reinforce the understanding of IAM security best practices.
- Gain actionable insights into AWS IAM policies and roles, using hands on approach.
#Prerequisites:
- Basic understanding of AWS services and architecture
- Familiarity with cloud security concepts
- Experience using the AWS Management Console or AWS CLI.
- For hands on lab create account on [killercoda.com](https://killercoda.com/cloudsecurity-scenario/)
# Scenario Covered:
- Basics of IAM in AWS
- Implementing IAM Policies with Least Privilege to Manage S3 Bucket
- Objective: Create an S3 bucket with least privilege IAM policy and validate access.
- Steps:
- Create S3 bucket.
- Attach least privilege policy to IAM user.
- Validate access.
- Exploiting IAM PassRole Misconfiguration
-Allows a user to pass a specific IAM role to an AWS service (ec2), typically used for service access delegation. Then exploit PassRole Misconfiguration granting unauthorized access to sensitive resources.
- Objective: Demonstrate how a PassRole misconfiguration can grant unauthorized access.
- Steps:
- Allow user to pass IAM role to EC2.
- Exploit misconfiguration for unauthorized access.
- Access sensitive resources.
- Exploiting IAM AssumeRole Misconfiguration with Overly Permissive Role
- An overly permissive IAM role configuration can lead to privilege escalation by creating a role with administrative privileges and allow a user to assume this role.
- Objective: Show how overly permissive IAM roles can lead to privilege escalation.
- Steps:
- Create role with administrative privileges.
- Allow user to assume the role.
- Perform administrative actions.
- Differentiation between PassRole vs AssumeRole
Try at [killercoda.com](https://killercoda.com/cloudsecurity-scenario/)
Tools & Techniques for Commissioning and Maintaining PV Systems W-Animations ...Transcat
Join us for this solutions-based webinar on the tools and techniques for commissioning and maintaining PV Systems. In this session, we'll review the process of building and maintaining a solar array, starting with installation and commissioning, then reviewing operations and maintenance of the system. This course will review insulation resistance testing, I-V curve testing, earth-bond continuity, ground resistance testing, performance tests, visual inspections, ground and arc fault testing procedures, and power quality analysis.
Fluke Solar Application Specialist Will White is presenting on this engaging topic:
Will has worked in the renewable energy industry since 2005, first as an installer for a small east coast solar integrator before adding sales, design, and project management to his skillset. In 2022, Will joined Fluke as a solar application specialist, where he supports their renewable energy testing equipment like IV-curve tracers, electrical meters, and thermal imaging cameras. Experienced in wind power, solar thermal, energy storage, and all scales of PV, Will has primarily focused on residential and small commercial systems. He is passionate about implementing high-quality, code-compliant installation techniques.
Supermarket Management System Project Report.pdfKamal Acharya
Supermarket management is a stand-alone J2EE using Eclipse Juno program.
This project contains all the necessary required information about maintaining
the supermarket billing system.
The core idea of this project to minimize the paper work and centralize the
data. Here all the communication is taken in secure manner. That is, in this
application the information will be stored in client itself. For further security the
data base is stored in the back-end oracle and so no intruders can access it.
Height and depth gauge linear metrology.pdfq30122000
Height gauges may also be used to measure the height of an object by using the underside of the scriber as the datum. The datum may be permanently fixed or the height gauge may have provision to adjust the scale, this is done by sliding the scale vertically along the body of the height gauge by turning a fine feed screw at the top of the gauge; then with the scriber set to the same level as the base, the scale can be matched to it. This adjustment allows different scribers or probes to be used, as well as adjusting for any errors in a damaged or resharpened probe.
Bayesian Performance Analysis for Optimization Algorithm Comparison
1. Josu Ceberio
Bayesian Analysis for
Algorithm Performance Comparison
Is it possible to compare optimization
algorithms without hypothesis testing?
2. Is there a reproducibility crisis?
Fuente: Monya Baker (2016) Is there a
reproducibility crisis? Nature, 533, 452-454
3. Hypothesis
Idea for solving a set
of problems more
efficiently.
Questions
Is my algorithm
better than the state-
of-the-art?
On which problems is
my algorithm better?
Why is my algorithm
better (or worse)?
Experimentation
Compare the performance
of my algorithm with the-
state-of-the-art on some
benchmark of problems.
The analysis of the results
should take into account
the associated
uncertainty.
Conclusions
What conclusions do we
draw from the
experimentation?
How do we answer to the
formulated questions?
Is there a reproducibility crisis?
4. The Questions
How likely is my proposal to
be the best algorithm to solve
a problem?
How likely is my proposal to
be the best algorithm from
the compared ones?
5. The Point
STATISTICAL ANALYSIS OF
EXPERIMENTAL RESULTS
NULL HYPOTHESIS
STATISTICAL TESTING
WHAT NHST COMPUTES
p(t(x) > ⌧|H0)<latexit sha1_base64="QScPf75YqpsLM08xO+kyaRgOrOs=">AAAB+XicbVBNS8NAEN3Ur1q/oh69LBahvZREBT1JwUuPFWwrtCFstpt26WYTdifFEvtPvHhQxKv/xJv/xm2bg7Y+GHi8N8PMvCARXIPjfFuFtfWNza3idmlnd2//wD48aus4VZS1aCxi9RAQzQSXrAUcBHtIFCNRIFgnGN3O/M6YKc1jeQ+ThHkRGUgeckrASL5tJxWoPFZvekDSp4bvVH277NScOfAqcXNSRjmavv3V68c0jZgEKojWXddJwMuIAk4Fm5Z6qWYJoSMyYF1DJYmY9rL55VN8ZpQ+DmNlSgKeq78nMhJpPYkC0xkRGOplbyb+53VTCK+9jMskBSbpYlGYCgwxnsWA+1wxCmJiCKGKm1sxHRJFKJiwSiYEd/nlVdI+r7kXNefuslx38ziK6ASdogpy0RWqowZqohaiaIye0St6szLrxXq3PhatBSufOUZ/YH3+ANqXknE=</latexit>
Unknown Behaviour
Observed Sample
7. The controversy with NHST
We assume the null hypothesis, the average
performance of the compared methods is the same.
Then, the observed difference is computed from data
and the probability of observing such a difference (or
bigger) is estimated: the p-value.
The p-value refers to the probability of erroneously
assuming that there are differences when actually
there are not. It is used to measure the magnitude of
difference, as it decreases when the difference
increases.
WHAT NHST COMPUTES
p(t(x) > ⌧|H0)<latexit sha1_base64="QScPf75YqpsLM08xO+kyaRgOrOs=">AAAB+XicbVBNS8NAEN3Ur1q/oh69LBahvZREBT1JwUuPFWwrtCFstpt26WYTdifFEvtPvHhQxKv/xJv/xm2bg7Y+GHi8N8PMvCARXIPjfFuFtfWNza3idmlnd2//wD48aus4VZS1aCxi9RAQzQSXrAUcBHtIFCNRIFgnGN3O/M6YKc1jeQ+ThHkRGUgeckrASL5tJxWoPFZvekDSp4bvVH277NScOfAqcXNSRjmavv3V68c0jZgEKojWXddJwMuIAk4Fm5Z6qWYJoSMyYF1DJYmY9rL55VN8ZpQ+DmNlSgKeq78nMhJpPYkC0xkRGOplbyb+53VTCK+9jMskBSbpYlGYCgwxnsWA+1wxCmJiCKGKm1sxHRJFKJiwSiYEd/nlVdI+r7kXNefuslx38ziK6ASdogpy0RWqowZqohaiaIye0St6szLrxXq3PhatBSufOUZ/YH3+ANqXknE=</latexit>
1 p(t(x) > ⌧|H0) = p(t(x) < ⌧|H0)<latexit sha1_base64="ixOtl42DABu1QXwNHfHlqHttk6E=">AAACDXicbZC7SgNBFIZnvcZ4W7W0GYxCUhh2VdBCJWCTMoK5QLIss5PZZMjshZmzYoh5ARtfxcZCEVt7O9/GSbKIJv4w8POdczhzfi8WXIFlfRlz8wuLS8uZlezq2vrGprm1XVNRIimr0khEsuERxQQPWRU4CNaIJSOBJ1jd612N6vVbJhWPwhvox8wJSCfkPqcENHLNffswzkP+rnDZApLcl12rcIEn5PyHuGbOKlpj4VljpyaHUlVc87PVjmgSsBCoIEo1bSsGZ0AkcCrYMNtKFIsJ7ZEOa2obkoApZzC+ZogPNGljP5L6hYDH9PfEgARK9QNPdwYEumq6NoL/1ZoJ+GfOgIdxAiykk0V+IjBEeBQNbnPJKIi+NoRKrv+KaZdIQkEHmNUh2NMnz5raUdE+LlrXJ7mSncaRQbtoD+WRjU5RCZVRBVURRQ/oCb2gV+PReDbejPdJ65yRzuygPzI+vgFYSZkn</latexit>
WHAT WE WOULD LIKE TO KNOW
1 p(H0|x) = p(H1|x)<latexit sha1_base64="1JettnS1nfDHVeV06DeUX+AEQ8Y=">AAAB/HicbZDLSgMxFIYz9VbrbbRLN8Ei1IVlooJuhIKbLivYC7TDkEnTNjSTGZKMOIz1Vdy4UMStD+LOtzHTzkJbfwh8/OcczsnvR5wp7TjfVmFldW19o7hZ2tre2d2z9w/aKowloS0S8lB2fawoZ4K2NNOcdiNJceBz2vEnN1m9c0+lYqG400lE3QCPBBsygrWxPLuMTqNqw3MeH06uM0AGPLvi1JyZ4DKgHCogV9Ozv/qDkMQBFZpwrFQPOZF2Uyw1I5xOS/1Y0QiTCR7RnkGBA6rcdHb8FB4bZwCHoTRPaDhzf0+kOFAqCXzTGWA9Vou1zPyv1ov18MpNmYhiTQWZLxrGHOoQZknAAZOUaJ4YwEQycyskYywx0SavkgkBLX55GdpnNXRec24vKnWUx1EEh+AIVAECl6AOGqAJWoCABDyDV/BmPVkv1rv1MW8tWPlMGfyR9fkDE+OTDg==</latexit>
p(H0|x)<latexit sha1_base64="/MpXzWcP8EqakOTUlXIzz1ULR90=">AAAB73icbVDLSgNBEOz1GeMr6tHLYBDiJeyqoMeAlxwjmAckS5idzCZDZmfXmV4xxPyEFw+KePV3vPk3TpI9aGJBQ1HVTXdXkEhh0HW/nZXVtfWNzdxWfntnd2+/cHDYMHGqGa+zWMa6FVDDpVC8jgIlbyWa0yiQvBkMb6Z+84FrI2J1h6OE+xHtKxEKRtFKraRU7bpPj2fdQtEtuzOQZeJlpAgZat3CV6cXszTiCpmkxrQ9N0F/TDUKJvkk30kNTygb0j5vW6poxI0/nt07IadW6ZEw1rYUkpn6e2JMI2NGUWA7I4oDs+hNxf+8dorhtT8WKkmRKzZfFKaSYEymz5Oe0JyhHFlCmRb2VsIGVFOGNqK8DcFbfHmZNM7L3kXZvb0sVrwsjhwcwwmUwIMrqEAValAHBhKe4RXenHvnxXl3PuatK042cwR/4Hz+ABOrj0c=</latexit>
8. The Point
Unknown Behaviour
Observed Sample
Many alternatives to handle uncertainty
associated with empirical results:
6WDWLVWLFDO QDOVLV
+DQGERRN
$ &RPSUHKHQVL H +DQGERRN RI 6 D LV LFDO
&RQFHS V 7HFKQLT HV DQG 6RI DUH 7RROV
(GL LRQ
'U 0LFKDHO - GH 6PL K
9. WHAT NHST COMPUTES
p(t(x) > ⌧|H0)<latexit sha1_base64="QScPf75YqpsLM08xO+kyaRgOrOs=">AAAB+XicbVBNS8NAEN3Ur1q/oh69LBahvZREBT1JwUuPFWwrtCFstpt26WYTdifFEvtPvHhQxKv/xJv/xm2bg7Y+GHi8N8PMvCARXIPjfFuFtfWNza3idmlnd2//wD48aus4VZS1aCxi9RAQzQSXrAUcBHtIFCNRIFgnGN3O/M6YKc1jeQ+ThHkRGUgeckrASL5tJxWoPFZvekDSp4bvVH277NScOfAqcXNSRjmavv3V68c0jZgEKojWXddJwMuIAk4Fm5Z6qWYJoSMyYF1DJYmY9rL55VN8ZpQ+DmNlSgKeq78nMhJpPYkC0xkRGOplbyb+53VTCK+9jMskBSbpYlGYCgwxnsWA+1wxCmJiCKGKm1sxHRJFKJiwSiYEd/nlVdI+r7kXNefuslx38ziK6ASdogpy0RWqowZqohaiaIye0St6szLrxXq3PhatBSufOUZ/YH3+ANqXknE=</latexit>
BAYESIAN STATISTICAL
ANALYSIS
The Point
STATISTICAL ANALYSIS OF
EXPERIMENTAL RESULTS
NULL HYPOTHESIS
STATISTICAL TESTING
Unknown Behaviour
Observed Sample
10. The Bayesian Approach
The method focuses on estimating relevant
information about the underlying performance
parametric distribution represented by a set of
parameters θ.
This method asses the distribution of θ
conditioned on a sample s drawn from the
performance distribution.
Instead of having a single probability distribution
to model the underlying performance, Bayesian
statistics considers all possible distributions
and assigns a probability to each.
P(✓|s) / P(s|✓)P(✓)<latexit sha1_base64="1oaUrufzQhQHrQgFYQ+vqg7duQg=">AAACEXicbVDLSsNAFJ34rPUVdelmsAjppiRV0GXRjcsI9gFtKJPppB06eTBzI5S0v+DGX3HjQhG37tz5N07bCNp6YOBwzr3cOcdPBFdg21/Gyura+sZmYau4vbO7t28eHDZUnErK6jQWsWz5RDHBI1YHDoK1EslI6AvW9IfXU795z6TicXQHo4R5IelHPOCUgJa6puVaHRgwIGNV7iQyTiDGrqXGc7GMf+xy1yzZFXsGvEycnJRQDrdrfnZ6MU1DFgEVRKm2YyfgZUQCp4JNip1UsYTQIemztqYRCZnyslmiCT7VSg8HsdQvAjxTf29kJFRqFPp6MiQwUIveVPzPa6cQXHoZj5IUWETnh4JUYB17Wg/ucckoiJEmhEqu/4rpgEhCQZdY1CU4i5GXSaNacc4q1dvzUu0qr6OAjtEJspCDLlAN3SAX1RFFD+gJvaBX49F4Nt6M9/noipHvHKE/MD6+ASzGnJY=</latexit>
Posterior distribution
of the parameters
Likelihood
function
Prior distribution
of the parameters
HOW DO WE COMPARE MULTIPLE
ALGORITHMS?
11. Minimizing some instances of a problemMinimizing a given instance of a problem
Algorithm f1
GA 100
PSO 90
ILP 135
SA 105
GP 95
.
.
.
.
.
.
From Results to Rankings
Observed Sample
σ1
3
1
5
4
2
.
.
.
Algorithm f2
GA 130
PSO 80
ILP 135
SA 30
GP 300
.
.
.
.
.
.
σ2
3
2
4
1
5
.
.
.
σ3
3
5
2
4
1
.
.
.
σ4
4
5
3
1
2
.
.
.
σ5
4
3
2
5
1
.
.
.
Algorithm f3
GA 37
PSO 352
ILP 19
SA 100
GP 10
.
.
.
.
.
.
Algorithm f4
GA 566
PSO 756
ILP 101
SA 56
GP 57
.
.
.
.
.
.
Algorithm f5
GA 256
PSO 125
ILP 89
SA 369
GP 36
.
.
.
.
.
.
rankings, permutations
12. ● Each algorithm in the comparison has a weight associated.
● The weights sum up 1.
● The weight associated to an algorithm represents its probability to appear at first rank.
Plackett-luce Model
P( ) =
nY
i=1
w i
Pn
j=i w j
!
<latexit sha1_base64="l2ncjWDTg/lJpaxSOQNZ0W4MK+s=">AAACQXicbVBLSwMxGMzWd31VPXoJFqFeyq4KeikIXjxWsFro1iWbZtvYJLsk3ypl2b/mxX/gzbsXD4p49WL6OGjrQGAyMx9fMmEiuAHXfXEKc/MLi0vLK8XVtfWNzdLW9rWJU01Zg8Yi1s2QGCa4Yg3gIFgz0YzIULCbsH8+9G/umTY8VlcwSFhbkq7iEacErBSUmvWKb3hXkoOan+i4E2S85uW3CvuCRVDxI01o9hBk45B18zy3l1QG2V2ND4O/zDtrYl/zbg8OglLZrboj4FniTUgZTVAPSs9+J6apZAqoIMa0PDeBdkY0cCpYXvRTwxJC+6TLWpYqIplpZ6MGcrxvlQ6OYm2PAjxSf09kRBozkKFNSgI9M+0Nxf+8VgrRaTvjKkmBKTpeFKUCQ4yHdeIO14yCGFhCqOb2rZj2iC0NbOlFW4I3/eVZcn1Y9Y6q7uVx+ex4Uscy2kV7qII8dILO0AWqowai6BG9onf04Tw5b86n8zWOFpzJzA76A+f7BwPtslo=</latexit>
16. The bayesian model
Posterior distribution of the weights Likelihood of the sample
Prior distribution of the weights
NY
k=1
nY
i=1
0
@
w (k)
i
Pn
j=i w (k)
j
1
A
<latexit sha1_base64="382jpMOvUOBX2CNNv68sNU9hmgk=">AAACUHicbVFNaxsxFHzrNh910sRNj72ImoBzMbtNoL0EArnkFFKonYDXWbSydq1Y0i7S2xQj9ifmklt/Ry49tLRa24U27gOheTPzkDRKSykshuG3oPXi5cbm1var9s7u6739zpuDoS0qw/iAFbIwNym1XArNByhQ8pvScKpSya/T2XmjX99zY0Whv+C85GNFcy0ywSh6KunkcWmKSeJmp1F9e7lqRNNoEkueYS/ODGXua+JiK3JFb11vdlR7T13XnqpU4u5ORWNfs9x5C4mNyKd4lHS6YT9cFFkH0Qp0YVVXSecxnhSsUlwjk9TaURSWOHbUoGCS1+24srykbEZzPvJQU8Xt2C0CqcmhZyYkK4xfGsmC/XvCUWXtXKXeqShO7XOtIf+njSrMPo2d0GWFXLPlQVklCRakSZdMhOEM5dwDyozwdyVsSn2A6P+g7UOInj95HQw/9KPjfvj5pHt2sopjG97Be+hBBB/hDC7gCgbA4AGe4Af8DB6D78GvVrC0/tnhLfxTrfZvGq62uA==</latexit>
R = { (1)
, . . . , (N)
}<latexit sha1_base64="p6uONzgcyQDNNmoWvv+HTlxo17g=">AAACD3icbVBNS8NAEN3Ur1q/oh69LBalhVISLehFKHjxJFVsKzSxbDbbdulmE3Y3Qgn5B178K148KOLVqzf/jds2iLY+GHi8N8PMPC9iVCrL+jJyC4tLyyv51cLa+sbmlrm905JhLDBp4pCF4tZDkjDKSVNRxchtJAgKPEba3vB87LfviZA05DdqFBE3QH1OexQjpaWueXh95iSOpP0A3SUlu5xWHOaHSlZ+tMty6qRds2hVrQngPLEzUgQZGl3z0/FDHAeEK8yQlB3bipSbIKEoZiQtOLEkEcJD1CcdTTkKiHSTyT8pPNCKD3uh0MUVnKi/JxIUSDkKPN0ZIDWQs95Y/M/rxKp36iaUR7EiHE8X9WIGVQjH4UCfCoIVG2mCsKD6VogHSCCsdIQFHYI9+/I8aR1V7eOqdVUr1mtZHHmwB/ZBCdjgBNTBBWiAJsDgATyBF/BqPBrPxpvxPm3NGdnMLvgD4+Mb1wab2w==</latexit>
P(w|R) /<latexit sha1_base64="CzIyNBVIpLnUlZDF5eJdtnMe9Lw=">AAAB/3icbVDLSgMxFM3UV62vUcGNm2AR6qbMaEGXBTcuq9gHdIaSSTNtaGYSkoxSpl34K25cKOLW33Dn35hpZ6GtBwKHc+7lnpxAMKq043xbhZXVtfWN4mZpa3tnd8/eP2gpnkhMmpgzLjsBUoTRmDQ11Yx0hCQoChhpB6PrzG8/EKkoj+/1WBA/QoOYhhQjbaSefdSoeBHSwyBMH6eTuzNPSC4079llp+rMAJeJm5MyyNHo2V9en+MkIrHGDCnVdR2h/RRJTTEj05KXKCIQHqEB6Roao4goP53ln8JTo/RhyKV5sYYz9fdGiiKlxlFgJrOsatHLxP+8bqLDKz+lsUg0ifH8UJgwqDnMyoB9KgnWbGwIwpKarBAPkURYm8pKpgR38cvLpHVedS+qzm2tXK/ldRTBMTgBFeCCS1AHN6ABmgCDCXgGr+DNerJerHfrYz5asPKdQ/AH1ucPKnKWJw==</latexit>
1
B
nY
i=1
w↵i 1
i
<latexit sha1_base64="/gfyjh4UDNfus5EbeDuQVHLsAyw=">AAACE3icbVDLSsNAFJ34rPUVdelmsAgiWBIt6EYounFZwT6gScNkMmmHTiZhZqKUkH9w46+4caGIWzfu/BunbRbaeuDC4Zx7ufceP2FUKsv6NhYWl5ZXVktr5fWNza1tc2e3JeNUYNLEMYtFx0eSMMpJU1HFSCcRBEU+I21/eD322/dESBrzOzVKiBuhPqchxUhpyTOPnVAgnNl5dpU7iYgDL6OXdt7j8MGjvcxBLBkgj8ITO/fMilW1JoDzxC5IBRRoeOaXE8Q4jQhXmCEpu7aVKDdDQlHMSF52UkkShIeoT7qachQR6WaTn3J4qJUAhrHQxRWcqL8nMhRJOYp83RkhNZCz3lj8z+umKrxwM8qTVBGOp4vClEEVw3FAMKCCYMVGmiAsqL4V4gHSISkdY1mHYM++PE9ap1X7rGrd1ir1WhFHCeyDA3AEbHAO6uAGNEATYPAInsEreDOejBfj3fiYti4Yxcwe+APj8wfKaZ4G</latexit>
B =
Qn
i=1 (↵i)
(
Pn
i=1 ↵i)<latexit sha1_base64="lQ2UQ095A4jrK9whnNjihdhrbPg=">AAACL3icbVDLSgMxFM34rPU16tJNsAh1U2ZU0I1QFNRlBauFTh3upBkbmmSGJCOUoX/kxl/pRkQRt/6Faa1vDwQO55zLzT1Rypk2nvfgTExOTc/MFuaK8wuLS8vuyuqFTjJFaJ0kPFGNCDTlTNK6YYbTRqooiIjTy6h7NPQvb6jSLJHnppfSloBryWJGwFgpdI8PD4JYAcmDVCXtMGcHfv9K4uAEhIByADztQMi2+vmHojPxlfq0Q7fkVbwR8F/ij0kJjVEL3UHQTkgmqDSEg9ZN30tNKwdlGOG0XwwyTVMgXbimTUslCKpb+ejePt60ShvHibJPGjxSv0/kILTuicgmBZiO/u0Nxf+8Zmbi/VbOZJoZKsn7ojjj2CR4WB5uM0WJ4T1LgChm/4pJB2x9xlZctCX4v0/+Sy62K/5OxTvbLVV3x3UU0DraQGXkoz1URaeohuqIoFs0QI/oyblz7p1n5+U9OuGMZ9bQDzivb04RqSw=</latexit>
No way to sample posterior
distribution exactly à MCMC
18. The Case of Study
23 FUNCTIONS TO OPTIMIZE:
• OneMax (F1) and W-model extensions (F4-F10)
• LeadingOnes (F2) and W-model extensions (F11-
F17)
• Harmonic (F3)
• LABS: Low Autocorrelation Binary Sequences (F18)
• Ising-Ring (F19)
• Ising-Torus (F20)
• Ising-Triangular (F21)
• MIVS: Maximum Independent Vertex Set (F22)
• NQP: N-Queens problem (F23)
n 2 {16, 64, 100, 625}<latexit sha1_base64="HS0JdBr8a6YmSKd4vVyu+TiOCPw=">AAAB/nicbVBNS8NAEJ3Ur1q/ouLJy2IRPJSS1Fr1VvDisYKthSaUzXbbLt1swu5GKKHgX/HiQRGv/g5v/hu3bQ7a+mDg8d4MM/OCmDOlHefbyq2srq1v5DcLW9s7u3v2/kFLRYkktEkiHsl2gBXlTNCmZprTdiwpDgNOH4LRzdR/eKRSsUjc63FM/RAPBOszgrWRuvaR8JjwUrdWqlVLruOUapULb9K1i07ZmQEtEzcjRcjQ6NpfXi8iSUiFJhwr1XGdWPsplpoRTicFL1E0xmSEB7RjqMAhVX46O3+CTo3SQ/1ImhIazdTfEykOlRqHgekMsR6qRW8q/ud1Et2/8lMm4kRTQeaL+glHOkLTLFCPSUo0HxuCiWTmVkSGWGKiTWIFE4K7+PIyaVXK7nm5clct1q+zOPJwDCdwBi5cQh1uoQFNIJDCM7zCm/VkvVjv1se8NWdlM4fwB9bnD6Ask0w=</latexit>
Problem Size:
11 Metaheuristic algorithms:
• greedy Hill Climber (gHC)
• Randomlized Local Search (RLS)
• (1+1) EA
• fast Genetic Algorithm (fGA)
• (1+10) EA
• (1+10) EAr/2,2r
• (1+10) EAnorm
• (1+10) EAvar
• (1+10) EAlog-n
• (1+(λ+λ)) GA
• “vanilla” GA (vGA)
Results of 11.132 runs are collected (23 x 4 x 11 x 11)
• Aggregation of performances across 11 instances.
• Median performance across 11 repetitions.
Estimate the probability of each algorithm being top-ranked
• as its expected weight in the posterior distribution of weights
Analyze the uncertainty about the probabilities
• By estimating the 90% credible intervals of the posterior distribution of weights (5% and 95%)
19. Inference analyses & results
QUALITATIVE SUMMARY
Similar perf. (1+(λ+λ)) GA, (1+1)-EA, (1+10)-EAvar, (1+10)-Ealog-n, (1+10)-Eanorm,(1+10)-EAr/2,2r and fGA.
Extreme perf. vGA and gHC.
Easily treated instances are F1-F6, F8, F11-F13 and F15-16.
Best solutions found for n=625
20. Inference analyses & results
Fixed-target perspective – Record Running-time
(1+( , )) GA
(1+1) EA
gHC
(1+10) EA_r/2,2r
(1+10) EA
(1+10) EA_log-n.
(1+10) EA_norm.
(1+1) EA_var.
fGA
vGA
RLS
0.0 0.2 0.4 0.6
Probability of winning
Algorithm
F17, n=625, φ=625 F19, n=100, φ=100
(1+( , )) GA
(1+1) EA
gHC
(1+10) EA_r/2,2r
(1+10) EA
(1+10) EA_log-n.
(1+10) EA_norm.
(1+1) EA_var.
fGA
vGA
RLS
0.0 0.1 0.2 0.3 0.4 0.5
Probability of winning
Algorithm
Credible Intervals
Only 11 samples to do inference à High uncertainty is expected!
The more samples, the lower the uncertainty à Credibility intervals are more tight!
Expected
probability
High
uncertainty
INTERPRETABILITY
21. Inference analyses & results
Fixed-target perspective – Record Running-time – Set of easy functions
(1+( , )) GA
(1+1) EA
gHC
(1+10) EA_r/2,2r
(1+10) EA
(1+10) EA_log-n.
(1+10) EA_norm.
(1+1) EA_var.
fGA
vGA
RLS
0.00 0.25 0.50 0.75 1.00
Probability of winning
Algorithm
n=625, all runs
(1+( , )) GA
(1+1) EA
gHC
(1+10) EA_r/2,2r
(1+10) EA
(1+10) EA_log-n.
(1+10) EA_norm.
(1+1) EA_var.
fGA
vGA
RLS
0.0 0.2 0.4 0.6
Probability of winning
Algorithm
n=625, median
Credible Intervals
Set of functions, two paths à (1) take all the runs, (2) take the median of the runs on each instance.
gHC is the best in both cases à with more samples the uncertainty is lower
22. Inference analyses & results
Fixed-target perspective – Record Running-time – Set of non-easy functions
Credible Intervals
Good estimations à credible intervals smaller than 0.05
Probabilities are similar à due to overlapping
Uncertainty about which is the best à but not due to
limitation of data, but due to equivalence in the
algorithms
(1+( , )) GA
(1+1) EA
gHC
(1+10) EA_r/2,2r
(1+10) EA
(1+10) EA_log-n.
(1+10) EA_norm.
(1+1) EA_var.
fGA
vGA
RLS
0.050 0.075 0.100 0.125 0.150
Probability of winning
Algorithm
n=625, all runs
23. Inference analyses & results
Fixed-budget perspective – Evolution winning probability - %90 credibility intervals
0.0
0.2
0.4
0.6
0 300 600 900
Budget
Winningprobability
(1+( , )) GA
(1+1) EA
gHC
(1+10) EA_r/2,2r
(1+10) EA
(1+10) EA_log-n.
(1+10) EA_norm.
(1+1) EA_var.
fGA
vGA
RLS
F21, n=100
gHC is the best, but probability decreases while the rest improve.
gHC becomes better, as the budget increases.
3 4 5 6 7 8 9 10 11
Algorithms ranked with average data
Wilcoxon test for pairwise comparisons, and
shaffer’s method for p-value correction.
BAYESIAN ANALYSIS
ESTIMATED PROBABILITY AND
NOTION OF UNCERTAINTY IN THE
FORM OF CREDIBLE INTERVAL
24. Inference analyses & results
Impact of the prior distribution – Comparison of three different priors
0.0
0.2
0.4
0.6
(1+(
,
))G
A
(1+1)EA
gH
C
(1+10)EA_r/2,2r
(1+10)EA
(1+10)EA_log-n.
(1+10)EA_norm
.
(1+1)EA_var.
fG
A
vG
A
R
LS
Algorithm
Winningprobability
Prior Unifor Empirical Deceptive
F9, n=100, φ=100
Empirical data favours the best
performing algorithms
Neligible effect (even when median
values are considered)
25. Discussion
Bayesian inference using Plackett-Luce for analysis of algorithms’ performance ranking
Include it in the practical EC performance comparison’ tool set à IOHProfiler
Strong points
Ability to handle multiple
algorithms
Interpretability
Exact description of the
uncertainty
WEAKNESSES
Aggregating performances into
rankings we loose information about
the magnitude of differences
Limitations of the Plackett-Luce model
à From n! to n parameters.
How do we deal with ties?