The document discusses enabling open and reproducible research at computer systems conferences. It outlines some of the challenges, including highly stochastic behavior making results difficult to reproduce, and ad-hoc non-portable scripts. The document presents initiatives like Artifact Evaluation to address these issues by promoting artifact sharing and developing common reproducibility methodologies. It also discusses both successes and remaining challenges in improving openness and reproducibility.
CGO/PPoPP'17 Artifact Evaluation Discussion (enabling open and reproducible r...Grigori Fursin
This year we had a record number of artifact submissions at CGO/PPoPP'17: 27 vs 17 two years ago. It is really great to see that researchers are now taking AE seriously, but it also highlighted new issues with AE scalability and lack of common experimental methodology and workflow frameworks in computer systems' research. Therefore, we discussed a few possible solutions for the next AE including public artifact reviewing, common workflow frameworks, artifact appendices, partial artifact evaluation (artifact available, artifact validated, experiment reproduced) and "tool" papers. Please, feel free to provide your own feedback to the AE steering committee!
More details:
* http://dividiti.blogspot.fr/2017/01/artifact-evaluation-discussion-session.html
* http://cTuning.org/ae
* http://cKnowledge.org
Adapting to a Cambrian AI/SW/HW explosion with open co-design competitions an...Grigori Fursin
Slides from ARM's Research Summit'17 about "Community-Driven and Knowledge-Guided Optimization of AI Applications Across the Whole SW/HW Stack" (http://cKnowledge.org/repo , http://cKnowledge.org/ai , http://tinyurl.com/zlbxvmw , https://developer.arm.com/research/summit )
Co-designing the whole AI/SW/HW stack in terms of speed, accuracy, energy consumption, size, costs and other metrics has become extremely complex, long and costly. With no rigorous methodology for analyzing performance and accumulating optimisation knowledge, we are simply destined to drown in the ever growing number of design choices, system
features and conflicting optimisation goals.
We present our novel community-driven approach to solve the above problems. Originating from natural sciences, this approach is embodied in Collective Knowledge (CK), our open-source cross-platform workflow framework and repository for automatic, collaborative and reproducible experimentation. CK helps organize, unify and share representative workloads, data sets, AI frameworks, libraries, compilers, scripts, models and other artifacts as customizable and reusable components with a common JSON API.
CK helps bring academia, industry and end-users together to
gradually expose optimisation choices at all levels (e.g. from parameterized models and algorithmic skeletons to compiler
flags and hardware configurations) and autotune them across diverse inputs and platforms. Optimization knowledge gets continuously aggregated in public or private repositories such as cKnowledge.org/repo in a reproducible way, and can be then mined and extrapolated to predict better AI algorithm choices, compiler transformations and hardware designs.
We also demonstrate how we use this approach in practice together with ARM and other companies to adapt to a Cambrian AI/SW/HW explosion by creating an open repository of reusable AI artifacts, and then collaboratively optimising and co-designing the whole deep learning stack (software, hardware and models).
Accelerating open science and AI with automated, portable, customizable and r...Grigori Fursin
Validating experimental results from articles has finally become a norm at many systems and ML conferences. Nowadays, more than half of accepted papers pass artifact evaluation and share related code and data. Unfortunately, lack of a common experimental framework, common research methodology and common formats places an increasing burden on evaluators to validate a growing number of ad-hoc artifacts. Furthermore, having too many ad-hoc artifacts and Docker snapshots is almost as bad as not having any (!), since they cannot be easily reused, customized and built upon.
While overviewing more than 100 papers during artifact evaluation at PPoPP, CGO, PACT, Supercomputing and other conferences, we noticed that many of them use similar experimental setups, benchmarks, models, data sets, environments and platforms. This motivated us to develop Collective Knowledge (CK), an open workflow framework with a unified Python API to automate common researchers’ tasks such as detecting software and hardware dependencies, installing missing packages, downloading data sets and models, compiling and running programs, performing autotuning and co-design, crowdsourcing time-consuming experiments across computing resources provided by volunteers similar to SETI@home, applying statistical analysis and machine learning, validating results and plotting them on a common scoreboard for open and fair comparison, automatically generating interactive articles, and so on: http://cKnowledge.org.
In this presentation we will introduce CK concepts and present several real world use cases from General Motors and Arm
on collaborative benchmarking, autotuning and co-design of efficient software/hardware stacks for deep learning. We also present results and reusable CK components from the 1st ACM ReQuEST optimization tournament: http://cKnowledge.org/request. Finally, we introduce our latest initiative to create
an open repository of reusable research components and workflows to reboot and accelerate open science, quantum computing and AI!
Collective Mind: bringing reproducible research to the massesGrigori Fursin
When trying to make auto-tuning practical using common infrastructure, public repository of knowledge, and machine
learning (cTuning.org), we faced a major problem with reproducibility of experimental results collected from multiple users. This was largely due to a lack of information about all software and hardware dependencies as well as a large variation of measured characteristics.
I will present a possible collaborative approach to solve aboveproblems using a new Collective Mind knowledge management system. This modular infrastructure is intended to preserve and share through Internet the whole experimental setups with all related artifacts and their software and hardware dependencies besides just performance data. Researchers can take advantage of shared components and data with extensible meta-description at http://c-mind.org/repo to quickly prototype and validate research techniques particularly on software and hardware optimization and co-design. At the same time, behavior anomalies or model mispredictions can be exposed in a reproducible way to interdisciplinary community for further analysis and improvement. This approach supports our new open publication model in computer engineering where all results and artifacts are continuously shared and validated by the community (c-mind.org/events/trust2014).
This presentations supports our recent publication:
* http://iospress.metapress.com/content/f255p63828m8l384
* http://hal.inria.fr/hal-01054763
'Scikit-project': How open source is empowering open science – and vice versaNathan Shammah
Open-source pipelines are accelerating scientific discovery, by empowering not only reproducibility of research results but also generalizability of methods. I address the rise of open source in scientific research in quantum physics and quantum information and introduce `scikit-project` a cookbook with best practices for (data) scientists.
See also https://github.com/Machine-Learning-Tokyo/MLT_Talks
Stay up-to-date on the latest news, events and resources for the OpenACC community. This month’s highlights covers the most recent 2019 GPU Hackathons, a complete schedule of upcoming events, new resources and more!
Stay up-to-date on the latest news, events and resources for the OpenACC community. This month’s highlights covers the upcoming OpenACC Summit, a complete schedule of upcoming events, using OpenACC to optimize structural analysis, new resources and more!
Stay up-to-date with the OpenACC Monthly Highlights. August's edition covers the OpenACC Summit 2021, upcoming GPU Hackathons and Bootcamps, pioneers of Frontier, new resources and more!
CGO/PPoPP'17 Artifact Evaluation Discussion (enabling open and reproducible r...Grigori Fursin
This year we had a record number of artifact submissions at CGO/PPoPP'17: 27 vs 17 two years ago. It is really great to see that researchers are now taking AE seriously, but it also highlighted new issues with AE scalability and lack of common experimental methodology and workflow frameworks in computer systems' research. Therefore, we discussed a few possible solutions for the next AE including public artifact reviewing, common workflow frameworks, artifact appendices, partial artifact evaluation (artifact available, artifact validated, experiment reproduced) and "tool" papers. Please, feel free to provide your own feedback to the AE steering committee!
More details:
* http://dividiti.blogspot.fr/2017/01/artifact-evaluation-discussion-session.html
* http://cTuning.org/ae
* http://cKnowledge.org
Adapting to a Cambrian AI/SW/HW explosion with open co-design competitions an...Grigori Fursin
Slides from ARM's Research Summit'17 about "Community-Driven and Knowledge-Guided Optimization of AI Applications Across the Whole SW/HW Stack" (http://cKnowledge.org/repo , http://cKnowledge.org/ai , http://tinyurl.com/zlbxvmw , https://developer.arm.com/research/summit )
Co-designing the whole AI/SW/HW stack in terms of speed, accuracy, energy consumption, size, costs and other metrics has become extremely complex, long and costly. With no rigorous methodology for analyzing performance and accumulating optimisation knowledge, we are simply destined to drown in the ever growing number of design choices, system
features and conflicting optimisation goals.
We present our novel community-driven approach to solve the above problems. Originating from natural sciences, this approach is embodied in Collective Knowledge (CK), our open-source cross-platform workflow framework and repository for automatic, collaborative and reproducible experimentation. CK helps organize, unify and share representative workloads, data sets, AI frameworks, libraries, compilers, scripts, models and other artifacts as customizable and reusable components with a common JSON API.
CK helps bring academia, industry and end-users together to
gradually expose optimisation choices at all levels (e.g. from parameterized models and algorithmic skeletons to compiler
flags and hardware configurations) and autotune them across diverse inputs and platforms. Optimization knowledge gets continuously aggregated in public or private repositories such as cKnowledge.org/repo in a reproducible way, and can be then mined and extrapolated to predict better AI algorithm choices, compiler transformations and hardware designs.
We also demonstrate how we use this approach in practice together with ARM and other companies to adapt to a Cambrian AI/SW/HW explosion by creating an open repository of reusable AI artifacts, and then collaboratively optimising and co-designing the whole deep learning stack (software, hardware and models).
Accelerating open science and AI with automated, portable, customizable and r...Grigori Fursin
Validating experimental results from articles has finally become a norm at many systems and ML conferences. Nowadays, more than half of accepted papers pass artifact evaluation and share related code and data. Unfortunately, lack of a common experimental framework, common research methodology and common formats places an increasing burden on evaluators to validate a growing number of ad-hoc artifacts. Furthermore, having too many ad-hoc artifacts and Docker snapshots is almost as bad as not having any (!), since they cannot be easily reused, customized and built upon.
While overviewing more than 100 papers during artifact evaluation at PPoPP, CGO, PACT, Supercomputing and other conferences, we noticed that many of them use similar experimental setups, benchmarks, models, data sets, environments and platforms. This motivated us to develop Collective Knowledge (CK), an open workflow framework with a unified Python API to automate common researchers’ tasks such as detecting software and hardware dependencies, installing missing packages, downloading data sets and models, compiling and running programs, performing autotuning and co-design, crowdsourcing time-consuming experiments across computing resources provided by volunteers similar to SETI@home, applying statistical analysis and machine learning, validating results and plotting them on a common scoreboard for open and fair comparison, automatically generating interactive articles, and so on: http://cKnowledge.org.
In this presentation we will introduce CK concepts and present several real world use cases from General Motors and Arm
on collaborative benchmarking, autotuning and co-design of efficient software/hardware stacks for deep learning. We also present results and reusable CK components from the 1st ACM ReQuEST optimization tournament: http://cKnowledge.org/request. Finally, we introduce our latest initiative to create
an open repository of reusable research components and workflows to reboot and accelerate open science, quantum computing and AI!
Collective Mind: bringing reproducible research to the massesGrigori Fursin
When trying to make auto-tuning practical using common infrastructure, public repository of knowledge, and machine
learning (cTuning.org), we faced a major problem with reproducibility of experimental results collected from multiple users. This was largely due to a lack of information about all software and hardware dependencies as well as a large variation of measured characteristics.
I will present a possible collaborative approach to solve aboveproblems using a new Collective Mind knowledge management system. This modular infrastructure is intended to preserve and share through Internet the whole experimental setups with all related artifacts and their software and hardware dependencies besides just performance data. Researchers can take advantage of shared components and data with extensible meta-description at http://c-mind.org/repo to quickly prototype and validate research techniques particularly on software and hardware optimization and co-design. At the same time, behavior anomalies or model mispredictions can be exposed in a reproducible way to interdisciplinary community for further analysis and improvement. This approach supports our new open publication model in computer engineering where all results and artifacts are continuously shared and validated by the community (c-mind.org/events/trust2014).
This presentations supports our recent publication:
* http://iospress.metapress.com/content/f255p63828m8l384
* http://hal.inria.fr/hal-01054763
'Scikit-project': How open source is empowering open science – and vice versaNathan Shammah
Open-source pipelines are accelerating scientific discovery, by empowering not only reproducibility of research results but also generalizability of methods. I address the rise of open source in scientific research in quantum physics and quantum information and introduce `scikit-project` a cookbook with best practices for (data) scientists.
See also https://github.com/Machine-Learning-Tokyo/MLT_Talks
Stay up-to-date on the latest news, events and resources for the OpenACC community. This month’s highlights covers the most recent 2019 GPU Hackathons, a complete schedule of upcoming events, new resources and more!
Stay up-to-date on the latest news, events and resources for the OpenACC community. This month’s highlights covers the upcoming OpenACC Summit, a complete schedule of upcoming events, using OpenACC to optimize structural analysis, new resources and more!
Stay up-to-date with the OpenACC Monthly Highlights. August's edition covers the OpenACC Summit 2021, upcoming GPU Hackathons and Bootcamps, pioneers of Frontier, new resources and more!
Stay up-to-date on the latest news, events and resources for the OpenACC community. This month’s highlights covers the GPU Hackathon Mentor program, Helmholtz virtual hackathon, GTC21, upcoming GPU Hackathons and Bootcamps, and new resources!
Stay up-to-date on the latest news, events and resources for the OpenACC community. This month’s highlights covers a Mentor Spotlight on Matthew Norman from ORNL, the first GPU Hackathon of the 2021 season, GTC21, Clacc, upcoming GPU Hackathons and Bootcamps, and new resources!
Stay up-to-date on the latest news, events and resources for the OpenACC community. This month’s highlights covers a complete schedule of upcoming events, using OpenACC for a biophysics problem, HPC Summit Digital, overview of the SDSC GPU Hackathon, OmpSs-2 programming model, new resources and more!
Stay up-to-date on the latest news, events and resources for the OpenACC community. This month’s highlights covers the first remote GPU Hackathons, a complete schedule of upcoming events, using OpenACC for a biophysics problem, NVIDIA HPC SDK, GCC 10, new resources and more!
Stay up-to-date on the latest news, events and resources for the OpenACC community. This month’s highlights covers pseudo random number generation, the first-ever MONAI Bootcamp, upcoming GPU Hackathons and Bootcamps, and new resources!
Personal point of view on scikit-learn: past, present, and future.
This talks gives a bit of history, mentions exciting development, and a personal vision on the future.
AI for All: Biology is eating the world & AI is eating Biology Intel® Software
Advances in cell biology and creation of an immense amount of data are converging with advances in Machine learning to analyze this data. Biology is experiencing its AI moment and driving the massive computation involved in understanding biological mechanisms and driving interventions. Learn about how cutting edge technologies such as Software Guard Extensions (SGX) in the latest Intel Xeon Processors and Open Federated Learning (OpenFL), an open framework for federated learning developed by Intel, are helping advance AI in gene therapy, drug design, disease identification and more.
Stay up-to-date on the latest news, events and resources for the OpenACC community. This month’s highlights covers the upcoming OpenACC Summit and GPU Bootcamp, a complete schedule of upcoming events, OpenACC and base language parallelism, FortranCon2020, VASP 6, OmpSs-2@OpenACC version of the ZIPC application, new resources and more!
Calendrier des activités de JEADER _ AFRIQUE _ 2016 JEADER
2016 a été une année spéciale grâce à vous ! Merci pour votre support et revisitez les activités phares de 2016.
Ensemble faisons de 2017 une année tout aussi EXTRAORDINAIRE !
Experience the wonders of Sri Lanka from the best individual tour operator with BUDGET PRICES.
TRUST US WITH YOUR HOLIDAYS WE HAVE FOR YOU .WE OFFER A WIDE CHOICE OF TOURS: OUR EXPERTS CAN ALSO “TAILOR “A HOLIDAY TO YOUR PERSONAL WISHES.
Stay up-to-date on the latest news, events and resources for the OpenACC community. This month’s highlights covers the GPU Hackathon Mentor program, Helmholtz virtual hackathon, GTC21, upcoming GPU Hackathons and Bootcamps, and new resources!
Stay up-to-date on the latest news, events and resources for the OpenACC community. This month’s highlights covers a Mentor Spotlight on Matthew Norman from ORNL, the first GPU Hackathon of the 2021 season, GTC21, Clacc, upcoming GPU Hackathons and Bootcamps, and new resources!
Stay up-to-date on the latest news, events and resources for the OpenACC community. This month’s highlights covers a complete schedule of upcoming events, using OpenACC for a biophysics problem, HPC Summit Digital, overview of the SDSC GPU Hackathon, OmpSs-2 programming model, new resources and more!
Stay up-to-date on the latest news, events and resources for the OpenACC community. This month’s highlights covers the first remote GPU Hackathons, a complete schedule of upcoming events, using OpenACC for a biophysics problem, NVIDIA HPC SDK, GCC 10, new resources and more!
Stay up-to-date on the latest news, events and resources for the OpenACC community. This month’s highlights covers pseudo random number generation, the first-ever MONAI Bootcamp, upcoming GPU Hackathons and Bootcamps, and new resources!
Personal point of view on scikit-learn: past, present, and future.
This talks gives a bit of history, mentions exciting development, and a personal vision on the future.
AI for All: Biology is eating the world & AI is eating Biology Intel® Software
Advances in cell biology and creation of an immense amount of data are converging with advances in Machine learning to analyze this data. Biology is experiencing its AI moment and driving the massive computation involved in understanding biological mechanisms and driving interventions. Learn about how cutting edge technologies such as Software Guard Extensions (SGX) in the latest Intel Xeon Processors and Open Federated Learning (OpenFL), an open framework for federated learning developed by Intel, are helping advance AI in gene therapy, drug design, disease identification and more.
Stay up-to-date on the latest news, events and resources for the OpenACC community. This month’s highlights covers the upcoming OpenACC Summit and GPU Bootcamp, a complete schedule of upcoming events, OpenACC and base language parallelism, FortranCon2020, VASP 6, OmpSs-2@OpenACC version of the ZIPC application, new resources and more!
Calendrier des activités de JEADER _ AFRIQUE _ 2016 JEADER
2016 a été une année spéciale grâce à vous ! Merci pour votre support et revisitez les activités phares de 2016.
Ensemble faisons de 2017 une année tout aussi EXTRAORDINAIRE !
Experience the wonders of Sri Lanka from the best individual tour operator with BUDGET PRICES.
TRUST US WITH YOUR HOLIDAYS WE HAVE FOR YOU .WE OFFER A WIDE CHOICE OF TOURS: OUR EXPERTS CAN ALSO “TAILOR “A HOLIDAY TO YOUR PERSONAL WISHES.
A Administração e a Contabilidade : Uma relação que gera vantagem competitiva...Cra-es Conselho
Nesta palestra o Prof. Hercules Vander de Lima Freire fala sobre as diferenças entre as profissões, a Características qualitativas da informação contábil-financeira útil; Características qualitativas fundamentais e Características qualitativas de melhoria.
CK: from ad hoc computer engineering to collaborative and reproducible data s...Grigori Fursin
Designing novel computer systems and optimizing their software is becoming too tedious, ad hoc, time consuming and error prone due to enormous number of available design and optimization choices. Empirical autotuning combined with run-time adaptation and machine learning has been demonstrating some potential to address above challenges for several decades but is still far from the widespread production. The main reasons include unbearably long exploration and training times, ever changing tools and their interfaces, lack of a common experimental methodology, lack of diverse and representative benchmarks, and lack of unified mechanisms for knowledge building and exchange apart from publications where reproducibility and reusability of results is often not even considered.
I will present our community-driven solution to above problems based on our open-source Collective Knowledge technology (CK) that can gradually organize, exchange and reuse knowledge and experience in computer engineering. CK helps share various artifacts (benchmarks, data sets, libraries, tools) as unified, reusable and Python-based components with JSON meta description via GITHUB. Researchers can then quickly prototype and crowdsource various experimental workflows such as performance and energy autotuning, design space exploration and run-time adaptation. At the same time, CK continuously analyzes and extrapolates all collected knowledge using powerful data science techniques to automatically model computer systems' behavior, predict better optimizations or hardware configurations, and eventually enable faster, more power efficient, reliable and self-tuning software and hardware. Furthermore, CK can record any unexpected behavior in a reproducible way and expose it to an interdisciplinary community to find missing features and improve models. Live demo of our approach is available at http://cknowledge.org/repo .
Collective Mind: a collaborative curation tool for program optimizationGrigori Fursin
Designing and optimizing applications becomes increasingly tedious, time consuming, ad-hoc and error prone due to ever changing and complex hardware and software stack. At the same time, it becomes difficult or even impossible to validate, reproduce and extend many proposed optimization and auto-tuning techniques from numerous publications. One of the main reasons is a lack of common and practical way to preserve, systematize and reuse available knowledge and artifacts including developments, optimizations and experimental data.
In this talk, I will present modular, extensible, python-based Collective Mind framework and web-based schema-free repository
(c-mind.org) that I developed at first to systematize my own research and experimentation on machine learning based program optimization and compiler design. This infrastructure can be customized to preserve, describe, share and reproduce whole experimental setups including benchmarks, data sets, libraries, tools, predictive models and optimization information with related JSON-based meta-data. I will also present and discuss positive and negative feedback during several recent academic and industrial usages of this framework to systematize benchmarking and program optimization, and to initiate new publication model where experimental results and related research artifacts are shared, reproduced and validated by the community. In a long term, I hope that such approach and collective knowledge will eventually help us to squeeze maximum performance from computer systems while minimizing energy, development time and other costs.
In this panel, we shared our practical experience on crowdsourcing evaluation of research artifacts and reviewing of publications since 2008. We also discussed encountered problems including reproducibility of experimental results and possible solutions. We also present our new publication model proposal for open access conferences and journals that we plan to validate at ADAPT 2015.
[3.6] Beyond Data Sharing - Pieter van Gorp [3TU.Datacentrum Symposium 2014, ...3TU.Datacentrum
3TU.Datacentrum Symposium Research Data Management:
Funder requirements, Questions and Solutions
At this symposium the funding organisation NWO and the European Commission explained their vision, plans and requirements. Researchers from the three universities of technology shared their experiences of data management in different stages of research. And the Research Data Services team informed the audience about research data management services offered by 3TU.Datacentrum.
The 3TU.Datacentrum symposium took place at the TU Delft (26 May), University of Twente (2 June) and TU Eindhoven (11 June) for and with local researchers.
More information on: datacentrum.3tu.nl/over-3tudatacentrum/symposium-2014
Keynote on software sustainability given at the 2nd Annual Netherlands eScience Symposium, November 2014.
Based on the article
Carole Goble ,
Better Software, Better Research
Issue No.05 - Sept.-Oct. (2014 vol.18)
pp: 4-8
IEEE Computer Society
http://www.computer.org/csdl/mags/ic/2014/05/mic2014050004.pdf
http://doi.ieeecomputersociety.org/10.1109/MIC.2014.88
http://www.software.ac.uk/resources/publications/better-software-better-research
German Conference on Bioinformatics 2021
https://gcb2021.de/
FAIR Computational Workflows
Computational workflows capture precise descriptions of the steps and data dependencies needed to carry out computational data pipelines, analysis and simulations in many areas of Science, including the Life Sciences. The use of computational workflows to manage these multi-step computational processes has accelerated in the past few years driven by the need for scalable data processing, the exchange of processing know-how, and the desire for more reproducible (or at least transparent) and quality assured processing methods. The SARS-CoV-2 pandemic has significantly highlighted the value of workflows.
This increased interest in workflows has been matched by the number of workflow management systems available to scientists (Galaxy, Snakemake, Nextflow and 270+ more) and the number of workflow services like registries and monitors. There is also recognition that workflows are first class, publishable Research Objects just as data are. They deserve their own FAIR (Findable, Accessible, Interoperable, Reusable) principles and services that cater for their dual roles as explicit method description and software method execution [1]. To promote long-term usability and uptake by the scientific community, workflows (as well as the tools that integrate them) should become FAIR+R(eproducible), and citable so that author’s credit is attributed fairly and accurately.
The work on improving the FAIRness of workflows has already started and a whole ecosystem of tools, guidelines and best practices has been under development to reduce the time needed to adapt, reuse and extend existing scientific workflows. An example is the EOSC-Life Cluster of 13 European Biomedical Research Infrastructures which is developing a FAIR Workflow Collaboratory based on the ELIXIR Research Infrastructure for Life Science Data Tools ecosystem. While there are many tools for addressing different aspects of FAIR workflows, many challenges remain for describing, annotating, and exposing scientific workflows so that they can be found, understood and reused by other scientists.
This keynote will explore the FAIR principles for computational workflows in the Life Science using the EOSC-Life Workflow Collaboratory as an example.
[1] Carole Goble, Sarah Cohen-Boulakia, Stian Soiland-Reyes,Daniel Garijo, Yolanda Gil, Michael R. Crusoe, Kristian Peters, and Daniel Schober FAIR Computational Workflows Data Intelligence 2020 2:1-2, 108-121 https://doi.org/10.1162/dint_a_00033.
We presented our experience with Artifact Evaluation for CGO'15 (Symposium on Code Generation and Optimization) and PPoPP'15 (Symposium on Principles and Practice of Parallel Programming). We listed some of the encountered challenges and a few possible solutions for the future AE.
In this talk, we discuss QuTrack, a Blockchain-based approach to track experiment and model changes primarily for AI and ML models. In addition, we discuss how change analytics can be used for process improvement and to enhance the model development and deployment processes.
SBFT Tool Competition 2024 -- Python Test Case Generation TrackSebastiano Panichella
Nicolas Erni, Al-Ameen, Mohammed, Christian Birchler, Pouria Derakhshanfar, Stephan Lukasczyk, Sebastiano Panichella: SBFT Tool Competition 2024 -- Python Test Case Generation Track 17th International Workshop on Search-Based and Fuzz Testing
Cloud Standards in the Real World: Cloud Standards Testing for DevelopersAlan Sill
Learn about standards studied in the US National Science Foundation Cloud and Autonomic Computing Industry/University Cooperative Research Center Cloud Standards Testing Lab and how you can get involved to extend the successes from these results in your own cloud software settings. Presented at the O'Reilly OSCON 2014 Open Cloud Day.
Video available at https://www.youtube.com/watch?v=eD2h0SqC7tY
Bridging Concepts and Practice in eScience via Simulation-driven EngineeringRafael Ferreira da Silva
The CyberInfrastructure (CI) has been the object of intensive research and development in the last decade, re- sulting in a rich set of abstractions and interoperable software implementations that are used in production today for supporting ongoing and breakthrough scientific discoveries. A key challenge is the development of tools and application execution frameworks that are robust in current and emerging CI configurations, and that can anticipate the needs of upcoming CI applications. This paper presents WRENCH, a framework that enables simulation-driven engineering for evaluating and developing CI application execution frameworks. WRENCH provides a set of high- level simulation abstractions that serve as building blocks for developing custom simulators. These abstractions rely on the scalable and accurate simulation models that are provided by the SimGrid simulation framework. Consequently, WRENCH makes it possible to build, with minimum software development effort, simulators that that can accurately and scalably simulate a wide spectrum of large and complex CI scenarios. These simulators can then be used to evaluate and/or compare alternate platform, system, and algorithm designs, so as to drive the development of CI solutions for current and emerging applications.
Similar to Enabling open and reproducible computer systems research: the good, the bad and the ugly (20)
Introduction:
RNA interference (RNAi) or Post-Transcriptional Gene Silencing (PTGS) is an important biological process for modulating eukaryotic gene expression.
It is highly conserved process of posttranscriptional gene silencing by which double stranded RNA (dsRNA) causes sequence-specific degradation of mRNA sequences.
dsRNA-induced gene silencing (RNAi) is reported in a wide range of eukaryotes ranging from worms, insects, mammals and plants.
This process mediates resistance to both endogenous parasitic and exogenous pathogenic nucleic acids, and regulates the expression of protein-coding genes.
What are small ncRNAs?
micro RNA (miRNA)
short interfering RNA (siRNA)
Properties of small non-coding RNA:
Involved in silencing mRNA transcripts.
Called “small” because they are usually only about 21-24 nucleotides long.
Synthesized by first cutting up longer precursor sequences (like the 61nt one that Lee discovered).
Silence an mRNA by base pairing with some sequence on the mRNA.
Discovery of siRNA?
The first small RNA:
In 1993 Rosalind Lee (Victor Ambros lab) was studying a non- coding gene in C. elegans, lin-4, that was involved in silencing of another gene, lin-14, at the appropriate time in the
development of the worm C. elegans.
Two small transcripts of lin-4 (22nt and 61nt) were found to be complementary to a sequence in the 3' UTR of lin-14.
Because lin-4 encoded no protein, she deduced that it must be these transcripts that are causing the silencing by RNA-RNA interactions.
Types of RNAi ( non coding RNA)
MiRNA
Length (23-25 nt)
Trans acting
Binds with target MRNA in mismatch
Translation inhibition
Si RNA
Length 21 nt.
Cis acting
Bind with target Mrna in perfect complementary sequence
Piwi-RNA
Length ; 25 to 36 nt.
Expressed in Germ Cells
Regulates trnasposomes activity
MECHANISM OF RNAI:
First the double-stranded RNA teams up with a protein complex named Dicer, which cuts the long RNA into short pieces.
Then another protein complex called RISC (RNA-induced silencing complex) discards one of the two RNA strands.
The RISC-docked, single-stranded RNA then pairs with the homologous mRNA and destroys it.
THE RISC COMPLEX:
RISC is large(>500kD) RNA multi- protein Binding complex which triggers MRNA degradation in response to MRNA
Unwinding of double stranded Si RNA by ATP independent Helicase
Active component of RISC is Ago proteins( ENDONUCLEASE) which cleave target MRNA.
DICER: endonuclease (RNase Family III)
Argonaute: Central Component of the RNA-Induced Silencing Complex (RISC)
One strand of the dsRNA produced by Dicer is retained in the RISC complex in association with Argonaute
ARGONAUTE PROTEIN :
1.PAZ(PIWI/Argonaute/ Zwille)- Recognition of target MRNA
2.PIWI (p-element induced wimpy Testis)- breaks Phosphodiester bond of mRNA.)RNAse H activity.
MiRNA:
The Double-stranded RNAs are naturally produced in eukaryotic cells during development, and they have a key role in regulating gene expression .
(May 29th, 2024) Advancements in Intravital Microscopy- Insights for Preclini...Scintica Instrumentation
Intravital microscopy (IVM) is a powerful tool utilized to study cellular behavior over time and space in vivo. Much of our understanding of cell biology has been accomplished using various in vitro and ex vivo methods; however, these studies do not necessarily reflect the natural dynamics of biological processes. Unlike traditional cell culture or fixed tissue imaging, IVM allows for the ultra-fast high-resolution imaging of cellular processes over time and space and were studied in its natural environment. Real-time visualization of biological processes in the context of an intact organism helps maintain physiological relevance and provide insights into the progression of disease, response to treatments or developmental processes.
In this webinar we give an overview of advanced applications of the IVM system in preclinical research. IVIM technology is a provider of all-in-one intravital microscopy systems and solutions optimized for in vivo imaging of live animal models at sub-micron resolution. The system’s unique features and user-friendly software enables researchers to probe fast dynamic biological processes such as immune cell tracking, cell-cell interaction as well as vascularization and tumor metastasis with exceptional detail. This webinar will also give an overview of IVM being utilized in drug development, offering a view into the intricate interaction between drugs/nanoparticles and tissues in vivo and allows for the evaluation of therapeutic intervention in a variety of tissues and organs. This interdisciplinary collaboration continues to drive the advancements of novel therapeutic strategies.
What is greenhouse gasses and how many gasses are there to affect the Earth.moosaasad1975
What are greenhouse gasses how they affect the earth and its environment what is the future of the environment and earth how the weather and the climate effects.
Seminar of U.V. Spectroscopy by SAMIR PANDASAMIR PANDA
Spectroscopy is a branch of science dealing the study of interaction of electromagnetic radiation with matter.
Ultraviolet-visible spectroscopy refers to absorption spectroscopy or reflect spectroscopy in the UV-VIS spectral region.
Ultraviolet-visible spectroscopy is an analytical method that can measure the amount of light received by the analyte.
Deep Behavioral Phenotyping in Systems Neuroscience for Functional Atlasing a...Ana Luísa Pinho
Functional Magnetic Resonance Imaging (fMRI) provides means to characterize brain activations in response to behavior. However, cognitive neuroscience has been limited to group-level effects referring to the performance of specific tasks. To obtain the functional profile of elementary cognitive mechanisms, the combination of brain responses to many tasks is required. Yet, to date, both structural atlases and parcellation-based activations do not fully account for cognitive function and still present several limitations. Further, they do not adapt overall to individual characteristics. In this talk, I will give an account of deep-behavioral phenotyping strategies, namely data-driven methods in large task-fMRI datasets, to optimize functional brain-data collection and improve inference of effects-of-interest related to mental processes. Key to this approach is the employment of fast multi-functional paradigms rich on features that can be well parametrized and, consequently, facilitate the creation of psycho-physiological constructs to be modelled with imaging data. Particular emphasis will be given to music stimuli when studying high-order cognitive mechanisms, due to their ecological nature and quality to enable complex behavior compounded by discrete entities. I will also discuss how deep-behavioral phenotyping and individualized models applied to neuroimaging data can better account for the subject-specific organization of domain-general cognitive systems in the human brain. Finally, the accumulation of functional brain signatures brings the possibility to clarify relationships among tasks and create a univocal link between brain systems and mental functions through: (1) the development of ontologies proposing an organization of cognitive processes; and (2) brain-network taxonomies describing functional specialization. To this end, tools to improve commensurability in cognitive science are necessary, such as public repositories, ontology-based platforms and automated meta-analysis tools. I will thus discuss some brain-atlasing resources currently under development, and their applicability in cognitive as well as clinical neuroscience.
This presentation explores a brief idea about the structural and functional attributes of nucleotides, the structure and function of genetic materials along with the impact of UV rays and pH upon them.
Nucleic Acid-its structural and functional complexity.
Enabling open and reproducible computer systems research: the good, the bad and the ugly
1. Enabling open and reproducible research
at computer systems' conferences
the good, the bad and the ugly
CNRS webinarCNRS webinar
GrenobleGrenoble
March 2017March 2017
GrigoriGrigori FursinFursin
Chief Scientist, cTuning foundation, FranceChief Scientist, cTuning foundation, France
CTO,CTO, dividitidividiti, UK, UK
fursin.net / researchfursin.net / research
cTuning.org /cTuning.org / aeae
2. Grigori Fursin “Enabling open and reproducible research at computer systems' conferences ( cKnowledge.org )” 22
• What is computer systems research?
• Major problems in computer systems’ research in the past 15 years
• Artifact Evaluation Initiative
• Good: active community participation with ACM/industrial support
• Bad: software and hardware chaos; highly stochastic behaviour
• Ugly: ad-hoc, non-portable scripts difficult to customize and reuse
• Improving Artifact Evaluation
• Preparing common replication/reproducibility methodology
(new ACM taskforce on reproducibility)
• Introducing community-driven artifact and paper reviewing
• Introducing common workflow framework (Collective Knowledge)
• Introducing simple JSON API and meta for artifacts
• Introducing portable and customizable package manager
• Demonstrating open and reproducible computer systems’ research
• Collaboratively optimizing deep learning across diverse datasets/SW/HW
• Conclusions and call for action!
Seminar outline
3. Grigori Fursin “Enabling open and reproducible research at computer systems' conferences ( cKnowledge.org )” 33
Computer System
ResultResult
Back to basics: what is computer systems’ research?
Users of computer systems (researchers, engineers,
entrepreneurs) want to quickly prototype their algorithms
or develop efficient, reliable and cheap products
4. Grigori Fursin “Enabling open and reproducible research at computer systems' conferences ( cKnowledge.org )” 44
ResultResult
ProgramProgram
CompilersCompilers
Binary and librariesBinary and libraries
Hardware,Hardware,
Simulators
Run-time environmentRun-time environment
State of the systemState of the system Data setData set
AlgorithmAlgorithm
Back to basics: what is computer systems’ research?
Computer systems’ researchers,
software developers and hardware
designers help end-users
improve all characteristics of their tasks
(execution time, power consumption,
numerical instability, size, faults, price …)
guarantee real-time constraints
(bandwidth, QoS, etc)
StorageStorage
Users of computer systems (researchers, engineers,
entrepreneurs) want to quickly prototype their algorithms
or develop efficient, reliable and cheap products
5. Grigori Fursin “Enabling open and reproducible research at computer systems' conferences ( cKnowledge.org )” 55
ResultResult
ProgramProgram
CompilersCompilers
Binary and librariesBinary and libraries
Hardware,Hardware,
Simulators
Run-time environmentRun-time environment
State of the systemState of the system Data setData set
AlgorithmAlgorithm
Two major problems: raising complexity and physical limitations
StorageStorage
Finding efficient, reliable and cheap solution
for end-user tasks is very non-trivial!
Thousands of benchmarks real applications
MPI, OpenMP, TBB, CUDA, OpenCL, StarPU, OmpSs
C,C++,Fortran,Java,Python,assembler
LLVM,GCC,ICC,PGI (hundreds of optimizations)
BLAS,MAGMA,ViennaCL,cuBLAS,clBLAST,cuDNN,
openBLAS, clBLAS
TensorFlow, Caffe, Torch, TensorRT
Infinite number of possible data sets
Linux, Windows, Android, MacOS
heterogeneous, many-core, out-of-order, cache
x86, ARM, PTX, NN, extensions
Numerous architecture and platform simulators
Too many design and optimization choices!
6. Grigori Fursin “Enabling open and reproducible research at computer systems' conferences ( cKnowledge.org )” 66
What are you paying for?
Smart and powerful mobile devices
are everywhere (mobile phones, IoT)
Weather prediction;
physics; medicine; finances
Unchanged algorithms may run
only a fraction of peak
performance thus wasting
expensive resources and energy!
Some years later you may get
more efficient algorithms
just before new systems arrives!
Various SW/HW optimizations
may result in
7x speedups, 5x energy savings,
but poor accuracy
2x speedups without
sacrificing accuracy –
enough to enable RT processing
Users expect new platforms to be faster, more energy efficient
more accurate and more reliable – is it true?
New systems require further tedious, ad-hoc and error-prone optimization.
This slows down innovation and development of new products.
Mobiledevices,sensors,IoTHPCplatforms
Many attempts to run DNN,
HOG, Slam algorithms
Supercomputers and data
centers cost millions of $
to build, install and use
7. Grigori Fursin “Enabling open and reproducible research at computer systems' conferences ( cKnowledge.org )” 77
How can we solve these problems?
What did I learn from other sciences that deal with complex systems:
physics, mathematics, chemistry, biology, AI …?
Major breakthroughs came from collaborative and reproducible R&D based on
sharing, validation, systematization
and reuse of artifacts and knowledge!
statistical analysis,
data mining,
machine learning
My original background is NOT in computer engineering
but in physics, electronics and machine learning
(first research project in 1994 to develop artificial neural networks chip)
8. Grigori Fursin “Enabling open and reproducible research at computer systems' conferences ( cKnowledge.org )” 88
ResultResult
Idea
cTuning.org (2008-cur) – collaborative, machine learning-based optimization
cTuning1 framework and public portal to
1) share realistic benchmarks and data sets
2) share whole experimental setups
(benchmarking and autotuning)
3) crowdsource empirical optimization
4) collect results in a centralized repository
5) apply machine learning to predict
optimizations
6) involve the community to improve models
• G. Fursin et.al. MILEPOST GCC: Machine learning based self-tuning compiler. 2008, 2011
• G. Fursin. Collective Tuning Initiative: automating and accelerating development and
optimization of computing systems, 2009
ProgramProgram
CompilersCompilers
Binary and librariesBinary and libraries
Hardware,Hardware,
Simulators
Run-time environmentRun-time environment
State of the systemState of the system Data setData set
AlgorithmAlgorithm
StorageStorage
9. Grigori Fursin “Enabling open and reproducible research at computer systems' conferences ( cKnowledge.org )” 99
cTuning.org/ae (2014-cur.) – artifact evaluation at CGO,PPoPP,PACT,SC
“Artifact Evaluation for Publications” (Dagstuhl Perspectives Workshop 15452), 2016,
Bruce R. Childers, Grigori Fursin, Shriram Krishnamurthi, Andreas Zeller,
http://drops.dagstuhl.de/opus/volltexte/2016/5762
Numerous publications My personal AE goals
• validate experimental results from
published articles and restore trust
(see ACM TRUST’14 @ PLDI workshop:
cTuning.org/event/acm-trust2014)
• promote artifact sharing (benchmarks,
data sets, tools, models)
• enable fair comparison of results and
techniques
• develop common methodology for
reproducible computer systems’ research
• build upon others’ research
10. Grigori Fursin “Enabling open and reproducible research at computer systems' conferences ( cKnowledge.org )” 1010
How Artifact Evaluation (AE) works?
Authors of accepted articles has an
option to submit related material for
an AE committee to be evaluated
http://cTuning.org/ae/submission.html
keep track of a methodology)
http://cTuning.org/ae/submission.html
• Abstract
• Packed artifact (or remote access)
• Artifact Appendix (with a version to
keep track of a methodology)
http://ctuning.org/ae/reviewing.html
Formalized reviewing process
Multiple criteria for artifact evaluation
Artifact ranking:
1.
2.
3.
4.
5.
http://ctuning.org/ae/reviewing.html
Formalized reviewing process
Multiple criteria for artifact evaluation
Artifact ranking:
1. Significantly exceeded expectations
2. Exceeded expectations
3. Met expectations
4. Fell below expectations
5. Significantly fell below expectations
PC members nominate one or two
senior PhD students/engineers
for AE committee
Pool of
reviewers!
Paper with artifacts which
passed evaluation receives
AE badge
11. Grigori Fursin “Enabling open and reproducible research at computer systems' conferences ( cKnowledge.org )” 1111
Artifact evaluation timeline
time line
paper
accepted
paper
accepted
artifacts
submitted
artifacts
submitted
evaluatorevaluator
bidding
artifacts
assigned
artifacts
assigned
evaluationsevaluations
available
evaluationsevaluations
finalized
artifactartifact
decision
7..12 days
to prepare
artifacts
according to
guidelines:
cTuning.org/
submission.html
2..4 days
for evaluators
to bid on
artifacts
(according
to their
knowledge and
access to
required
SW/HW)
2 days
to assign
artifacts –
ensure at least
3 reviews
per artifact,
reduce risks,
avoid mix ups
minimize
conflicts of
interests
2 weeks
to review
artifacts
according to
guidelines:
cTuning.org/
reviewing.html
3..4 days
for authors to
respond to
reviews and fix
problems
2..3 days
to finalize
reviews
Light communication between
authors and reviewers
is allowed via AE chairs
(to preserve anonymity
of the reviewers)
2..3 days
to add
AE stamp
and AE appendix
to a camera-
ready paper
12. Grigori Fursin “Enabling open and reproducible research at computer systems' conferences ( cKnowledge.org )” 1212
Artifact Evaluation: good
Year PPoPP CGO PACT Total Problems Rejected
2015 10 8 18 7 2
2016 12 11 23 4 0
2016 5 5 2 0
2017 14 13 27 7 0
• Strong support from academia, industry and ACM
• Active participation in AE discussion sessions
• Lots of feedback
• Many interesting artifacts!
NOTE: we consider AE a cooperative process and try to help authors fix artifacts
and pass evaluation (particularly if artifacts will be open-sourced)
We use this practical experience (> 70 papers in the past 3 years!)
to continuously improve common experimental methodology
for reproducible computer systems’ research
13. Grigori Fursin “Enabling open and reproducible research at computer systems' conferences ( cKnowledge.org )” 1313
In 2016 ACM organized a special taskforce (former AE chairs) to develop
common methodology for artifact sharing and evaluation across all SIGS!
We produced “Result and Artifact Review and Badging” policy:
http://www.acm.org/publications/policies/artifact-review-badging
1) Define terminology
Repeatability (Same team, same experimental setup)
Replicability (Different team, same experimental setup)
Reproducibility (Different team, different experimental setup)
2) Prepare new sets of badges (covering various SIGs)
Artifacts Evaluated – Functional
Artifacts Evaluated – Reusable
Artifacts Available
Results Replicated
Results Reproduced
ACM taskforce on reproducibility
14. Grigori Fursin “Enabling open and reproducible research at computer systems' conferences ( cKnowledge.org )” 1414
We introduced Artifact Appendices to describe experiments
Two years ago we introduced Artifact Appendix templates to unify
Artifact submissions and let authors add up to two pages of such
appendices to their camera ready paper:
http://cTuning.org/ae/submission.html
http://cTuning.org/ae/submission_extra.html
The idea is to help readers better understand what was evaluated
and let them reproduce published research and build upon it.
We did not receive complaints about our appendices and many
researchers decided to add them to their camera ready papers
(see http://cTuning.org/ae/artifacts.html).
Similar AE appendices are now used by other conferences
(SC,RTSS):
http://sc17.supercomputing.org/submitters/technical-
papers/reproducibility-initiatives-for-technical-papers/artifact-description-
paper-title
We are now trying to unify Artifact Appendices across all SIGs
15. Grigori Fursin “Enabling open and reproducible research at computer systems' conferences ( cKnowledge.org )” 1515
Artifact Evaluation: bad
• too many artifacts to evaluate – need to somehow scale AE while keeping the
quality (41 evaluators, ~120 reviews to handle during 2.5 weeks)
• difficult to find evaluators with appropriate skills and access
to proprietary SW and rare HW
• very intense schedule and not enough time for rebuttals
• communication between authors and reviewers via AE chairs is a bottleneck
time line
paper
accepted
paper
accepted
artifacts
submitted
artifacts
submitted
evaluatorevaluator
bidding
artifacts
assigned
artifacts
assigned
evaluationsevaluations
available
evaluationsevaluations
finalized
artifactartifact
decision
7..12 days 2..4 days 2 days 2 weeks 3..4 days 2..3 days 2..3 days
16. Grigori Fursin “Enabling open and reproducible research at computer systems' conferences ( cKnowledge.org )” 1616
New option of open artifact evaluation!
time line
paper
accepted
paper
accepted
artifacts
submitted
artifacts
submitted
evaluatorevaluator
bidding
artifacts
assigned
artifacts
assigned
evaluationsevaluations
available
evaluationsevaluations
finalized
artifactartifact
decision
7..12 days 2..4 days 2 days 2 weeks 3..4 days 2..3 days 2..3 days
Introduce two evaluation options: private and public
a) traditional evaluation for private artifacts (for example, from industry, though less and less common)
time line
paper
accepted
paper
accepted
artifacts
submitted
artifacts
submitted
AE chairs announce
public artifacts at
XSEDE/GRID5000/etc
AE chairs announce
public artifacts at
XSEDE/GRID5000/etc
AE chairs monitor open
artifacts are evaluated
AE chairs monitor open
discussions until
artifacts are evaluated
artifactartifact
decision
any time 1..2 days from a few days to 2 weeks 3..4 days 2..3 days
b) open evaluation of public and open-source artifacts (if already avialable at
GitHub, BitBucket, GitLab with “discussion mechanisms” during submission…)
17. Grigori Fursin “Enabling open and reproducible research at computer systems' conferences ( cKnowledge.org )” 1717
Trying open artifact and paper evaluation
At CGO/PPoPP’17, we have sent out requests to validate several open-source artifacts to the public
mailing lists from the conferences, network of excellence, supercomputer centers, etc.
We found evaluators willing to help and having an access to rare hardware or supercomputers
as well as required software and proprietary benchmarks
Authors quickly fixed issues and answered research questions while AE chairs steered the discussion!
GRID5000 users participated in open evaluation of a PPoPP’17 artifact:
https://github.com/thu-pacman/self-checkpoint/issues/1
See other public evaluation examples:
cTuning.org/ae/artifacts.html
We validated open reviewing of publications via Reddit at ADAPT’16:
http://adapt-workshop.org
18. Grigori Fursin “Enabling open and reproducible research at computer systems' conferences ( cKnowledge.org )” 1818
Artifact Evaluation: what can possibly be ugly ;) ?
Algorithm
ProgramProgram
CompilersCompilers
Binary and librariesBinary and libraries
Hardware,Hardware,
Simulators
Run-time environmentRun-time environment
State of the systemState of the system Data setData set
AlgorithmAlgorithm
StorageStorage
ResultResult
19. Grigori Fursin “Enabling open and reproducible research at computer systems' conferences ( cKnowledge.org )” 1919
Ugly: no common experimental methodology and SW/HW chaos
• difficult (sometimes impossible) to reproduce
empirical results across ever changing
software and hardware stack
(highly stochastic behavior)
• everyone uses their own ad-hoc scripts to
prepare and run experiments with many
hardwired paths
• practically impossible to customize and reuse
artifacts (for example, try another compiler,
library, data set)
• practically impossible to run on another OS
or platform
• no common API and meta information for
shared artifacts and results
(benchmarks, data sets, tools)ResultResult
Algorithm
GCC 4.1.x
GCC 4.2.x
GCC 4.3.x
GCC 4.4.x
GCC 4.5.x
GCC 4.6.x
GCC 4.7.x
ICC 10.1
ICC 11.0
ICC 11.1
ICC 12.0
ICC 12.1
LLVM 2.6
LLVM 2.7
LLVM 3.7
LLVM 2.9
LLVM 3.0
Phoenix
MVS 2013
XLC
Open64
Jikes
Testarossa
OpenMP MPI
HMPP
OpenCL
CUDA 4.x
gprofprof
perf
oprofile
PAPI
TAU
Scalasca
VTune
Amplifier
predictive
scheduling
algorithm-
level
TBB
MKL
ATLAS
program-
level
function-
level
Codelet
loop-level
hardware
counters
IPA
polyhedral
transformations
LTO
pass
reordering
per phase
reconfiguration
frequency
bandwidth
HDD size
TLB
memory size
execution time
GCC 5.2
LLVM 3.4
SVM
ARM v8
Intel SandyBridge
SSE4
AVX
CUDA 7.x
SimpleScalar
algorithm precision
20. Joint CGO/PPoPP Distinguished Artifact Award
Xiuxia Zhang1, Guangming Tan1, Shuangbai Xue1, Jiajia Li2, Mingyu Chen1
1 Chinese Academy of Sciences 2 Georgia Institute of Technology
February 2017
for
“Demystifying GPU Microarchitecture
to Tune SGEMM Performance”
The cTuning foundation and
NVIDIA are pleased to present
Awarding distinguished artifactsAwarding distinguished artifacts ––not enough!not enough!
21. Grigori Fursin “Enabling open and reproducible research at computer systems' conferences ( cKnowledge.org )” 2121
ResultResult
VM
Using Virtual Machines and Docker
GCC 4.1.x
GCC 4.2.x
GCC 4.3.x
GCC 4.4.x
GCC 4.5.x
GCC 4.6.x
GCC 4.7.x
ICC 10.1
ICC 11.0
ICC 11.1
ICC 12.0
ICC 12.1
LLVM 2.6
LLVM 2.7
LLVM 3.7
LLVM 2.9
LLVM 3.0
Phoenix
MVS 2013
XLC
Open64
Jikes
Testarossa
OpenMP MPI
HMPP
OpenCL
CUDA 4.x
gprofprof
perf
oprofile
PAPI
TAU
Scalasca
VTune
Amplifier
predictive
scheduling
algorithm-
level
TBB
MKL
ATLAS
program-
level
function-
level
Codelet
loop-level
hardware
counters
IPA
polyhedral
transformations
LTO
pass
reordering
per phase
reconfiguration
bandwidth
HDD size
TLB
memory size
execution time
GCC 5.2
LLVM 3.4
SVM
ARM v8
Intel SandyBridge
SSE4
AVX
CUDA 7.x
SimpleScalar
algorithm precision
VM or Docker images
good at hiding complexity
but they do not solve
major problems
in computer systems’ research
Docker is useful to archive artifacts
while hiding all underlying complexity!
However Docker does not address many other
issues vital for open and collaborative
computer systems’ research, i.e. how to
1) work with a native user SW/HW environment
2) customize and reuse artifacts and workflows
3) capture run-time state
4) deal with hardware dependencies
5) deal with proprietary benchmarks and tools
6) automate validation of experiments
Images are also often too large in size!
Need portable and customizable
workflow framework suitable for
computer systems’ research!
22. Grigori Fursin “Enabling open and reproducible research at computer systems' conferences ( cKnowledge.org )” 2222
Some attempts to clean up this mess in computer systems’ R&D
ResultResult
Algorithm
GCC 4.1.x
GCC 4.2.x
GCC 4.3.x
GCC 4.4.x
GCC 4.5.x
GCC 4.6.x
GCC 4.7.x
ICC 10.1
ICC 11.0
ICC 11.1
ICC 12.0
ICC 12.1
LLVM 2.6
LLVM 2.7
LLVM 3.7
LLVM 2.9
LLVM 3.0
Phoenix
MVS 2013
XLC
Open64
Jikes
Testarossa
OpenMP MPI
HMPP
OpenCL
CUDA 4.x
gprofprof
perf
oprofile
PAPI
TAU
Scalasca
VTune
Amplifier
predictive
scheduling
algorithm-
level
TBB
MKL
ATLAS
program-
level
function-
level
Codelet
loop-level
hardware
counters
IPA
polyhedral
transformations
LTO
pass
reordering
per phase
reconfiguration
frequency
bandwidth
HDD size
TLB
memory size
execution time
GCC 5.2
LLVM 3.4
SVM
ARM v8
Intel SandyBridge
SSE4
AVX
CUDA 7.x
SimpleScalar
algorithm precision
Better package managersBetter package managers
apt; yum; spack;
pip ; homebrew …
Smart build toolsSmart build tools
cmake; easybuild …
Multi-versioningMulti-versioning
spack; easybuild; virtualenv …
Third-party resources
cKnowledge.org/reproducibility
Numerous online projects
But we want to systematize our local artifacts
Numerous online projects
But we want to systematize our local artifacts
without being locked up on third-party
web services or learn complex GUI
Missing: portable and customizable workflow
framework suitable for computer systems’ research!
23. Grigori Fursin “Enabling open and reproducible research at computer systems' conferences ( cKnowledge.org )” 2323
Open-source Collective Knowledge Framework (2015-cur.)
ResultResult
Algorithm
GCC 4.1.x
GCC 4.2.x
GCC 4.3.x
GCC 4.4.x
GCC 4.5.x
GCC 4.6.x
GCC 4.7.x
ICC 10.1
ICC 11.0
ICC 11.1
ICC 12.0
ICC 12.1
LLVM 2.6
LLVM 2.7
LLVM 3.7
LLVM 2.9
LLVM 3.0
Phoenix
MVS 2013
XLC
Open64
Jikes
Testarossa
OpenMP MPI
HMPP
OpenCL
CUDA 4.x
gprofprof
perf
oprofile
PAPI
TAU
Scalasca
VTune
Amplifier
predictive
scheduling
algorithm-
level
TBB
MKL
ATLAS
program-
level
function-
level
Codelet
loop-level
hardware
counters
IPA
polyhedral
transformations
LTO
pass
reordering
per phase
reconfiguration
frequency
bandwidth
HDD size
TLB
memory size
execution time
GCC 5.2
LLVM 3.4
SVM
ARM v8
Intel SandyBridge
SSE4
AVX
CUDA 7.x
SimpleScalar
algorithm precision
cKnowledge.org ; github.com/ctuning/ck
Eventually we didn’t have a choices but to use all
our experience and develop portable and
customizable workflow framework
for computer systems’ research:
1) organize your artifacts (programs, data sets,
tools, scripts) as customizable and reusable
components with JSON API and meta data
2) assemble portable and customizable
workflows with JSON API from shared artifacts
as LEGO ™
3) integrate portable package and environment
manager which can detect multiple versions
of required software or installed one across
Linux, MacOS, Windows and Android
4) integrate web server to show interactive
reports (locally!) or exchange data when
crowdsourcing experiments
24. Grigori Fursin “Enabling open and reproducible research at computer systems' conferences ( cKnowledge.org )” 2424
ResultResult
ProgramProgram
CompilerCompiler
Binary and librariesBinary and libraries
ArchitectureArchitecture
Run-time environmentRun-time environment
AlgorithmAlgorithm
Idea
Noticed during AE: all projects have similar structure
Data setData setState of the systemState of the system
image corner detectionimage corner detection
matmul OpenCLmatmul OpenCL
compressioncompression
neural network CUDAneural network CUDA
Ad-hoc scripts to
compile and run a
program…
Have some
common meta:
which datasets
can use, how to
compile, CMD, …
image-jpeg-0001image-jpeg-0001
bzip2-0006bzip2-0006
txt-0012txt-0012
video-raw-1280x1024video-raw-1280x1024
Ad-hoc dirs for data
sets with some ad-hoc
scripts to find them,
extract features, etc
Have some
(common)
meta:
filename, size,
width, height,
colors, …
Ad-hoc scripts to set
up environment for
a given and possibly
proprietary compiler
Have some
common meta:
compilation,
linking and
optimization
flags
Create project directory
Ad-hoc dirs and
scripts to record
and analyze
experiments
cvs speedupscvs speedups
txt hardware counterstxt hardware counters
xls table with graphxls table with graph
Have some
common meta:
features,
characteristics,
optimizations
GCC V5.2GCC V5.2
LLVM 3.6LLVM 3.6
Intel Compilers 2015Intel Compilers 2015
25. Grigori Fursin “Enabling open and reproducible research at computer systems' conferences ( cKnowledge.org )” 2525
image corner detectionimage corner detection
matmul OpenCLmatmul OpenCL
compressioncompression
neural network CUDAneural network CUDA
meta.json
image-jpeg-0001image-jpeg-0001
bzip2-0006bzip2-0006
txt-0012txt-0012
video-raw-1280x1024video-raw-1280x1024
meta.json
meta.json
meta.json
GCC V5.2GCC V5.2
LLVM 3.6LLVM 3.6
Intel Compilers 2015Intel Compilers 2015
Python module
compile and run
Python module
“program”
with functions:
compile and run
Python module
setup
Python module
“soft”
with function:
setup
Python module
extract_features
Python module
“dataset”
with function:
extract_features
Python module
add, get, analyze
Python module
“experiment”
with function:
add, get, analyze
Collective Knowledge: organize and share artifacts as reusable components
data UID and alias
cvs speedupscvs speedups
txt hardware counterstxt hardware counters
xls table with graphxls table with graph
26. Grigori Fursin “Enabling open and reproducible research at computer systems' conferences ( cKnowledge.org )” 2626
image corner detectionimage corner detection
matmul OpenCLmatmul OpenCL
compressioncompression
neural network CUDAneural network CUDA
image-jpeg-0001image-jpeg-0001
bzip2-0006bzip2-0006
txt-0012txt-0012
video-raw-1280x1024video-raw-1280x1024
GCC V5.2GCC V5.2
LLVM 3.6LLVM 3.6
Intel Compilers 2015Intel Compilers 2015
Python module
compile and run
Python module
“program”
with functions:
compile and run
Python module
setup
Python module
“soft”
with function:
setup
Python module
extract_features
Python module
“dataset”
with function:
extract_features
Python module
add, get, analyze
Python module
“experiment”
with function:
add, get, analyze
Collective Knowledge: organize and share your artifacts as reusable components
data UID and alias
JSON
input
JSON
input
JSON
input
JSON
input
JSON
output
JSON
output
JSON
output
JSON
output
CK: small python module (~200Kb); no extra dependencies; Linux; Win; MacOS
ck <function> <module>:<data UID> @input.json
meta.json
meta.json
meta.json
meta.json
cvs speedupscvs speedups
txt hardware counterstxt hardware counters
xls table with graphxls table with graph
27. Grigori Fursin “Enabling open and reproducible research at computer systems' conferences ( cKnowledge.org )” 2727
image corner detectionimage corner detection
matmul OpenCLmatmul OpenCL
compressioncompression
neural network CUDAneural network CUDA
image-jpeg-0001image-jpeg-0001
bzip2-0006bzip2-0006
txt-0012txt-0012
video-raw-1280x1024video-raw-1280x1024
GCC V5.2GCC V5.2
LLVM 3.6LLVM 3.6
Intel Compilers 2015Intel Compilers 2015
Python module
compile and run
Python module
“program”
with functions:
compile and run
Python module
setup
Python module
“soft”
with function:
setup
Python module
extract_features
Python module
“dataset”
with function:
extract_features
Python module
add, get, analyze
Python module
“experiment”
with function:
add, get, analyze
Collective Knowledge: organize and share your artifacts as reusable components
data UID and alias
JSON
input
JSON
output
CK: small python module (~200Kb); no extra dependencies; Linux; Win; MacOS
Connect into workflows as LEGO™
meta.json
meta.json
meta.json
meta.json
cvs speedupscvs speedups
txt hardware counterstxt hardware counters
xls table with graphxls table with graph
28. Grigori Fursin “Enabling open and reproducible research at computer systems' conferences ( cKnowledge.org )” 2828
Local repository structure (with internal UID and meta)!
program
soft
image corner detectionimage corner detection
matmul OpenCLmatmul OpenCL
compressioncompression
neural network CUDAneural network CUDA
gcc 5.2gcc 5.2
llvm 3.6llvm 3.6
icc 2015icc 2015
dataset image-jpeg-0001image-jpeg-0001
bzip2-0006bzip2-0006
video-raw-1280x1024video-raw-1280x1024
……
……
……
……
……
……
module programprogram
softsoft
datasetdataset
……
……
……
/ module UID and alias / data UID or alias / .cm / meta.json
CK
local
project
repo
experimentexperiment ……
……
……
……
……
Both code (with API) and data (with meta) inside repository
Can be referenced and cross-linked via CID (similar to DOI):
module UOA : data UOA
Local files –
no locks up on
third-party
web services
No complex GUI
Can be shared
via GIT/SVN/etc
Can be easily
indexed via
ElasticSearch to
speed up search
Can be easily
connected to
Python-based
predictive
analytics
(sklearn-kit)
29. Grigori Fursin “Enabling open and reproducible research at computer systems' conferences ( cKnowledge.org )” 2929
Provide customizable and SW/HW independent access to tools
CLBLast
OpenBLASOpenBLAS
LLVM 3.8LLVM 3.8
GCC 7.0
CUDA 8.0CUDA 8.0 cuBLAS “Black box”“Black box”
“big data”
database
Hardwired
experiment
scripts /
cmake
Ad-hoc
scripts to
process CSV,
XLS, TXT, etc.Caffe OpenCLCaffe OpenCL
Caffe CUDACaffe CUDA
Caffe CPU
JSONAPI(input)JSONAPI(input)
Source files and auxiliary scriptsSource files and auxiliary scripts
CK program entry (ck list program)CK program entry (ck list program)
$ ck pull repo:ck-autotuning
$ ck compile program:cbench-automotive-susan –speed
$ ck run program:cbench-automotive-susan
$ ck run program:caffe
Implement SW/HW independent, reusable
and shareable CK modules with unified CMD
and JSON API for various experimental
scenarios (such as program module)
.cm/meta.json – describes soft dependencies ,
data sets, and how to compile and run this program
.cm/meta.json – describes soft dependencies ,
data sets, and how to compile and run this program
Read
program
Read
program
meta
Detect all softwareDetect all software
dependencies; ask user if
multiple versions exists
Prepare
environment
CompileCompile
program
Run
program
JSONAPI(output)JSONAPI(output)
CK program module
Use program entries to describe a given program
in meta.json and store sources (if needed)
Ad-hoc experimental workflows
30. Grigori Fursin “Enabling open and reproducible research at computer systems' conferences ( cKnowledge.org )” 3030
local / env / 03ca0be16962f471 / env.sh
Tags: compiler,cuda,v8.0
local / env / 03ca0be16962f471 / env.sh
Tags: compiler,cuda,v8.0
local / env / 0a5ba198d48e3af3 / env.bat
Tags: lib,blas,cublas,v8.0
local / env / 0a5ba198d48e3af3 / env.bat
Tags: lib,blas,cublas,v8.0
Soft entries in CK describe how
to detect if a given software is
already installed, how to set up
all its environment including
all paths (to binaries, libraries,
include, aux tools, etc),
and how to detect its version.
$ ck detect soft --tags=compiler,cuda$ ck detect soft --tags=compiler,cuda
$ ck detect soft:compiler.gcc$ ck detect soft:compiler.gcc
$ ck detect soft:compiler.llvm --target_os=android19-arm$ ck detect soft:compiler.llvm --target_os=android19-arm
$ ck list soft:compiler*$ ck list soft:compiler*
$ ck detect soft:lib.cublas$ ck detect soft:lib.cublas
Env entries are created in CK local
repo for all found software
instances together with their meta
and an auto-generated environment
script env.sh (on Linux) or env.bat
(on Windows).
Package entries describe how to
install a given software if it is not
installed (using install.sh script on
Linux host or install.bat on
Windows host).
$ ck install package:caffemodel-bvlc-googlenet$ ck install package:caffemodel-bvlc-googlenet
$ ck install package:imagenet-2012-val$ ck install package:imagenet-2012-val
$ ck install package:lib-caffe-bvlc-master-cuda$ ck install package:lib-caffe-bvlc-master-cuda
$ ck list package:*caffemodel*$ ck list package:*caffemodel*
LocalCKrepoLocalCKrepo
Portable and customizable package manager in the CK
$ ck search soft --tags=blas$ ck search soft --tags=blas
$ ck show env$ ck show env
$ ck show env –tags=cublas$ ck show env –tags=cublas
$ ck rm env:* –tags=cublas$ ck rm env:* –tags=cublas
$ ck search package –tags=caffe$ ck search package –tags=caffe
Works on Linux, Windows, MacOS, Android
Detects multiple versions of various software
Builds missing software
Extended by the community
github.com/ctuning/ck-env
31. Grigori Fursin “Enabling open and reproducible research at computer systems' conferences ( cKnowledge.org )” 3131
Customizable and reusable workflows with JSON API as LEGO™
local / env / 0a5ba198d48e3af3 / env.bat
Tags: lib,blas,cublas,v8.0
local / env / 0a5ba198d48e3af3 / env.bat
Tags: lib,blas,cublas,v8.0
Chaining CK modules to an experimental pipeline via JSON API to quickly prototype research ideas
Public modular auto-tuning and machine
learning repository and buildbot
Unified
web services Interdisciplinary crowd
Choose
exploration
strategy
Select choices (program,
data set, compiler, flags,
opts, hardware design …)
Compile
program
or kernel
Run
program
or kernel
Perform
statistical
analysis
Apply
Pareto
filter
Model and
predict choices
and behavior
Reduce
complexity
Shared scenarios from past research
…
Expose and unify information needed for performance
analysis and optimization combined with data mining!
Optimizing/modeling behavior b
of any object in the CK (program, library function, kernel, …)
as a function of design and optimization choices c, features f
and run-time state s
b = B( c , f , s )
… … … …
Flattened JSON vectors
32. Grigori Fursin “Enabling open and reproducible research at computer systems' conferences ( cKnowledge.org )” 3232
Gradually add JSON specification (depends on research scenario)
Autotuning and machine learning specification:
{
"characteristics":{
"execution times": ["10.3","10.1","13.3"],
"code size": "131938", ...},
"choices":{
"os":"linux", "os version":"2.6.32-5-amd64",
"compiler":"gcc", "compiler version":"4.6.3",
"compiler_flags":"-O3 -fno-if-conversion",
"platform":{"processor":"intel xeon e5520",
"l2":"8192“, ...}, ...},
"features":{
"semantic features": {"number_of_bb": "24", ...},
"hardware counters": {"cpi": "1.4" ...}, ... }
"state":{
"frequency":"2.27", ...}
}
CK flattened JSON key
##characteristics#execution_times@1
"flattened_json_key”:{
"type": "text”|"integer" | “float" | "dict" | "list”
| "uid",
"characteristic": "yes" | "no",
"feature": "yes" | "no",
"state": "yes" | "no",
"has_choice": "yes“ | "no",
"choices": [ list of strings if categorical
choice],
"explore_start": "start number if numerical
range",
"explore_stop": "stop number if numerical
range",
"explore_step": "step if numerical range",
"can_be_omitted" : "yes" | "no"
...
}
33. Grigori Fursin “Enabling open and reproducible research at computer systems' conferences ( cKnowledge.org )” 3333
• "Collective Mind: Towards practical and collaborative autotuning“,
Journal of Scientific Programming 22 (4), 2014
http://hal.inria.fr/hal-01054763
• “Collective Mind, Part II: Towards Performance- and Cost-Aware
Software Engineering as a Natural Science”, CPC 2015, London, UK
http://arxiv.org/abs/1506.06256
• Android application to crowdsource autotuning across mobile devices:
https://play.google.com/store/apps/details?id=openscience.crowdsource.experiments
• “Collective Knowledge: towards R&D sustainability”, DATE 2016, Dresden, Germany
http://bit.ly/ck-date16
• “Optimizing convolutional neural networks on embedded platforms with OpenCL”,
IWOCL 2016, Amsterdam
http://dl.acm.org/citation.cfm?id=2909449
The community now use CK to collaboratively tackle old problems
Crowdsource performance analysis and optimization, machine-learning,
run-time adaptation across diverse workloads and hardware provided
by volunteers
$ sudo pip install ck
$ ck pull repo:ck-crowdtuning
$ ck crowdtune program --gcc
34. Grigori Fursin “Enabling open and reproducible research at computer systems' conferences ( cKnowledge.org )” 3434
Reproducibility came as a side effect!
• Can preserve the whole experimental setup with all data and software dependencies
• Can perform statistical analysis for characteristics
• Community can add missing features or improve machine learning models
Execution time:
10 sec.
Reproducibility of experimental results as a side effect
35. Grigori Fursin “Enabling open and reproducible research at computer systems' conferences ( cKnowledge.org )” 3535
Reproducibility came as a side effect!
• Can preserve the whole experimental setup with all data and software dependencies
• Can perform statistical analysis for characteristics
• Community can add missing features or improve machine learning models
Variation of experimental results:
10 ± 5 secs.
Reproducibility of experimental results as a side effect
36. Grigori Fursin “Enabling open and reproducible research at computer systems' conferences ( cKnowledge.org )” 3636
Execution time (sec.)
Distribution
Unexpected behavior - expose to the community including experts
to explain, find missing feature and add to the system
Reproducibility of experimental results as a side effect
Reproducibility came as a side effect!
• Can preserve the whole experimental setup with all data and software dependencies
• Can perform statistical analysis for characteristics
• Community can add missing features or improve machine learning models
37. Grigori Fursin “Enabling open and reproducible research at computer systems' conferences ( cKnowledge.org )” 3737
Execution time (sec.)
Distribution
Class A Class B
800MHz CPU Frequency 2400MHz
Unexpected behavior - expose to the community including experts
to explain, find missing feature and add to the system
Reproducibility of experimental results as a side effect
Reproducibility came as a side effect!
• Can preserve the whole experimental setup with all data and software dependencies
• Can perform statistical analysis for characteristics
• Community can add missing features or improve machine learning models
38. Grigori Fursin “Enabling open and reproducible research at computer systems' conferences ( cKnowledge.org )” 3838
Gradually extend/fix shared workflows by the community during AE
•Init pipeline
•Detected system information
•Initialize parameters
•Prepare dataset
•Clean program
•Prepare compiler flags
•Use compiler profiling
•Use cTuning CC/MILEPOST GCC for fine-grain program analysis and tuning
•Use universal Alchemist plugin (with any OpenME-compatible compiler or tool)
•Use Alchemist plugin (currently for GCC)
•Compile program
•Get objdump and md5sum (if supported)
•Use OpenME for fine-grain program analysis and online tuning (build & run)
•Use 'Intel VTune Amplifier' to collect hardware counters
•Use 'perf' to collect hardware counters
•Set frequency (in Unix, if supported)
•Get system state before execution
•Run program
•Check output for correctness (use dataset UID to save different outputs)
•Finish OpenME
•Misc info
•Observed characteristics
•Observed statistical characteristics
•Finalize pipeline
We’ve shared and continuously extend our
experimental workflow for autotuning:
http://github.com/ctuning/ck-autotuning
http://cknowledge.org/repo
http://github.com/ctuning
• Hundreds of benchmarks/kernels/codelets
(CPU, OpenMP, OpenCL, CUDA)
• Thousands of data sets
• Wrappers around all main tools and libs
• Optimization description of major compilers
39. Grigori Fursin “Enabling open and reproducible research at computer systems' conferences ( cKnowledge.org )” 3939
Distinguished CGO’17 artifact in Collective Knowledge format
Highest ranked artifact at CGO’17 turned out to be implemented
using Collective Knowledge Framework
"Software Prefetching for Indirect Memory Accesses",
Sam Ainsworth and Timothy M. Jones
https://github.com/SamAinsworth/reproduce-cgo2017-paper
It take advantage of a portable package manager
to install required LLVM for either x86 or ARM platforms,
automatically build LLVM plugins,
run empirical experiments on a user machine via CK workflow,
compare speedups with pre-recorded results by authors,
and prepare interactive report
See PDF with Artifact Evaluation Appendix:
http://ctuning.org/ae/resources/paper-with-distinguished-ck-artifact-and-ae-appendix-cgo2017.pdf
See snapshot of an interactive CK dashboard:
https://github.com/SamAinsworth/reproduce-cgo2017-paper/files/618737/ck-aarch64-dashboard.pdf
40. Grigori Fursin “Enabling open and reproducible research at computer systems' conferences ( cKnowledge.org )” 4040
Collective Knowledge mini-tutorials
Create repository: ck add repo:my_new_project
Add new module: ck add my_new_project:module:my_module
Add new data for this module: ck add my_new_project:my_module:my_data @@dict
{“tags”:”cool”,”data”}
Add dummy function to module: ck add_action my_module –func=my_func
Test dummy function: ck my_func my_module --param1=var1
List my_module data: ck list my_module
Find data by tags: ck search my_module –tags=cool
Archive repository: ck zip repo:my_new_project
Pull existing repo from GitHub: ck pull repo:ck-autotuning
List modules from this repo: ck list ck-autotuning:module:*
Compile program (using GCC): ck compile program: cbench-automotive-susan --speed
Run program: ck run program: cbench-automotive-susan
Start server for crowdsourcing: ck start web
View interactive articles: ck browser http://localhost:3344
https://michel-steuwer.github.io/About-CK/
https://github.com/ctuning/ck/wiki/Compiler-autotuning
http://github.com/ctuning/ck/wiki
41. Grigori Fursin “Enabling open and reproducible research at computer systems' conferences ( cKnowledge.org )” 4141
CK demo: workflow to collaboratively optimizing DNN
• “Deep” (multi-layered) neural networks that take advantage of
structure in digital signals (e.g. images).
• State-of-the-art for applications in automotive, robotics,
healthcare, entertainment (e.g. image classification).
• Commonly trained on workstations or clusters with GPGPUs.
• Increasingly deployed on mobile and embedded systems.
Difficult and time consuming to build and optimize
due to multiple programming models, multiple objectives
(performance, energy, accuracy, memory footprint)
and continuously evolving hardware
particularly with constrained resources (sensors, IoT)
Perfect use case for Collective Knowledge Framework!
42. Grigori Fursin “Enabling open and reproducible research at computer systems' conferences ( cKnowledge.org )” 4242
Deploying DNNs on resource-constrained platforms
• Can we deploy a DNN to achieve desired performance
(rate and accuracy of recognition) on a given platform?
• Can we identify or build such a platform under given
constraints such as those on power, memory, price?
• If all else fails, can we design (autotune) another DNN
(topology) by trading off performance, accuracy and costs?
NVIDIA Drive PX2 ~ 250 Watts ~ $15,000 (early
development boards)
~ $1,000 per unit?
? < 10 Watts < $100
43. Grigori Fursin “Enabling open and reproducible research at computer systems' conferences ( cKnowledge.org )” 4343
Customizable CK workflow to collaborative co-design DNN
We collaborate with General Motors and ARM to develop open-source
CK-based tools to collaboratively evaluate and optimize various DNN engines
and models across diverse platforms and datasets
cKnowledge.org/ai
cKnowledge.org/repo
Download Android app to participate in collaborative evaluation
and optimization of DNN using your mobile device and CK web-service:
https://play.google.com/store/apps/details?id=openscience.crowdsource.video.experiments
CK portable workflow systems and package manager allows users to run DNN
experiments on Windows, Linux, MacOS, Android!
$ ck pull repo --url=http://github.com/dividiti/ck-caffe
$ ck install package:lib-caffe-bvlc-master-cpu-universal
$ ck install package:lib-caffe-bvlc-opencl-viennacl-universal --target_os=android21-arm64
$ ck run program:caffe-classification
$ firefox http://cKnowledge.org/repo
44. Grigori Fursin “Enabling open and reproducible research at computer systems' conferences ( cKnowledge.org )” 4444
• Collective Knowledge approach helps fight SW/HW chaos
• CK is also changing the mentality of computer systems’ researchers:
• sharing artifacts as customizable and reusable components with JSON API
• sharing customizable and portable experimental workflows
• building a repository of representative benchmarks and data sets
• crowdsourcing experiments and sharing negative/unexpected results
• collaboratively improving reproducibility of empirical experiments
• using data mining (statistical analysis and machine learning) to predict
efficient software optimizations and hardware designs
• collaboratively improving prediction models and finding missing features
• formulating and solving important real-world problems
• CK brings closer together industry and academia (common research
methodology, reproducible experiments, validated techniques)
cKnowledge.org: from ad-hoc computer systems’ research to data science
45. Grigori Fursin “Enabling open and reproducible research at computer systems' conferences ( cKnowledge.org )” 4545
GCC 4.1.x
GCC 4.2.x
GCC 4.3.x
GCC 4.4.x
GCC 4.5.x
GCC 4.6.x
GCC 4.7.x
ICC 10.1
ICC 11.0
ICC 11.1
ICC 12.0
ICC 12.1
LLVM 2.6
LLVM 2.7
LLVM 2.8
LLVM 2.9
LLVM 3.1
Phoenix
MVS XLC
Open64
Jikes
Testarossa
OpenMP
MPI
HMPP
OpenCL
CUDA
gprof
prof
perf
oprofile
PAPI
TAU
Scalasca
VTune
Amplifier
scheduling
algorithm-level
TBB
MKL
ATLASprogram-level
function-level
Codelet
loop-level
hardware
counters
IPA
polyhedral
transformations
LTO
threads
process pass reordering
run-time adaptation
per phase
reconfiguration
cache size
frequency
bandwidth
HDD size
TLB
ISA
memory size
coresprocessors
threads
power consumption
execution time reliability
Current state of computer engineering
likwid
Classification,Classification,
predictive
modeling
Optimal
solutions
Systematization and unification
of collective knowledge
(big data)
“crowd”
The community
Task, idea
Result
Quick, non-reproducible hack?
Ad-hoc heuristic?
Quick publication?
Waste of expensive resources
and energy?
Collaboratively optimize shared
programs and data sets across
diverse hardware
Extrapolate shared knowledge to build fast, energy efficient, reliable and cheap
computer systems to boost innovation in science and technology!
Collective
Knowledge
open RDK
and repository
cKnowledge.org: enable open & reproducible computer systems’ research!
46. Grigori Fursin “Enabling open and reproducible research at computer systems' conferences ( cKnowledge.org )” 4646
Help us improve artifact evaluation!
We need your feedback - remember that new AE procedures
may affect you at the future conferences
• AE steering committee: http://cTuning.org/ae/committee.html
• Mailing list: https://groups.google.com/forum/#!forum/collective-knowledge
Feel free to reuse and improve our Artifact Evaluation procedures
and Artifact Appendices:
http://cTuning.org/ae https://github.com/ctuning/ck-web-artifact-evaluation
Extra resources
• ACM Result and Artifact Review and Badging policy:
http://www.acm.org/publications/policies/artifact-review-badging
• Evaluation of frameworks for reproducible R&D:
https://wilkie.github.io/reproducibility-site
• Community driven artifact evaluation and paper reviewing:
http://dl.acm.org/citation.cfm?doid=2618137.2618142
47. Grigori Fursin “Enabling open and reproducible research at computer systems' conferences ( cKnowledge.org )” 4747
Questions, comments, collaborations?
Many new exciting opportunities for collaboration
Artifact sharing, customizable and portable workflows, experiment crowdsourcing,
collaborative deep learning optimization, public repositores of knowledge, adaptive systems
Acknowledgments