From Provenance Standards and Tools to Queries and Actionable ProvenanceBertram Ludäscher
The document discusses computational provenance and the need for tracking data lineage and workflow processes. It presents several tools and projects that aim to capture and manage provenance information, including DataONE, SKOPE, KURATOR, WHOLE-TALE, and YesWorkflow. The document argues that provenance is important for understanding what happened in computational and data-driven research in order to ensure transparency and reproducibility.
This document provides an overview of machine learning concepts including:
1. Machine learning aims to create computer programs that improve with experience by learning from data. It involves tasks like classification, regression, and clustering.
2. Data comes in different types like text, numbers, images and is generated in massive quantities daily from sources like Google, Facebook, and sensors.
3. Machine learning algorithms are either supervised, using labeled training data, or unsupervised, using unlabeled data. Common supervised techniques are decision trees, neural networks, and support vector machines while clustering is a major unsupervised technique.
This document provides an overview of machine learning concepts. It defines machine learning as creating computer programs that improve with experience. Supervised learning uses labeled training data to build models that can classify or predict new examples, while unsupervised learning finds patterns in unlabeled data. Examples of machine learning applications include spam filtering, recommendation systems, and medical diagnosis. The document also discusses important machine learning techniques like k-nearest neighbors, decision trees, regularization, and cross-validation.
The document benchmarks 20 machine learning models on two datasets to compare their accuracy and speed. On the smaller Car Evaluation dataset, bagged decision trees, random forests and boosted decision trees achieved over 99% accuracy, while neural networks, decision stumps and support vector machines exceeded 95% accuracy. On the larger Nursery dataset, similar models exceeded 99% accuracy, while other models like decision rules and k-nearest neighbors exceeded 95% accuracy. However, models varied significantly in speed depending on the hardware, with decision trees, mixture discriminant analysis and gradient boosting as the fastest on Car Evaluation, and mixture discriminant analysis, one rule and boosted decision trees as the fastest on Nursery. The findings imply the importance of regular benchmarking
Natural Language Processing in R (rNLP)fridolin.wild
The introductory slides of a workshop given to the doctoral school at the Institute of Business Informatics of the Goethe University Frankfurt. The tutorials are available on http://crunch.kmi.open.ac.uk/w/index.php/Tutorials
The document discusses recent developments in the R programming environment for data analysis, including packages like magrittr, readr, tidyr, and dplyr that enable data wrangling workflows. It provides an overview of the key functions in these packages that allow users to load, reshape, manipulate, model, visualize, and report on data in a pipeline using the %>% operator.
From Provenance Standards and Tools to Queries and Actionable ProvenanceBertram Ludäscher
The document discusses computational provenance and the need for tracking data lineage and workflow processes. It presents several tools and projects that aim to capture and manage provenance information, including DataONE, SKOPE, KURATOR, WHOLE-TALE, and YesWorkflow. The document argues that provenance is important for understanding what happened in computational and data-driven research in order to ensure transparency and reproducibility.
This document provides an overview of machine learning concepts including:
1. Machine learning aims to create computer programs that improve with experience by learning from data. It involves tasks like classification, regression, and clustering.
2. Data comes in different types like text, numbers, images and is generated in massive quantities daily from sources like Google, Facebook, and sensors.
3. Machine learning algorithms are either supervised, using labeled training data, or unsupervised, using unlabeled data. Common supervised techniques are decision trees, neural networks, and support vector machines while clustering is a major unsupervised technique.
This document provides an overview of machine learning concepts. It defines machine learning as creating computer programs that improve with experience. Supervised learning uses labeled training data to build models that can classify or predict new examples, while unsupervised learning finds patterns in unlabeled data. Examples of machine learning applications include spam filtering, recommendation systems, and medical diagnosis. The document also discusses important machine learning techniques like k-nearest neighbors, decision trees, regularization, and cross-validation.
The document benchmarks 20 machine learning models on two datasets to compare their accuracy and speed. On the smaller Car Evaluation dataset, bagged decision trees, random forests and boosted decision trees achieved over 99% accuracy, while neural networks, decision stumps and support vector machines exceeded 95% accuracy. On the larger Nursery dataset, similar models exceeded 99% accuracy, while other models like decision rules and k-nearest neighbors exceeded 95% accuracy. However, models varied significantly in speed depending on the hardware, with decision trees, mixture discriminant analysis and gradient boosting as the fastest on Car Evaluation, and mixture discriminant analysis, one rule and boosted decision trees as the fastest on Nursery. The findings imply the importance of regular benchmarking
Natural Language Processing in R (rNLP)fridolin.wild
The introductory slides of a workshop given to the doctoral school at the Institute of Business Informatics of the Goethe University Frankfurt. The tutorials are available on http://crunch.kmi.open.ac.uk/w/index.php/Tutorials
The document discusses recent developments in the R programming environment for data analysis, including packages like magrittr, readr, tidyr, and dplyr that enable data wrangling workflows. It provides an overview of the key functions in these packages that allow users to load, reshape, manipulate, model, visualize, and report on data in a pipeline using the %>% operator.
The document discusses information visualization and data mapping. It provides examples of early information visualization works from the 1980s to 2000s. It then discusses visual perception principles like pre-attentive features and Gestalt laws that can be applied to design effective visualizations. Next, it covers different types of data like quantitative, ordinal, categorical, and network data. Finally, it discusses the differences between scientific visualization of concrete data versus information visualization of abstract data, which requires visual metaphors. The overall focus is on understanding how to map different data types to appropriate visual representations.
Facilitating Data Curation: a Solution Developed in the Toxicology DomainChristophe Debruyne
Christophe Debruyne, Jonathan Riggio, Emma Gustafson, Declan O'Sullivan, Mathieu Vinken, Tamara Vanhaecke, Olga De Troyer.
Presented at the 2020 IEEE 14th International Conference on Semantic Computing, San Diego, California, 3-5 February 2020
Toxicology aims to understand the adverse effects of
chemical compounds or physical agents on living organisms. For
chemicals, much information regarding safety testing of cosmetic
ingredients is now scattered in a plethora of safety evaluation
reports. Toxicologists in our university intend to collect this
information into a single repository. Their current approach uses
spreadsheets, does not scale well, and makes data curation and
querying cumbersome. Semantic technologies (e.g., RDF, OWL,
and Linked Data principles) would be more appropriate for
this purpose. However, this technology is not very accessible to
toxicologists without extensive training. In this paper, we report
on a tool that supports subject matter experts in the construction
of an RDF–based knowledge base for the toxicology domain. The
tool is using the jigsaw metaphor for guiding the subject matter
experts. We demonstrate that the jigsaw metaphor is a viable
option for generating RDF. Future work includes investigating
appropriate methods and tools for knowledge evolution and data
analysis.
I summarize requirements for an "Open Analytics Environment" (aka "the Cauldron"), and some work being performed at the University of Chicago and Argonne National Laboratory towards its realization.
Marius Eriksen discusses Reflow, a new cloud-native workflow framework for bioinformatics. Reflow programs workflows directly using a functional programming language for simplicity and composability. It leverages lazy evaluation and caching to efficiently parallelize and distribute work across private clusters. Reflow aims to untie the hands of implementors compared to traditional workflow systems through its unified approach to programming, execution, and infrastructure.
WEKA is a collection of machine learning algorithms for data mining tasks developed in Java by the University of Waikato. It contains tools for data pre-processing, classification, regression, clustering, association rules, and feature selection. The Explorer interface in WEKA provides tools to load data, preprocess data using filters, analyze data using these machine learning algorithms, and evaluate results.
Prepares the students for (and is a prerequisite for) the more advanced material students will encounter in later courses. Data structures organize data Þ more efficient programs.
This document discusses web-scale semantic search and knowledge graphs. It introduces the concept of semantic search, which deals with understanding the meaning of queries, terms, documents and results. This is achieved by linking text to unambiguous concepts or entities. The document then discusses knowledge graphs, which define entities, attributes, types, relations and more, and form the backbone of semantic search. It also covers tasks involved in semantic search like information extraction, entity linking, query understanding and result ranking.
Visualization of Supervised Learning with {arules} + {arulesViz}Takashi J OZAKI
This document discusses visualizing supervised learning models using association rules and the arules and arulesViz packages in R. It shows how association rules generated from sample user activity data can be represented as graphs, allowing intuitive visualization of relationships between variables even in high-dimensional data. The visualizations are compared to results from GLMs and random forests to show how nodes are located based on their "closeness" in different supervised learning models. While less quantitative, this technique provides a more intuitive understanding of supervised learning that is useful for presentations.
Architectural decisions in designing data and computation intensive systems can have a major impact on the ability of these systems to perform statistical and other complex calculations efficiently. The storage, processing, tools, and associated databases coupled with the networking and compute infrastructure make some kinds of computations easier, and other harder. This talk will provide an introduction to software and data systems components that are important for understanding how these choices impact data analysis uncertainties and costs, and thus for developing system and software designs best suited to statistical analyses.
Useing PSO to optimize logit model with TensorflowYi-Fan Liou
This project aim to use particle swarm optimization (PSO), one the evolutionary algorithms, to optimize the weights and bias in logistic regression using Tensorflow.
This document discusses the evolution of systems performance analysis tools from closed source to open source environments.
In the early 2000s with Solaris 9, performance analysis was limited due to closed source tools that provided only high-level metrics. Opening the Solaris kernel code with OpenSolaris in 2005 allowed deeper insight through understanding undocumented metrics and dynamic tracing tools like DTrace. This filled observability gaps across the entire software stack.
Modern performance analysis leverages both traditional Unix tools and new dynamic tracing tools. With many high-resolution metrics available, the focus is on visualization and collecting metrics across cloud environments. Overall open source improved systems analysis by providing full source code access.
Expanded set of slides (original was 5 slides) of a short presentation on
"Advanced Tools and Techniques for Logic-Based Knowledge Representation, Process Documentation, and Data Curation".
Presented at GSLIS Research Showcase, April 3, 2015.
The document discusses data structures and their implementation in C++. It covers topics like the need for data structures to organize data efficiently, commonly used data structures like arrays, linked lists, stacks and queues, selecting appropriate data structures based on algorithm requirements, and implementing dynamic arrays in C++ using pointers and the new operator.
Cytoscape Tutorial Session 1 at UT-KBRIN Bioinformatics Summit 2014 (4/11/2014)Keiichiro Ono
This document outlines a tutorial on biological data analysis and visualization using Cytoscape. The tutorial covers basic concepts like networks and tables in Cytoscape, data import, network analysis features, and visualization techniques. It discusses loading sample network data, calculating network statistics, filtering networks, basic search functionality, and applying visual styles. The tutorial is intended to provide a practical introduction to Cytoscape's core features through examples and demos.
Bjarne Stroustrup - The Essence of C++: With Examples in C++84, C++98, C++11,...Complement Verb
C++11 is being deployed and the shape of C++14 is becoming clear. This talk examines the foundations of C++. What is essential? What sets C++ apart from other languages? How do new and old features support (or distract from) design and programming relying on this essence?
I focus on the abstraction mechanisms (as opposed to the mapping to the machine): Classes and templates. Fundamentally, if you understand vector, you understand C++.
Type safety and resource safety are key design aims for a program. These aims must be met without limiting the range of applications and without imposing significant run-time or space overheads. I address issues of resource management (garbage collection is not an ideal answer and pointers should not be used as resource handles), generic programming (we must make it simpler and safer), compile-time computation (how and when?), and type safety (casts belongs in the lowest-level hardware interface). I will touch upon move semantics, exceptions, concepts, type aliases, and more. My aim is not so much to present novel features and technique, but to explore how C++’s feature set supports a new and more effective design and programming style.
FlinkForward Asia 2019 - Evolving Keystone to an Open Collaborative Real Time...Zhenzhong Xu
Netflix is obsessed with customer joy, we relentlessly focus on product experience and high-quality content. In recent years, we have been making heavy investments in the tech-driven studio and content production. As a result, a lot of unique challenges arise in the real-time data infrastructure space. For example, in a microservices architecture, domain entities are spread in different applications and persistence storages, this made low latency consistent operational reporting and entity searching especially challenging.
In this talk, we’ll talk about some interesting use cases, the various challenges lay in the fundamentals of distributed systems, and how did we solve them. We will also discuss the learnings, things we could’ve done differently, and the new vision towards an open self-serving Data Mesh platform that empowers our partners and users to build flexible real-time data pipelines.
Machine learning is the science of getting computers to act without being explicitly programmed. In the past decade, machine learning has given us self-driving cars, practical speech recognition, effective web search, and a vastly improved understanding of the human genome. Machine learning is so pervasive today that you probably use it dozens of times a day without knowing it.
We will review some modern machine learning applications, understand variety of machine learning problem definitions, go through particular approaches of solving machine learning tasks.
This year 2015 Amazon and Microsoft introduced services to perform machine learning tasks in cloud. Microsoft Azure Machine Learning offers a streamlined experience for all data scientist skill levels, from setting up with only a web browser, to using drag and drop gestures and simple data flow graphs to set up experiments.
We will briefly review Azure ML Studio features and run machine learning experiment.
Reconciling Conflicting Data Curation Actions: Transparency Through Argument...Bertram Ludäscher
Yilin Xia (yilinx2@illinois.edu),
Shawn Bowers (bowers@gonzaga.edu),
Lan Li (lanl2@illinois.edu), and
Bertram Ludäscher (ludaesch@illinois.edu)
Presented at IDCC-2024 in Edinburg.
ABSTRACT. We propose a new approach for modeling and reconciling conflicting data cleaning actions. Such conflicts arise naturally in collaborative data curation settings where multiple experts work independently and then aim to put their efforts together to improve and accelerate data cleaning. The key idea of our approach is to model conflicting updates as a formal argumentation framework (AF). Such argumentation frameworks can be automatically analyzed and solved by translating them to a logic program PAF whose declarative semantics yield a transparent solution with many desirable properties, e.g., uncontroversial updates are accepted, unjustified ones are rejected, and the remaining ambiguities are exposed and presented to users for further analysis. After motivating the problem, we introduce our approach and illustrate it with a detailed running example introducing both well-founded and stable semantics to help understand the AF solutions. We have begun to develop open source tools and Jupyter notebooks that demonstrate the practicality of our approach. In future work we plan to develop a toolkit for conflict resolution that can be used in conjunction with OpenRefine, a popular interactive data cleaning tool.
Games, Queries, and Argumentation Frameworks: Time for a Family ReunionBertram Ludäscher
Research Seminar Talk (online) at KRR@UP (Uni Potsdam) on Dec 6, 2023, loosely based on a paper with the same title at the 7th Workshop on Advances in Argumentation in Artificial Intelligence (AI3)
The document discusses information visualization and data mapping. It provides examples of early information visualization works from the 1980s to 2000s. It then discusses visual perception principles like pre-attentive features and Gestalt laws that can be applied to design effective visualizations. Next, it covers different types of data like quantitative, ordinal, categorical, and network data. Finally, it discusses the differences between scientific visualization of concrete data versus information visualization of abstract data, which requires visual metaphors. The overall focus is on understanding how to map different data types to appropriate visual representations.
Facilitating Data Curation: a Solution Developed in the Toxicology DomainChristophe Debruyne
Christophe Debruyne, Jonathan Riggio, Emma Gustafson, Declan O'Sullivan, Mathieu Vinken, Tamara Vanhaecke, Olga De Troyer.
Presented at the 2020 IEEE 14th International Conference on Semantic Computing, San Diego, California, 3-5 February 2020
Toxicology aims to understand the adverse effects of
chemical compounds or physical agents on living organisms. For
chemicals, much information regarding safety testing of cosmetic
ingredients is now scattered in a plethora of safety evaluation
reports. Toxicologists in our university intend to collect this
information into a single repository. Their current approach uses
spreadsheets, does not scale well, and makes data curation and
querying cumbersome. Semantic technologies (e.g., RDF, OWL,
and Linked Data principles) would be more appropriate for
this purpose. However, this technology is not very accessible to
toxicologists without extensive training. In this paper, we report
on a tool that supports subject matter experts in the construction
of an RDF–based knowledge base for the toxicology domain. The
tool is using the jigsaw metaphor for guiding the subject matter
experts. We demonstrate that the jigsaw metaphor is a viable
option for generating RDF. Future work includes investigating
appropriate methods and tools for knowledge evolution and data
analysis.
I summarize requirements for an "Open Analytics Environment" (aka "the Cauldron"), and some work being performed at the University of Chicago and Argonne National Laboratory towards its realization.
Marius Eriksen discusses Reflow, a new cloud-native workflow framework for bioinformatics. Reflow programs workflows directly using a functional programming language for simplicity and composability. It leverages lazy evaluation and caching to efficiently parallelize and distribute work across private clusters. Reflow aims to untie the hands of implementors compared to traditional workflow systems through its unified approach to programming, execution, and infrastructure.
WEKA is a collection of machine learning algorithms for data mining tasks developed in Java by the University of Waikato. It contains tools for data pre-processing, classification, regression, clustering, association rules, and feature selection. The Explorer interface in WEKA provides tools to load data, preprocess data using filters, analyze data using these machine learning algorithms, and evaluate results.
Prepares the students for (and is a prerequisite for) the more advanced material students will encounter in later courses. Data structures organize data Þ more efficient programs.
This document discusses web-scale semantic search and knowledge graphs. It introduces the concept of semantic search, which deals with understanding the meaning of queries, terms, documents and results. This is achieved by linking text to unambiguous concepts or entities. The document then discusses knowledge graphs, which define entities, attributes, types, relations and more, and form the backbone of semantic search. It also covers tasks involved in semantic search like information extraction, entity linking, query understanding and result ranking.
Visualization of Supervised Learning with {arules} + {arulesViz}Takashi J OZAKI
This document discusses visualizing supervised learning models using association rules and the arules and arulesViz packages in R. It shows how association rules generated from sample user activity data can be represented as graphs, allowing intuitive visualization of relationships between variables even in high-dimensional data. The visualizations are compared to results from GLMs and random forests to show how nodes are located based on their "closeness" in different supervised learning models. While less quantitative, this technique provides a more intuitive understanding of supervised learning that is useful for presentations.
Architectural decisions in designing data and computation intensive systems can have a major impact on the ability of these systems to perform statistical and other complex calculations efficiently. The storage, processing, tools, and associated databases coupled with the networking and compute infrastructure make some kinds of computations easier, and other harder. This talk will provide an introduction to software and data systems components that are important for understanding how these choices impact data analysis uncertainties and costs, and thus for developing system and software designs best suited to statistical analyses.
Useing PSO to optimize logit model with TensorflowYi-Fan Liou
This project aim to use particle swarm optimization (PSO), one the evolutionary algorithms, to optimize the weights and bias in logistic regression using Tensorflow.
This document discusses the evolution of systems performance analysis tools from closed source to open source environments.
In the early 2000s with Solaris 9, performance analysis was limited due to closed source tools that provided only high-level metrics. Opening the Solaris kernel code with OpenSolaris in 2005 allowed deeper insight through understanding undocumented metrics and dynamic tracing tools like DTrace. This filled observability gaps across the entire software stack.
Modern performance analysis leverages both traditional Unix tools and new dynamic tracing tools. With many high-resolution metrics available, the focus is on visualization and collecting metrics across cloud environments. Overall open source improved systems analysis by providing full source code access.
Expanded set of slides (original was 5 slides) of a short presentation on
"Advanced Tools and Techniques for Logic-Based Knowledge Representation, Process Documentation, and Data Curation".
Presented at GSLIS Research Showcase, April 3, 2015.
The document discusses data structures and their implementation in C++. It covers topics like the need for data structures to organize data efficiently, commonly used data structures like arrays, linked lists, stacks and queues, selecting appropriate data structures based on algorithm requirements, and implementing dynamic arrays in C++ using pointers and the new operator.
Cytoscape Tutorial Session 1 at UT-KBRIN Bioinformatics Summit 2014 (4/11/2014)Keiichiro Ono
This document outlines a tutorial on biological data analysis and visualization using Cytoscape. The tutorial covers basic concepts like networks and tables in Cytoscape, data import, network analysis features, and visualization techniques. It discusses loading sample network data, calculating network statistics, filtering networks, basic search functionality, and applying visual styles. The tutorial is intended to provide a practical introduction to Cytoscape's core features through examples and demos.
Bjarne Stroustrup - The Essence of C++: With Examples in C++84, C++98, C++11,...Complement Verb
C++11 is being deployed and the shape of C++14 is becoming clear. This talk examines the foundations of C++. What is essential? What sets C++ apart from other languages? How do new and old features support (or distract from) design and programming relying on this essence?
I focus on the abstraction mechanisms (as opposed to the mapping to the machine): Classes and templates. Fundamentally, if you understand vector, you understand C++.
Type safety and resource safety are key design aims for a program. These aims must be met without limiting the range of applications and without imposing significant run-time or space overheads. I address issues of resource management (garbage collection is not an ideal answer and pointers should not be used as resource handles), generic programming (we must make it simpler and safer), compile-time computation (how and when?), and type safety (casts belongs in the lowest-level hardware interface). I will touch upon move semantics, exceptions, concepts, type aliases, and more. My aim is not so much to present novel features and technique, but to explore how C++’s feature set supports a new and more effective design and programming style.
FlinkForward Asia 2019 - Evolving Keystone to an Open Collaborative Real Time...Zhenzhong Xu
Netflix is obsessed with customer joy, we relentlessly focus on product experience and high-quality content. In recent years, we have been making heavy investments in the tech-driven studio and content production. As a result, a lot of unique challenges arise in the real-time data infrastructure space. For example, in a microservices architecture, domain entities are spread in different applications and persistence storages, this made low latency consistent operational reporting and entity searching especially challenging.
In this talk, we’ll talk about some interesting use cases, the various challenges lay in the fundamentals of distributed systems, and how did we solve them. We will also discuss the learnings, things we could’ve done differently, and the new vision towards an open self-serving Data Mesh platform that empowers our partners and users to build flexible real-time data pipelines.
Machine learning is the science of getting computers to act without being explicitly programmed. In the past decade, machine learning has given us self-driving cars, practical speech recognition, effective web search, and a vastly improved understanding of the human genome. Machine learning is so pervasive today that you probably use it dozens of times a day without knowing it.
We will review some modern machine learning applications, understand variety of machine learning problem definitions, go through particular approaches of solving machine learning tasks.
This year 2015 Amazon and Microsoft introduced services to perform machine learning tasks in cloud. Microsoft Azure Machine Learning offers a streamlined experience for all data scientist skill levels, from setting up with only a web browser, to using drag and drop gestures and simple data flow graphs to set up experiments.
We will briefly review Azure ML Studio features and run machine learning experiment.
Reconciling Conflicting Data Curation Actions: Transparency Through Argument...Bertram Ludäscher
Yilin Xia (yilinx2@illinois.edu),
Shawn Bowers (bowers@gonzaga.edu),
Lan Li (lanl2@illinois.edu), and
Bertram Ludäscher (ludaesch@illinois.edu)
Presented at IDCC-2024 in Edinburg.
ABSTRACT. We propose a new approach for modeling and reconciling conflicting data cleaning actions. Such conflicts arise naturally in collaborative data curation settings where multiple experts work independently and then aim to put their efforts together to improve and accelerate data cleaning. The key idea of our approach is to model conflicting updates as a formal argumentation framework (AF). Such argumentation frameworks can be automatically analyzed and solved by translating them to a logic program PAF whose declarative semantics yield a transparent solution with many desirable properties, e.g., uncontroversial updates are accepted, unjustified ones are rejected, and the remaining ambiguities are exposed and presented to users for further analysis. After motivating the problem, we introduce our approach and illustrate it with a detailed running example introducing both well-founded and stable semantics to help understand the AF solutions. We have begun to develop open source tools and Jupyter notebooks that demonstrate the practicality of our approach. In future work we plan to develop a toolkit for conflict resolution that can be used in conjunction with OpenRefine, a popular interactive data cleaning tool.
Games, Queries, and Argumentation Frameworks: Time for a Family ReunionBertram Ludäscher
Research Seminar Talk (online) at KRR@UP (Uni Potsdam) on Dec 6, 2023, loosely based on a paper with the same title at the 7th Workshop on Advances in Argumentation in Artificial Intelligence (AI3)
Games, Queries, and Argumentation Frameworks: Time for a Family Reunion!Bertram Ludäscher
7th Workshop on Advances in Argumentation in Artificial Intelligence (AI3) at
AIxIA 2023: 22nd International Conference of the Italian Association for Artificial Intelligence.
Presentation of a paper by Bertram Ludäscher, Shawn Bowers, and Yilin Xia, given virtually on November 9, 2023.
[Flashback] Integration of Active and Deductive Database RulesBertram Ludäscher
Slides of my PhD defense at the University of Freiburg, 1998.
Statelog and similar state-oriented extensions of Datalog have seen renewed interest subsequently, e.g., see
[Hel10] Hellerstein, J.M., 2010. The declarative imperative: experiences and conjectures in distributed logic. ACM SIGMOD Record, 39(1), pp.5-19.
[AMC+11]
Alvaro, P., Marczak, W.R., Conway, N., Hellerstein, J.M., Maier, D. and Sears, R., 2011. Dedalus: Datalog in time and space. In Datalog Reloaded: First International Workshop, Datalog 2010, Oxford, UK, March 16-19, 2010. Revised Selected Papers (pp. 262-281). Springer
[Flashback] Statelog: Integration of Active & Deductive Database RulesBertram Ludäscher
This document discusses Statelog, which integrates active and deductive database rules. Statelog allows both active rules, which trigger actions and modify the database, and deductive rules, which derive new facts. It defines the semantics of different types of rules and how they interact. Statelog guarantees termination of rule evaluation at both compile-time and runtime through techniques like state-stratification and delta-monotonicity. It can express complex temporal queries and supports features like nested transactions.
Answering More Questions with Provenance and Query PatternsBertram Ludäscher
This document discusses using provenance information to improve transparency and reproducibility in research. It begins by asking questions about the input data, methods, and parameter settings used in a study in order to assess its reliability. It then provides examples of how workflow systems can capture provenance at both the design level (prospective provenance) and runtime level (retrospective provenance). These include a Kepler workflow that simulates X-ray data collection and provenance traces captured by DataONE. The document argues that provenance is a critical link between workflow modeling and runtime traces that can increase trust in research findings.
Computational Reproducibility vs. Transparency: Is It FAIR Enough?Bertram Ludäscher
Keynote at CLIR Workshop (Webinar): Torward Open, Reproducible, and Reusable Research. February 10, 2021. https://reusableresearch.com/
ABSTRACT. The “reproducibility crisis” has resulted in much interest in methods and tools to improve computational reproducibility. FAIR data principles (data should be findable, accessible, interoperable, and reusable) are also being adapted and evolved to apply to other artifacts, notably computational analyses (scientific workflows, Jupyter notebooks, etc.). The current focus on computational reproducibility of scripts and other computational workflows sometimes overshadows a somewhat neglected and arguably more important issue: transparency of data analysis, including data wrangling and cleaning. In this talk I will ask the question: What information is gained by conducting a reproducibility experiment? This leads to a simple model (PRIMAD) that aims to answer this question by sorting out different scenarios. Finally, I will present some features of Whole-Tale, a computational platform for reproducible and transparent computational experiments.
By Michael Gryk and Bertram Ludäscher. Presented at 2020 JCDL-SIGCM Workshop, August 1, 2020.
ABSTRACT. Conceptual models can serve multiple purposes: communication of information between stakeholders, information abstraction and generalization, and information organization for archival and retrieval. An ongoing research question is how to formally define the fit-for-purpose of a conceptual model as well as to define metrics or tests to determine whether a given model faithfully supports a designated purpose.
This paper summarizes preliminary investigations in this area by presenting toy problems along with different conceptual models for the system under study. It is argued that the different models are adequate in supporting a sophisticated query and yet they adopt different normalization schemes and will differ in expressiveness depending on the implied purpose of the models. As the subtitle suggests, this work is intended to be primarily exploratory as to the constraints a formal system would require in defining the “usefulness”, “expressiveness” and “equivalence” of conceptual models.
From Workflows to Transparent Research Objects and Reproducible Science TalesBertram Ludäscher
The document discusses prospective and retrospective provenance in scientific workflows. Prospective provenance involves modeling the workflow design, while retrospective provenance records the workflow execution. The YesWorkflow and noWorkflow tools demonstrate these two types of provenance. YesWorkflow annotates scripts to recreate a workflow model from the script, while noWorkflow records step-by-step runtime logs. Combining both approaches provides a more complete view of a workflow's provenance. Maintaining provenance is important for reproducibility and understanding the origins of scientific results.
From Research Objects to Reproducible Science TalesBertram Ludäscher
University of Southampton. Electronics & Computer Science. Research Seminar (Invited Talk).
TITLE: From Research Objects to Reproducible Science Tales
ABSTRACT. Rumor has it that there is a reproducibility crisis in science. Or maybe there are multiple crises? What do we mean by reproducibility and replicability anyways? In this talk I will first make an attempt at sorting out some of the terminological confusion in this area, focusing on computational aspects. The PRIMAD model is another attempt to describe different aspects of reproducibility studies by focusing on the "delta" between those studies and the original study. In addition to these more theoretical investigations, I will discuss practical efforts to create more reproducible and more transparent computational platforms such as the one developed by the Whole-Tale project: here 'tales' are executable research objects that may combine data, code, runtime environments, and narratives (i.e., the traditional "science story"). I will conclude with some thoughts about the remaining challenges and opportunities to bridge the large conceptual gaps that continue to exist despite the recognition of problems of reproducibility and transparency in science.
ABOUT the Speaker. Bertram Ludäscher is a professor at the School of Information Sciences at the University of Illinois, Urbana-Champaign and a faculty affiliate with the National Center for Supercomputing Applications (NCSA) and the Department of Computer Science at Illinois. Until 2014 he was a professor at the Department of Computer Science at the University of California, Davis. His research interests range from practical questions in scientific data and workflow management, to database theory and knowledge representation and reasoning. Prior to his faculty appointments, he was a research scientist at the San Diego Supercomputer Center (SDSC) and an adjunct faculty at the CSE Department at UC San Diego. He received his M.S. (Dipl.-Inform.) in computer science from the University of Karlsruhe (now K.I.T.), and his PhD (Dr. rer. nat.) from the University of Freiburg, in Germany.
Possible Worlds Explorer: Datalog & Answer Set Programming for the Rest of UsBertram Ludäscher
PWE: Datalog & ASP for the Rest of Us discusses using Possible Worlds Explorer (PWE) to make Datalog and Answer Set Programming (ASP) more accessible to non-experts. It covers topics like using provenance to explain query results, capturing rule firings to track provenance, representing provenance as a graph, using states to track derivation rounds, and declarative profiling of Datalog programs. The presentation advocates for tools like PWE that wrap Datalog/ASP engines to combine them with Python ecosystems and allow interactive use in Jupyter notebooks. This makes the languages more approachable and helps users build on existing work by experimenting further.
Deduktive Datenbanken & Logische Programme: Eine kleine ZeitreiseBertram Ludäscher
Deductive Databases & Logic Programs: Back to the Future!
Colloquium talk on the occasion of the retirement of Prof. Dr. Georg Lausen, May 10th, 2019, Universität Freiburg, Germany
Dissecting Reproducibility: A case study with ecological niche models in th...Bertram Ludäscher
1) The document describes a workshop on research synthesis and reproducibility.
2) It discusses challenges with reproducibility in science and proposes provenance and conceptual tools like PRIMAD to help address these challenges.
3) The document presents a case study where an intern was able to reproduce results from a 2006 ecological niche modeling paper using the Whole Tale environment and MaxEnt software, demonstrating computational reproducibility.
Incremental Recomputation: Those who cannot remember the past are condemned ...Bertram Ludäscher
Talk given at "Problems and techniques for Incremental Re-computation: provenance and beyond".
A workshop co-organized with Provenance Week 2018
King's College London, 12th and 13th July, 2018
Organizers: Paolo Missier (Newcastle University), Tanu Malik (DePaul University), Jacek Cala (Newcastle University)
Abstract: Incremental recomputation has applications, e.g., in databases and workflow systems. Methods and algorithms for recomputation depend on the underlying model of computation (MoC) and model of provenance (MoP). This relation is explored with some examples from databases and workflow systems.
Validation and Inference of Schema-Level Workflow Data-Dependency AnnotationsBertram Ludäscher
Presentation slides of paper by Shawn Bowers, Timothy McPhillips, and Bertram Ludäscher, given by Shawn at Provenance and Annotation of Data and Processes - 7th International Provenance and Annotation Workshop, IPAW 2018, King's College London, UK, July 9-10, 2018.
The paper won a the IPAW best paper award: https://twitter.com/kbelhajj/status/1017082775856467968
ABSTRACT. An advantage of scientific workflow systems is their ability to collect runtime provenance information as an execution trace. Traces include the computation steps invoked as part of the workflow run along with the corresponding data consumed and produced by each workflow step. The information captured by a trace is used to infer "lineage'' relationships among data items, which can help answer provenance queries to find workflow inputs that were involved in producing specific workflow outputs. Determining lineage relationships, however, requires an understanding of the dependency patterns that exist between each workflow step's inputs and outputs, and this information is often under-specified or generally assumed by workflow systems. For instance, most approaches assume all outputs depend on all inputs, which can lead to lineage "false positives''. In prior work, we defined annotations for specifying detailed dependency relationships between inputs and outputs of computation steps. These annotations are used to define corresponding rules for inferring fine-grained data dependencies from a trace. In this paper, we extend our previous work by considering the impact of dependency annotations on workflow specifications. In particular, we provide a reasoning framework to ensure the set of dependency annotations on a workflow specification is consistent. The framework can also infer a complete set of annotations given a partially annotated workflow. Finally, we describe an implementation of the reasoning framework using answer-set programming.
An ontology-driven framework for data transformation in scientific workflowsBertram Ludäscher
Presentation given by Bertram at the Data Integration in the Life Sciences (DILS) Workshop in Leipzig, Germany, 2004.
Reference:
Bowers, Shawn, and Bertram Ludäscher. "An ontology-driven framework for data transformation in scientific workflows." In International Workshop on Data Integration in the Life Sciences (DILS), pp. 1-16. Springer, 2004.
So this isn't new -- but still relevant :-)
ABSTRACT. Ecologists spend considerable effort integrating heterogeneous data for statistical analyses and simulations, for example, to run and test predictive models. Our research is focused on reducing this effort by providing data integration and transformation tools, allowing researchers to focus on “real science,” that is, discovering new knowledge through analysis and modeling. This paper defines a generic framework for transforming heterogeneous data within scientific workflows. Our approach relies on a formalized ontology, which serves as a simple, unstructured global schema. In the framework, inputs and outputs of services within scientific workflows can have structural types and separate seman- tic types (expressions of the target ontology). In addition, a registration mapping can be defined to relate input and output structural types to their corresponding semantic types. Using registration mappings, ap- propriate data transformations can then be generated for each desired service composition. Here, we describe our proposed framework and an initial implementation for services that consume and produce XML data.
The document describes the Whole Tale platform, which aims to facilitate reproducibility in computational research. Whole Tale allows researchers to package computational narratives, data, code, and provenance information into "tales" that can be shared and re-executed. Key features of Whole Tale include running interactive notebooks, versioning and sharing tales, and integrating provenance tracking tools to provide transparency into computational workflows. The speaker demonstrates several example tales and discusses upcoming Whole Tale features and applications in different domains like archaeology, astronomy, and materials science.
Wild Ideas at TDWG'17: Embrace multiple possible worlds; abandon techno-ligionBertram Ludäscher
The document discusses two ideas: 1) Embracing multiple possible worlds by using techniques like answer set programming to represent alternative scenarios rather than a single consensus view. 2) Abandoning strict adherence to technology stacks and standards ("techno-ligion") by focusing on simple powerful solutions, using natural language when possible, and paying a fee each time a complex technical term is used. It suggests using techniques like technology golf to explore problems through minimal programs instead of lengthy debates over formal representations.
Global Situational Awareness of A.I. and where its headedvikram sood
You can see the future first in San Francisco.
Over the past year, the talk of the town has shifted from $10 billion compute clusters to $100 billion clusters to trillion-dollar clusters. Every six months another zero is added to the boardroom plans. Behind the scenes, there’s a fierce scramble to secure every power contract still available for the rest of the decade, every voltage transformer that can possibly be procured. American big business is gearing up to pour trillions of dollars into a long-unseen mobilization of American industrial might. By the end of the decade, American electricity production will have grown tens of percent; from the shale fields of Pennsylvania to the solar farms of Nevada, hundreds of millions of GPUs will hum.
The AGI race has begun. We are building machines that can think and reason. By 2025/26, these machines will outpace college graduates. By the end of the decade, they will be smarter than you or I; we will have superintelligence, in the true sense of the word. Along the way, national security forces not seen in half a century will be un-leashed, and before long, The Project will be on. If we’re lucky, we’ll be in an all-out race with the CCP; if we’re unlucky, an all-out war.
Everyone is now talking about AI, but few have the faintest glimmer of what is about to hit them. Nvidia analysts still think 2024 might be close to the peak. Mainstream pundits are stuck on the wilful blindness of “it’s just predicting the next word”. They see only hype and business-as-usual; at most they entertain another internet-scale technological change.
Before long, the world will wake up. But right now, there are perhaps a few hundred people, most of them in San Francisco and the AI labs, that have situational awareness. Through whatever peculiar forces of fate, I have found myself amongst them. A few years ago, these people were derided as crazy—but they trusted the trendlines, which allowed them to correctly predict the AI advances of the past few years. Whether these people are also right about the next few years remains to be seen. But these are very smart people—the smartest people I have ever met—and they are the ones building this technology. Perhaps they will be an odd footnote in history, or perhaps they will go down in history like Szilard and Oppenheimer and Teller. If they are seeing the future even close to correctly, we are in for a wild ride.
Let me tell you what we see.
Learn SQL from basic queries to Advance queriesmanishkhaire30
Dive into the world of data analysis with our comprehensive guide on mastering SQL! This presentation offers a practical approach to learning SQL, focusing on real-world applications and hands-on practice. Whether you're a beginner or looking to sharpen your skills, this guide provides the tools you need to extract, analyze, and interpret data effectively.
Key Highlights:
Foundations of SQL: Understand the basics of SQL, including data retrieval, filtering, and aggregation.
Advanced Queries: Learn to craft complex queries to uncover deep insights from your data.
Data Trends and Patterns: Discover how to identify and interpret trends and patterns in your datasets.
Practical Examples: Follow step-by-step examples to apply SQL techniques in real-world scenarios.
Actionable Insights: Gain the skills to derive actionable insights that drive informed decision-making.
Join us on this journey to enhance your data analysis capabilities and unlock the full potential of SQL. Perfect for data enthusiasts, analysts, and anyone eager to harness the power of data!
#DataAnalysis #SQL #LearningSQL #DataInsights #DataScience #Analytics
The Building Blocks of QuestDB, a Time Series Databasejavier ramirez
Talk Delivered at Valencia Codes Meetup 2024-06.
Traditionally, databases have treated timestamps just as another data type. However, when performing real-time analytics, timestamps should be first class citizens and we need rich time semantics to get the most out of our data. We also need to deal with ever growing datasets while keeping performant, which is as fun as it sounds.
It is no wonder time-series databases are now more popular than ever before. Join me in this session to learn about the internal architecture and building blocks of QuestDB, an open source time-series database designed for speed. We will also review a history of some of the changes we have gone over the past two years to deal with late and unordered data, non-blocking writes, read-replicas, or faster batch ingestion.
Enhanced Enterprise Intelligence with your personal AI Data Copilot.pdfGetInData
Recently we have observed the rise of open-source Large Language Models (LLMs) that are community-driven or developed by the AI market leaders, such as Meta (Llama3), Databricks (DBRX) and Snowflake (Arctic). On the other hand, there is a growth in interest in specialized, carefully fine-tuned yet relatively small models that can efficiently assist programmers in day-to-day tasks. Finally, Retrieval-Augmented Generation (RAG) architectures have gained a lot of traction as the preferred approach for LLMs context and prompt augmentation for building conversational SQL data copilots, code copilots and chatbots.
In this presentation, we will show how we built upon these three concepts a robust Data Copilot that can help to democratize access to company data assets and boost performance of everyone working with data platforms.
Why do we need yet another (open-source ) Copilot?
How can we build one?
Architecture and evaluation
4th Modern Marketing Reckoner by MMA Global India & Group M: 60+ experts on W...Social Samosa
The Modern Marketing Reckoner (MMR) is a comprehensive resource packed with POVs from 60+ industry leaders on how AI is transforming the 4 key pillars of marketing – product, place, price and promotions.
ViewShift: Hassle-free Dynamic Policy Enforcement for Every Data LakeWalaa Eldin Moustafa
Dynamic policy enforcement is becoming an increasingly important topic in today’s world where data privacy and compliance is a top priority for companies, individuals, and regulators alike. In these slides, we discuss how LinkedIn implements a powerful dynamic policy enforcement engine, called ViewShift, and integrates it within its data lake. We show the query engine architecture and how catalog implementations can automatically route table resolutions to compliance-enforcing SQL views. Such views have a set of very interesting properties: (1) They are auto-generated from declarative data annotations. (2) They respect user-level consent and preferences (3) They are context-aware, encoding a different set of transformations for different use cases (4) They are portable; while the SQL logic is only implemented in one SQL dialect, it is accessible in all engines.
#SQL #Views #Privacy #Compliance #DataLake
06-04-2024 - NYC Tech Week - Discussion on Vector Databases, Unstructured Data and AI
Round table discussion of vector databases, unstructured data, ai, big data, real-time, robots and Milvus.
A lively discussion with NJ Gen AI Meetup Lead, Prasad and Procure.FYI's Co-Found
7. Computational Provenance …
• Origin, processing history of artifacts
– data products, figures, ...
– also: underlying workflow
è understand methods, dataflow, and dependencies
Ludäscher: Workflows & Provenance => Understanding 7
Climate Change Impacts
in the United States
U.S. National Climate Assessment
U.S. Global Change Research Program
12. • … NSF SKOPE: system and tools to discover,
access, analyze, visualize paleoenvironmental
data
– unprecedented ability to explore provenance
(detailed, comprehensible record of computational
derivation of results)
– for researchers, tinkerers, and modelers
• … NSF Whole Tale:
– leverage & contribute to existing CI to support the
whole tale (“living paper”), from workflow run to
scholarly publication
– integrate tools & CI (DataONE, Globus, iRODS,
NDS, ...) to simplify use and promote best
practices.
– driven by science WGs (Archaeology/SKOPE,
materials science, astro, bio ..)
Related Projects: NSF DataONE (ProvONE ..) + …
Ludäscher: Workflows & Provenance => Understanding 12
14. SKOPE: Synthesized Knowledge Of Past Environments
Bocinsky, Kohler et al. study rain-fed maize of Anasazi
– Four Corners; AD 600–1500. Climate change influenced Mesa Verde Migrations; late
13th century AD. Uses network of tree-ring chronologies to reconstruct a spatio-
temporal climate field at a fairly high resolution (~800 m) from AD 1–2000. Algorithm
estimates joint information in tree-rings and a climate signal to identify “best” tree-ring
chronologies for climate reconstructing.
K. Bocinsky, T. Kohler, A 2000-year reconstruction of the rain-fed
maize agricultural niche in the US Southwest. Nature
Communications. doi:10.1038/ncomms6618
… implemented as an R Script …
Ludäscher: Workflows & Provenance => Understanding 14
17. YW Demo Use Cases (IDCC’17)
Domain Use case Programming language Provenance methods
Climate science C3C4 MATLAB YW + MATLAB
RunManager
Astrophysics LIGO Python YW + NW (code-level)
Protein crystal samples Simulate data
collection
Python YW + NW (code-level)
Biodiversity data
curation
kurator-SPNHC Python YW-recon + YW-logging
Social network analysis Twitter Python YW + NW (file-level)
Oceanography OHIBC Howe Sound
(multi-run multi-script)
R YW + R RunManager
Ludäscher: Workflows & Provenance => Understanding 17
23. Hybrid Provenance:
YW Model + Runtime
Observables (file level)
Ludäscher: Workflows & Provenance => Understanding
23
�����������������
�����
���������
��������������
����������������
����������
�����������������
����������������
�������
����������
������������������
����������������
�����������������
�������������������
�����������
������������������
����������
�����������������
�����������
������������
�������������
���������������������
�������������������������������������������������������������������
�����������������
�������������������������������������������������������������������������
• The YW model can be connected
with runtime observables
• è YW recon (prov reconstruction)
• Here:
• What specific files were read,
written and where do they occur
in the workflow?
43. Adding YesWorkflow to DataONE
Yaxing’s script with
inputs & output
products
Christopher’s
YesWorkflow
model
Christopher using
Yaxing’s outputs as
inputs for his script
Christopher’s results
can be traced back all
the way to Yaxing’s
input
Ludäscher: Workflows & Provenance => Understanding
43
46. Whole Tale: What’s in a name?
(1) Whole Tale ⇔ Whole Story:
◦ Support (computational / data) scientists
◦ … along the complete research lifecycle
◦ ... from experiment to (new kind of) publication
◦ ... and back!
(2) Whole Tale ⇔ for the Long Tail of Science
–Easy sharing of your computational narratives, data, and
exec-env since 2017!
–Power applications for everyone!
46Ludäscher: Workflows & Provenance => Understanding
47. Whole Tale Vision
• Can't reproduce result because:
• Don't know how to run analysis
• Can't get the software running
• Can't pay for the computer or compute
power the result was computed on
Source: Bryce Mecum, NCEAS (WT team)
47
54. Last not least:
Non-unitary syntheses
of systematic knowledge
Please
@taxonbytes
Nico Franz
School of Life Sciences, Arizona State University
CIRSS Seminar – Center for Informatics Research in Science and Scholarship
February 17, 2017 – iSchool, University of Illinois Urbana-Champaign
@ http://www.slideshare.net/taxonbytes/franz-2017-uiuc-cirss-non-unitary-syntheses-of-systematic-knowledge 54
57. Use case 1.a. Aligning Microcebus + Mirza sec. MSW3 (2005)
"Taxonomic concept labels"
identify input concept regions
RCC–5 articulations provided
for each species-level concept
• Input visualization: MSW3 (2005) versus MSW2 (1993)
Source: Franz et al. 2016. Two influential primate classifications logical aligned. doi:10.1093/sysbio/syw023
57
58. • Alignment visualization: "grey means taxonomically congruent"
Use case 1.a. Aligning Microcebus + Mirza sec. MSW3 (2005)
58
59. One name &
congruent region
Many names &
congruent region
One name &
non-congruent regions
Many names &
non-congruent regions
New names &
exclusive regions
• Application of coverage constraint: parent-to-parent articulations (><) are
fully defined by alignment signal propagated from their respective children.
è Sensible when complete sampling of children is intended.
Use case 1.a. Aligning Microcebus + Mirza sec. MSW3 (2005)
• Alignment visualization: "grey means taxonomically congruent"
59
60. 1 in 3 names is unreliable across MSW2/MSW3 classifications
Source: Franz et al. 2016. Two influential primate classifications logical aligned. doi:10.1093/sysbio/syw023
60
61. The 'consensus' The
'bible'
The (formerly)
federal
'standard'
The 'best', latest
regional flora
"Controllingthetaxonomicvariable"
Expert views
are in
conflict
"Just bad"
Source: Franz et al. 2016. Controlling the taxonomic variable: […]. RIO Journal. doi:10.3897/rio.2.e10610
61
62. The 'consensus' The
'bible'
The (formerly)
federal
'standard'
The 'best', latest
regional flora
Impact:
Name-based aggregation has created
a novel synthesis that nobody believes in
"Controllingthetaxonomicvariable"
"Just bad"
Source: Franz et al. 2016. Controlling the taxonomic variable: […]. RIO Journal. doi:10.3897/rio.2.e10610
62
63. The 'consensus' The
'bible'
The (formerly)
federal
'standard'
The 'best', latest
regional flora
"Controllingthetaxonomicvariable"
"Just
bad"
Expert views
are
reconciled
Solution:
Instead of aggregating
an artificial 'consensus',
build translation services
Source: Franz et al. 2016. Controlling the taxonomic variable: […]. RIO Journal. doi:10.3897/rio.2.e10610
63
65. Yi-Yun Cheng1, Nico Franz2, Jodi Schneider1, Shizhuo Yu3, Thomas Rodenhausen4, Bertram Ludäscher1
1
School of Information Sciences, University of Illinois at Urbana-Champaign; 2
School of Life Sciences, Arizona State University;
3
Department of Computer Science, University of California at Davis; 4
School of Information, University of Arizona
Agreeing to Disagree: Reconciling Conflicting Taxonomic Views
using a Logic-based Approach
Acknowledgments
Support of the authors’ research through the National Science
Foundation is kindly acknowledged (DEB-1155984, DBI-1342595, and
DBI-1643002). The authors thank Professor Kathryn La Barre for her
comments and suggestions. We would also like to thank Dr. Laetitia
Navarro and Jeff Terstriep for help with creating map overlays in QGIS.
CONCLUSION
• Our logic-based taxonomy alignment approach can be used to solve
crosswalking issues
We will be able to mitigate the membership condition problems that
occur in equivalent crosswalking.
• RCC-5 approach preserves the original taxonomies while providing an
alignment view
We can solve data integration problems that happen in the more
coarse-grained relative crosswalking, which otherwise is subjected to
information loss.
• Our study also underscores the benefits of designing different
alignment workflows (Bottom up vs. Top-down) to match the needs
of specific taxonomy alignment problems
Bottom-up approach: seems to work well whenever we have non-
overlapping relationships at the leaf-level (lowest-level) articulations,
and we are not sure how the higher-level concepts should be aligned.
Top-down approach: seems favorable when there is an expectation of
certain higher-level articulations in conjunction with under-specified,
complex, and often overlapping leaf-level relations.
RELATED WORK
• Taxonomy Alignment Problems (TAP)
Taxonomies T1, T2 are inter-linked via a set of input articulations A,
defined as RCC-5 relations, to yield a “merged” taxonomy T3 .
• Euler/X
Articulations – a constraint or rule that defines a relationship (a set
constraint) between two concepts from different taxonomies .
Region Connection Calculus (RCC-5)
Possible Worlds – When encoding and solving TAPs via ASP, the
different answer sets represent alternative taxonomy merge solutions
or possible worlds (PWs).
INTRODUCTION
Tina: Hey Amy, can you recommend a signature dish from where you
live?
Amy: Oh, definitely the half-smokes from the Northeast! They are
these tasty half-pork and half-beef sausages.
Tina: What a coincidence! We have half-smokes in the South, too!
Where do you live in the Northeast? New York? Boston?
Amy: Wrong guesses! Where do you live in the South?
Tina and Amy together: Washington, D.C.
[The two of them look at each other, confused.]
“In the face of incompatible information or data structures among
users or among those specifying the system, attempts to create
unitary knowledge categories are futile. Rather, parallel or multiple
representational forms are required…” (Bowker & Star, 2000).
CASE 1 RESULTS: CEN vs. NDC
• State-level alignments are all congruent (Bottom-up)
• Inferred new articulations for regional-level alignments
CASE 2 RESULTS: CEN vs. TZ
Figure 3. (Left) CEN-NDC taxonomy alignment problem with 49 input articulations between TCEN and TNDC
Figure 4. (Right) The unique possible world (PW) T3 reconciling TCEN and TNDC via inferred relationships
Figure 1. National Diversity Council map (NDC) vs. Census Bureau map (CEN)
• Github link:
https://github.com/EulerProject/ASIST17
• Email: yiyunyc2@illinois.edu
West
Southwest Southeast
Midwest North-
east
West
South
Midwest North-
east
Pacific
Mountain
Central
Eastern
West
South
Midwest
North-
east
RESEARCH DESIGN
Step 1. Supply input taxonomies T1 and T2
Step 2. Formulate RCC-5 articulations between T1 and T2
Step 3. Iteratively edit articulations in Euler/X
Y X X YX Y X Y X Y
Congruence
X == Y
Inclusion
X > Y
Inverse Inclusion
X < Y
Overlap
X>< Y
Disjointness
X ! Y
T1
T2
T1
T2
Inconsistent (N=0)
Ambiguous (N>1)
T3
Add/Edit
Articulations A
Euler/X
N Possible Worlds
N=1 N=0 or N>1
R1
R2
R3
R4
R5
R6
R7
R8
R9
CEN.Midwest
CEN.USA
TZ.USA
CEN.West
CEN.Northeast
TZ.EasternCEN.Midwest
TZ.EasternCEN.South
CEN.South
CEN.South*TZ.Central
TZ.CentralCEN.Midwest
CEN.SouthTZ.Eastern
CEN.SouthTZ.Mountain
TZ.Central
CEN.MidwestTZ.Eastern
TZ.MountainCEN.South
TZ.Mountain
CEN.MidwestTZ.Mountain
TZ.MountainCEN.Midwest
CEN.Midwest*TZ.Mountain
CEN.MidwestTZ.Central
TZ.MountainCEN.West
CEN.Midwest*TZ.Eastern
CEN.West*TZ.Mountain
CEN.South*TZ.Mountain
CEN.SouthTZ.Central
TZ.Eastern
CEN.South*TZ.Eastern
CEN.Midwest*TZ.Central
TZ.CentralCEN.South
TZ.Pacific
CEN.WestTZ.Mountain
Nodes
CEN 4
newComb 18
comb 1
TZ 4
Edges
input 6
inferred 37
CEN.IL NDC.IL==
CEN.IN NDC.IN
==
CEN.RI NDC.RI==
CEN.IA NDC.IA==
CEN.WV NDC.WV
==
CEN.KS NDC.KS==
CEN.KY NDC.KY==
CEN.TX
NDC.TX
==
CEN.Northeast
CEN.VT
CEN.MA
CEN.ME
CEN.CT
CEN.PA
CEN.NY
CEN.NH
CEN.NJ
CEN.South
CEN.TN
CEN.MS
CEN.MD
CEN.DC
CEN.DE
CEN.VA
CEN.FL
CEN.AR
CEN.AL
CEN.OK
CEN.SC
CEN.LA
CEN.GA
CEN.NC
CEN.ID NDC.ID==
NDC.TN==
CEN.WY NDC.WY==
NDC.VT==
NDC.MS==
CEN.MT NDC.MT==
NDC.MA
==
CEN.USA
CEN.Midwest
CEN.West
NDC.ME==
NDC.MD==
CEN.MI NDC.MI==
CEN.MN NDC.MN==
NDC.DC==
NDC.DE==
CEN.OR NDC.OR==
CEN.OH NDC.OH==
NDC.VA==
NDC.FL==
NDC.AR==
CEN.AZ NDC.AZ==
NDC.AL==
NDC.OK
==
NDC.CT==
CEN.CO NDC.CO
==
CEN.CA NDC.CA==
CEN.SD NDC.SD
==
NDC.SC==
CEN.MO
CEN.ND
CEN.NE
CEN.WI
NDC.LA==
NDC.MO==
CEN.UT NDC.UT==
NDC.GA==
NDC.PA==
CEN.NV
CEN.NM
CEN.WA
NDC.NY==
NDC.NV==
NDC.NM==
NDC.WA
==
NDC.NH==
NDC.NJ==
NDC.ND==
NDC.NE==
NDC.WI==
NDC.NC==
NDC.West
NDC.Midwest
NDC.Northeast
NDC.Southeast
NDC.USA
NDC.Southwest
Nodes
CEN 54
NDC 55
Edges
isa_CEN 53
isa_NDC 54
Art. 49
CEN.West
NDC.Southwest
CEN.USA
NDC.USA
CEN.Northeast
NDC.Northeast
CEN.South
NDC.Southeast
NDC.West
CEN.DC
NDC.DC
CEN.NM
NDC.NM
CEN.ND
NDC.ND
CEN.Midwest
NDC.Midwest
CEN.AZ
NDC.AZ
CEN.CA
NDC.CA
CEN.MT
NDC.MT
CEN.MA
NDC.MA
CEN.IN
NDC.IN
CEN.NV
NDC.NV
CEN.MD
NDC.MD
CEN.CT
NDC.CT
CEN.NH
NDC.NH
CEN.KY
NDC.KY
CEN.PA
NDC.PA
CEN.CO
NDC.CO
CEN.WA
NDC.WA
CEN.MI
NDC.MI
CEN.VA
NDC.VA
CEN.WI
NDC.WI
CEN.NE
NDC.NE
CEN.SD
NDC.SD
CEN.MN
NDC.MN
CEN.MS
NDC.MS
CEN.ID
NDC.ID
CEN.WV
NDC.WV
CEN.NY
NDC.NY
CEN.NJ
NDC.NJ
CEN.UT
NDC.UT
CEN.ME
NDC.ME
CEN.IL
NDC.IL
CEN.TN
NDC.TN
CEN.VT
NDC.VT
CEN.GA
NDC.GA
CEN.DE
NDC.DE
CEN.NC
NDC.NC
CEN.OK
NDC.OK
CEN.MO
NDC.MO
CEN.SC
NDC.SC
CEN.AR
NDC.AR
CEN.TX
NDC.TX
CEN.LA
NDC.LA
CEN.OH
NDC.OH
CEN.IA
NDC.IA
CEN.KS
NDC.KS
CEN.RI
NDC.RI
CEN.WY
NDC.WY
CEN.FL
NDC.FL
CEN.OR
NDC.OR
CEN.AL
NDC.AL
Nodes
CEN 3
NDC 4
comb 51
Edges
input 61
inferred 3
overlapsinferred 3
CEN.Northeast
TZ.Eastern
<
CEN.Midwest
><
TZ.Mountain
><
TZ.Pacific
!
CEN.South
><
><
!
TZ.Central
><
CEN.USA
CEN.West
TZ.USA
==
!
><
!
Nodes
CEN 5
TZ 5
Edges
isa_CEN 4
isa_TZ 4
Art. 12
CEN.Midwest
CEN.USA
TZ.USA
TZ.Eastern
TZ.Central
TZ.Mountain
CEN.South
CEN.Northeast
CEN.West TZ.Pacific
Nodes
CEN 4
comb 1
TZ 4
Edges
input 7
overlapsinput 6
overlapsinferred 1
R1
R2
R3
R4
R5
R6
R7
R8
R9
Figure 2. The process of aligning
taxonomies T1 and T2 with Euler/X
Figure 5. Top-down
input alignments
between TCEN and TTZ
Figure 6. The unique
PW for the TCEN with
TTZ alignment
Figure 10. Combined concepts
solution for TCEN and TTZ
taxonomy CEN Census_Regions
(USA Northeast Midwest South West)
(Northeast CT MA ME NH NJ NY PA RI VT)
(Midwest IL IN IA KS MI MN MO NE ND OH
SD WI)
(South AL AR DE DC FL GA KY LA MD MS NC
OK SC TN TX VA WV)
(West AZ CA CO ID MT NV NM OR UT WA WY)
taxonomy NDC
National_Diversity_Council
(USA Midwest Northeast Southeast
Southwest West)
(Northeast CT DC DE MD MA ME NH NJ NY
PA RI VT)
(Midwest IA IL IN KS MI MN MO ND NE OH
SD WI)
(Southeast AL AR FL GA KY LA MS NC SC
TN VA WV)
(Southwest AZ NM OK TX)
(West CA CO ID MT NV OR WA WY UT)
articulations CEN NDC
[CEN.AL equals NDC.AL]
[CEN.AR equals NDC.AR]
[CEN.AZ equals NDC.AZ]
[CEN.CA equals NDC.CA]
[CEN.CO equals NDC.CO]
[CEN.CT equals NDC.CT]
[CEN.DC equals NDC.DC]
[CEN.DE equals NDC.DE]
[CEN.FL equals NDC.FL]
[CEN.GA equals NDC.GA]
[CEN.IA equals NDC.IA]
[CEN.ID equals NDC.ID]
[CEN.IL equals NDC.IL]
[CEN.IN equals NDC.IN]
[CEN.KS equals NDC.KS]
[CEN.KY equals NDC.KY]
[CEN.LA equals NDC.LA]
[CEN.MA equals NDC.MA]
[CEN.MD equals NDC.MD]
[CEN.ME equals NDC.ME]
[CEN.MI equals NDC.MI]
[CEN.MN equals NDC.MN]
...
Quick Scan!
taxonomy CEN Census_Regions
(USA Midwest South West Northeast)
taxonomy TZ Time_Zone
(USA Pacific Mountain Central Eastern)
articulations CEN TZ
[CEN.Midwest disjoint TZ.Pacific]
[CEN.Midwest overlaps TZ.Eastern]
[CEN.Midwest overlaps TZ.Mountain]
[CEN.Northeast is_included_in TZ.Eastern]
[CEN.South disjoint TZ.Pacific]
[CEN.South overlaps TZ.Central]
[CEN.South overlaps TZ.Eastern]
[CEN.South overlaps TZ.Mountain]
[CEN.USA equals TZ.USA]
[CEN.West disjoint TZ.Central]
[CEN.West disjoint TZ.Eastern]
[CEN.West overlaps TZ.Mountain]
66. Two Taxonomies: NDC vs CEN
“…in the face of incompatible information or data structures among users or among those
specifying the system, attempts to create unitary knowledge categories are futile. Rather, parallel
or multiple representational forms are required” [Bowker & Star, 2000, p.159]
West
Southwest Southeast
Midwest North-
east
West
South
Midwest North-
east
National Diversity Council map (NDC) US Census Buero map (CEN)
Source: Yi-Yun (Jessica) Cheng (PhD student, iSchool @ Illinois)
72. How we align two taxonomies T1 and T2
• Step 1. Supply input taxonomies T1
and T2
• Step 2. Describe the relationships
between T1 and T2
• Step 3. Iteratively edit articulations
in Euler/X
T1
T2
T1
T2
Inconsistent (N=0)
Ambiguous (N>1)
T3
Add/Edit
Articulations A
Euler/X
N Possible Worlds
N=1 N=0 or N>1
• … but where do the articulations
come from??
– expert opinion
– automatically derived from data
84. Implications
• Logic-based taxonomy alignment approach
– Disambiguate name-based taxonomy alignment over time
• 40% of the concepts in biology taxonomies undergoes
name change over time (Franz et al., 2016)
– May mitigate problems in equivalent crosswalking
• Membership condition problem that was often criticized in
crosswalking
– Preserves the original taxonomies while providing an
alignment view
• Solve data integration problems that happen in the more
coarse-grained relative crosswalking
11/01/17
Cheng
https://github.com/EulerProject/ASIST17
yiyunyc2@illinois.edu
85. • … Aristotle …
• … Euler …
• …
• … Greg Whitbread …
• [BPB93] J. H. Beach, S. Pramanik, and J. H. Beaman. Hierarchic
taxonomic databases.,Advances in Computer Methods for Systematic
Biology: Artificial Intelligence, Databases, Computer Vision, 1993
• [Ber95] Walter G. Berendsohn. The concept of “potential taxa” in
databases. Taxon, 44:207–212, 1995.
• [Ber03] Walter G. Berendsohn. MoReTax – Handling Factual Information
Linked to Taxonomic Concepts in Biology. No. 39 in Schriftenreihe für
Vegetationskunde. Bundesamt für Naturschutz, 2003.
• [GG03] M. Geoffroy and A. Güntsch. Assembling and navigating the
potential taxon graph. In [Ber03], pages 71–82, 2003.
• [TL07] Thau, D., & Ludäscher, B. (2007). Reasoning about taxonomies in
first-order logic. Ecological Informatics, 2(3), 195-209.
• [FP09] Franz, N. M., & Peet, R. K. (2009). Perspectives: towards a
language for mapping relationships among taxonomic concepts.
Systematics and Biodiversity, 7(1), 5-20.
• … 85
Some History