Ambiguity in interpreting signs is not a new idea, yet the vast majority of research in machine interpretation of signals such as speech, language, images, video, audio, etc., tend to ignore ambiguity. This is evidenced by the fact that metrics for quality of machine understanding rely on a ground truth, in which each instance (a sentence, a photo, a sound clip, etc) is assigned a discrete label, or set of labels, and the machine’s prediction for that instance is compared to the label to determine if it is correct. This determination yields the familiar precision, recall, accuracy, and f-measure metrics, but clearly presupposes that this determination can be made. CrowdTruth is a form of collective intelligence based on a vector representation that accommodates diverse interpretation perspectives and encourages human annotators to disagree with each other, in order to expose latent elements such as ambiguity and worker quality. In other words, CrowdTruth assumes that when annotators disagree on how to label an example, it is because the example is ambiguous, the worker isn’t doing the right thing, or the task itself is not clear. In previous work on CrowdTruth, the focus was on how the disagreement signals from low quality workers and from unclear tasks can be isolated. Recently, we observed that disagreement can also signal ambiguity. The basic hypothesis is that, if workers disagree on the correct label for an example, then it will be more difficult for a machine to classify that example. The elaborate data analysis to determine if the source of the disagreement is ambiguity supports our intuition that low clarity signals ambiguity, while high clarity sentences quite obviously express one or more of the target relations. In this talk I will share the experiences and lessons learned on the path to understanding diversity in human interpretation and the ways to capture it as ground truth to enable machines to deal with such diversity.
Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)Lora Aroyo
http://crowdtruth.org
Processing real-world data with the crowd leaves one thing absolutely clear - there is no single notion of truth, but rather a spectrum that has to account for context, opinions, perspectives and shades of grey. CrowdTruth is a new framework for processing of human semantics drawn more from the notion of consensus then from set theory.
Harnessing diversity in crowds and machines for better ner performanceoanainel
Over the last years, information extraction tools have gained a great popularity and brought significant performance improvement in extracting meaning from structured or unstructured data. For example, named entity recognition (NER) tools identify types such as people, organizations or places in text. However, despite their high F1 performance, NER tools are still prone to brittleness due to their highly specialized and constrained input and training data. Thus, each tool is able to extract only a subset of the named entities (NE) mentioned in a given text. In order to improve \emph{NE Coverage}, we propose a hybrid approach, where we first aggregate the output of various NER tools and then validate and extend it through crowdsourcing. The results from our experiments show that this approach performs significantly better than the individual state-of-the-art tools (including existing tools that integrate individual outputs already). Furthermore, we show that the crowd is quite effective in (1) identifying mistakes, inconsistencies and ambiguities in currently used ground truth, as well as in (2) a promising approach to gather ground truth annotations for NER that capture a multitude of opinions.
Crowdsourcing ambiguity aware ground truth - collective intelligence 2017Lora Aroyo
The process of gathering ground truth data through human annotation is a major bottleneck in the use of information extraction methods. Crowdsourcing-based approaches are gaining popularity in the attempt to solve the issues related to the volume of data and lack of annotators. Typically these practices use inter-annotator agreement as a measure of quality. However, this assumption often creates issues in practice. Previous experiments we performed found that inter-annotator disagreement is usually never captured, either because the number of annotators is too small to capture the full diversity of opinion, or because the crowd data is aggregated with metrics that enforce consensus, such as majority vote. These practices create artificial data that is neither general nor reflects the ambiguity inherent in the data.
To address these issues, we proposed the method for crowdsourcing ground truth by harnessing inter-annotator disagreement. We present an alternative approach for crowdsourcing ground truth data that, instead of enforcing an agreement between annotators, captures the ambiguity inherent in semantic annotation through the use of disagreement-aware metrics for aggregating crowdsourcing responses. Based on this principle, we have implemented the CrowdTruth framework for machine-human computation, that first introduced the disagreement-aware metrics and built a pipeline to process crowdsourcing data with these metrics.
In this paper, we apply the CrowdTruth methodology to collect data over a set of diverse tasks: medical relation extraction, Twitter event identification, news event extraction and sound interpretation. We prove that capturing disagreement is essential for acquiring a high-quality ground truth. We achieve this by comparing the quality of the data aggregated with CrowdTruth metrics with a majority vote, a method which enforces consensus among annotators. By applying our analysis over a set of diverse tasks we show that, even though ambiguity manifests differently depending on the task, our theory of inter-annotator disagreement as a property of ambiguity is generalizable.
Truth is a Lie: 7 Myths about Human Annotation @CogComputing Forum 2014Lora Aroyo
Big data is having a disruptive impact across the sciences.
Human annotation of semantic interpretation tasks is a critical
part of big data semantics, but it is based on an antiquated
ideal of a single correct truth that needs to be similarly
disrupted.We expose seven myths about human annotation,
most of which derive from that antiquated ideal of truth,
and dispell these myths with examples from our research.We
propose a new theory of truth, Crowd Truth, that is based
on the intuition that human interpretation is subjective, and
that measuring annotations on the same objects of interpretation (in our examples, sentences) across a crowd will provide a useful representation of their subjectivity and the range of reasonable interpretations.
Crowds & Niches Teaching Machines to Diagnose: NLeSC Kick off eHumanities pr...Lora Aroyo
This presentation was given at the NL eSchience Center during the "De Geest Uit De Fles" event for the kick off of eHumanities project in 2014:
http://esciencecenter.nl/agenda/703-26-may-de-geest-uit-de-fles/
Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)Lora Aroyo
http://crowdtruth.org
Processing real-world data with the crowd leaves one thing absolutely clear - there is no single notion of truth, but rather a spectrum that has to account for context, opinions, perspectives and shades of grey. CrowdTruth is a new framework for processing of human semantics drawn more from the notion of consensus then from set theory.
Harnessing diversity in crowds and machines for better ner performanceoanainel
Over the last years, information extraction tools have gained a great popularity and brought significant performance improvement in extracting meaning from structured or unstructured data. For example, named entity recognition (NER) tools identify types such as people, organizations or places in text. However, despite their high F1 performance, NER tools are still prone to brittleness due to their highly specialized and constrained input and training data. Thus, each tool is able to extract only a subset of the named entities (NE) mentioned in a given text. In order to improve \emph{NE Coverage}, we propose a hybrid approach, where we first aggregate the output of various NER tools and then validate and extend it through crowdsourcing. The results from our experiments show that this approach performs significantly better than the individual state-of-the-art tools (including existing tools that integrate individual outputs already). Furthermore, we show that the crowd is quite effective in (1) identifying mistakes, inconsistencies and ambiguities in currently used ground truth, as well as in (2) a promising approach to gather ground truth annotations for NER that capture a multitude of opinions.
Crowdsourcing ambiguity aware ground truth - collective intelligence 2017Lora Aroyo
The process of gathering ground truth data through human annotation is a major bottleneck in the use of information extraction methods. Crowdsourcing-based approaches are gaining popularity in the attempt to solve the issues related to the volume of data and lack of annotators. Typically these practices use inter-annotator agreement as a measure of quality. However, this assumption often creates issues in practice. Previous experiments we performed found that inter-annotator disagreement is usually never captured, either because the number of annotators is too small to capture the full diversity of opinion, or because the crowd data is aggregated with metrics that enforce consensus, such as majority vote. These practices create artificial data that is neither general nor reflects the ambiguity inherent in the data.
To address these issues, we proposed the method for crowdsourcing ground truth by harnessing inter-annotator disagreement. We present an alternative approach for crowdsourcing ground truth data that, instead of enforcing an agreement between annotators, captures the ambiguity inherent in semantic annotation through the use of disagreement-aware metrics for aggregating crowdsourcing responses. Based on this principle, we have implemented the CrowdTruth framework for machine-human computation, that first introduced the disagreement-aware metrics and built a pipeline to process crowdsourcing data with these metrics.
In this paper, we apply the CrowdTruth methodology to collect data over a set of diverse tasks: medical relation extraction, Twitter event identification, news event extraction and sound interpretation. We prove that capturing disagreement is essential for acquiring a high-quality ground truth. We achieve this by comparing the quality of the data aggregated with CrowdTruth metrics with a majority vote, a method which enforces consensus among annotators. By applying our analysis over a set of diverse tasks we show that, even though ambiguity manifests differently depending on the task, our theory of inter-annotator disagreement as a property of ambiguity is generalizable.
Truth is a Lie: 7 Myths about Human Annotation @CogComputing Forum 2014Lora Aroyo
Big data is having a disruptive impact across the sciences.
Human annotation of semantic interpretation tasks is a critical
part of big data semantics, but it is based on an antiquated
ideal of a single correct truth that needs to be similarly
disrupted.We expose seven myths about human annotation,
most of which derive from that antiquated ideal of truth,
and dispell these myths with examples from our research.We
propose a new theory of truth, Crowd Truth, that is based
on the intuition that human interpretation is subjective, and
that measuring annotations on the same objects of interpretation (in our examples, sentences) across a crowd will provide a useful representation of their subjectivity and the range of reasonable interpretations.
Crowds & Niches Teaching Machines to Diagnose: NLeSC Kick off eHumanities pr...Lora Aroyo
This presentation was given at the NL eSchience Center during the "De Geest Uit De Fles" event for the kick off of eHumanities project in 2014:
http://esciencecenter.nl/agenda/703-26-may-de-geest-uit-de-fles/
"Why the Semantic Web will Never Work" (note the quotes)James Hendler
This talk refutes some criticisms of the semantic web, but also outlines some research challenges we must overcome if we are to ever realize Tim Berners-Lee's original Semantic Web vision.
Utilizing Social Health Websites for Cognitive Computing and Clinical Decisio...CrowdTruth
Crowdsourced annotations data offers cognitive computing systems insights in lay semantics. This is especially important in health care, where medical terminology is often not aligned with patients `lay' language. However, the general crowd often has limited medical knowledge. Therefore this research investigated the opportunities of social health websites for obtaining ground truth annotations data for cognitive computing systems including clinical decision support systems. By identifying these websites and analyzing their data, it offers a starting point for the future utilization of user-generated health content for cognitive systems. However, the opportunities of social health data are currently limited by various legal regulations. Therefore this paper also dwells on the legal aspects of implementing social health data for cognitive computing systems.
Introduction to Data Science Talk Given to Girl Develop It! Central VA members
Note: some slides had animations in Excel, so unfortunately, the images overlap on the SlideShare version.
On Beyond OWL: challenges for ontologies on the WebJames Hendler
The need for ontologies in the real world is manifest and increasing. On the Web, ontologies are everywhere — but OWL isn’t. In this talk, I look at some of the things that are not in OWL, but which are needed for the use of OWL in many Web domains. This talk explores some of the needs for ontologies on the Web in data integration, emerging technologies, and linked data applications – and asks where the features needed for these are in OWL. The talk ends with some challenges to the OWL, and greater ontology, community needed to see more eventual use of standard ontologies on the Web.
Explore dozens case studies from a wide variety of institutions showcasing how WordPress is being used in higher ed: from course catalogs to permission systems, event management to LMS, commerce to academic publications, directories to food management, digital signage to internationalization. Interspersed are the results from a 486 respondent higher ed survey conducted in May 2016.
Case Studies come from interviews with MIT, Stanford, Boise State, Columbia, Boston University, Harvard, University of Washington, Conroe School District and many, many more.
"Why the Semantic Web will Never Work" (note the quotes)James Hendler
This talk refutes some criticisms of the semantic web, but also outlines some research challenges we must overcome if we are to ever realize Tim Berners-Lee's original Semantic Web vision.
Utilizing Social Health Websites for Cognitive Computing and Clinical Decisio...CrowdTruth
Crowdsourced annotations data offers cognitive computing systems insights in lay semantics. This is especially important in health care, where medical terminology is often not aligned with patients `lay' language. However, the general crowd often has limited medical knowledge. Therefore this research investigated the opportunities of social health websites for obtaining ground truth annotations data for cognitive computing systems including clinical decision support systems. By identifying these websites and analyzing their data, it offers a starting point for the future utilization of user-generated health content for cognitive systems. However, the opportunities of social health data are currently limited by various legal regulations. Therefore this paper also dwells on the legal aspects of implementing social health data for cognitive computing systems.
Introduction to Data Science Talk Given to Girl Develop It! Central VA members
Note: some slides had animations in Excel, so unfortunately, the images overlap on the SlideShare version.
On Beyond OWL: challenges for ontologies on the WebJames Hendler
The need for ontologies in the real world is manifest and increasing. On the Web, ontologies are everywhere — but OWL isn’t. In this talk, I look at some of the things that are not in OWL, but which are needed for the use of OWL in many Web domains. This talk explores some of the needs for ontologies on the Web in data integration, emerging technologies, and linked data applications – and asks where the features needed for these are in OWL. The talk ends with some challenges to the OWL, and greater ontology, community needed to see more eventual use of standard ontologies on the Web.
Explore dozens case studies from a wide variety of institutions showcasing how WordPress is being used in higher ed: from course catalogs to permission systems, event management to LMS, commerce to academic publications, directories to food management, digital signage to internationalization. Interspersed are the results from a 486 respondent higher ed survey conducted in May 2016.
Case Studies come from interviews with MIT, Stanford, Boise State, Columbia, Boston University, Harvard, University of Washington, Conroe School District and many, many more.
A new(ish) perspective on knowledge management in small organisations, with a little bit of Frank Zappa and Superman 3 thrown in. Originally delivered at the NCVO Information Management Conference, London, Nov 2008.
PSH Mobile Voice 2016 Personal Virtual Assistants Are Not Enough?Paul Heirendt
Personal Virtual Assistants Are Not Enough? NO, the reality is that they only have part of the story and very little of the context required to understand the intent and model the user's behavior
Data Science with Human in the Loop @Faculty of Science #Leiden UniversityLora Aroyo
Software systems are becoming ever more intelligent and more useful, but the way we interact with these machines too often reveals that they don’t actually understand people. Knowledge Representation and Semantic Web focus on the scientific challenges involved in providing human knowledge in machine-readable form. However, we observe that various types of human knowledge cannot yet be captured by machines, especially when dealing with wide ranges of real-world tasks and contexts. The key scientific challenge is to provide an approach to capturing human knowledge in a way that is scalable and adequate to real-world needs. Human Computation has begun to scientifically study how human intelligence at scale can be used to methodologically improve machine-based knowledge and data management. My research is focusing on understanding human computation for improving how machine-based systems can acquire, capture and harness human knowledge and thus become even more intelligent. In this talk I will show how the CrowdTruth framework (http://crowdtruth.org) facilitates data collection, processing and analytics of human computation knowledge.
Some project links:
- http://controcurator.org/
- http://crowdtruth.org/
- http://diveproject.beeldengeluid.nl/
- http://vu-amsterdam-web-media-group.github.io/linkflows/
Opportunities and strategies for crowdsourcing in the cultural heritage sector (GLAMs) are the focus of this presentation by Olaf Janssen, project manager for the KB, National Library of the Netherlands
You’ll read what crowdsourcing is, what motivates people to spend their time & money on it and how it differs from old-school voluntary work.
You’ll also learn what added-value and advantages it can bring, compared to frequently mentioned downsides. Furthermore a number of tips for setting up and running successful crowdsourced projects are given.
Then we'll focus on crowdsourcing within the cultural heritage sector. We distinguish six forms of crowdsourcing within GLAMs, each illustrated by a number of examples.
Tallk given at #SXSW2019 in the Intelligent Future track as part of the Interactive Festival. We explain 3 frameworks for MachineEthics and how they affect the supervised and unsupervised methods, and the data engineering discipline.
The Rijksmuseum Collection as Linked DataLora Aroyo
Presentation at ISWC2018: http://iswc2018.semanticweb.org/sessions/the-rijksmuseum-collection-as-linked-data/ of our paper published originally in the Semantic Web Journal: http://www.semantic-web-journal.net/content/rijksmuseum-collection-linked-data-2
Many museums are currently providing online access to their collections. The state of the art research in the last decade shows that it is beneficial for institutions to provide their datasets as Linked Data in order to achieve easy cross-referencing, interlinking and integration. In this paper, we present the Rijksmuseum linked dataset (accessible at http://datahub.io/dataset/rijksmuseum), along with collection and vocabulary statistics, as well as lessons learned from the process of converting the collection to Linked Data. The version of March 2016 contains over 350,000 objects, including detailed descriptions and high-quality images released under a public domain license.
FAIRview: Responsible Video Summarization @NYCML'18Lora Aroyo
Presentation at the NYC Media Lab (NYCML2018). There is a growing demand for news videos online, with more consumers preferring to watch the news than read or listen to it. On the publisher side, there is a growing effort to use video summarization technology in order to create easy-to-consume previews (trailers) for different types of broadcast programs. How can we measure the quality of video summaries and their potential to misinform? This workshop will inform participants about automatic video summarization algorithms and how to produce more “representative” video summaries. The research presented is from the FAIRview project and is supported by the Digital News Innovation Fund (DNI Fund), which is part of the Google News Initiative.
DH Benelux 2017 Panel: A Pragmatic Approach to Understanding and Utilising Ev...Lora Aroyo
Lora Aroyo, Chiel van den Akker, Marnix van Berchum, Lodewijk
Petram, Gerard Kuys, Tommaso Caselli, Jacco van Ossenbruggen, Victor de Boer, Sabrina Sauer, Berber Hagedoorn
Crowdsourcing & Nichesourcing: Enriching Cultural Heritagewith Experts & Cr...Lora Aroyo
Presentation at the "Past, Present and Future of Digital Humanities & Social Sciences in the Netherlands" event, http://www.ehumanities.nl/past-present-and-future-of-digital-humanities-social-sciences-in-the-netherlands-programme-and-abstracts-2/
Stitch by Stitch: Annotating Fashion at the RijksmuseumLora Aroyo
https://www.rijksmuseum.nl/en/stitch-by-stitch
http://annotate.accurator.nl/
Fashion can be found everywhere in museums. Fashion heritage collected over centuries: costumes, accessories, paintings, prints and photographs. But while some clothes and accessories are easily found and identified, others are obscure and require a trained eye to describe. What are we looking at? What kind of sleeve is this? Which materials and techniques have been used? More specific descriptions of the images facilitate better use of digital collections and enable users to wander through them in detail.
Museums & the Web 2016 Presentation: Enriching Collections with Expert Knowle...Lora Aroyo
http://mw2016.museumsandtheweb.com/proposal/accurator-enriching-collections-with-expert-knowledge-from-the-crowd/
Crowdsourcing is not a new phenomenon for museums. There are good examples for museums (e.g., Powerhouse museum, steve.museum). But not all crowdsourcing initiatives are successful. Crowdsourced tagging does not always contribute to a better understanding of art and can even be confusing.
The Rijksmuseum and the VU University Amsterdam developed the Accurator: a visual tool to get experts in domains like birds, bibles, ships, castles, etc. involved in annotating art and enrich the museums’ metadata with expertise that is not available internally.
In this how-to session, we demonstrate the tool and the ways other museums can implement this Open Web application for their own collections.
Connector Corner: Automate dynamic content and events by pushing a buttonDianaGray10
Here is something new! In our next Connector Corner webinar, we will demonstrate how you can use a single workflow to:
Create a campaign using Mailchimp with merge tags/fields
Send an interactive Slack channel message (using buttons)
Have the message received by managers and peers along with a test email for review
But there’s more:
In a second workflow supporting the same use case, you’ll see:
Your campaign sent to target colleagues for approval
If the “Approve” button is clicked, a Jira/Zendesk ticket is created for the marketing design team
But—if the “Reject” button is pushed, colleagues will be alerted via Slack message
Join us to learn more about this new, human-in-the-loop capability, brought to you by Integration Service connectors.
And...
Speakers:
Akshay Agnihotri, Product Manager
Charlie Greenberg, Host
Accelerate your Kubernetes clusters with Varnish CachingThijs Feryn
A presentation about the usage and availability of Varnish on Kubernetes. This talk explores the capabilities of Varnish caching and shows how to use the Varnish Helm chart to deploy it to Kubernetes.
This presentation was delivered at K8SUG Singapore. See https://feryn.eu/presentations/accelerate-your-kubernetes-clusters-with-varnish-caching-k8sug-singapore-28-2024 for more details.
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
UiPath Test Automation using UiPath Test Suite series, part 4DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 4. In this session, we will cover Test Manager overview along with SAP heatmap.
The UiPath Test Manager overview with SAP heatmap webinar offers a concise yet comprehensive exploration of the role of a Test Manager within SAP environments, coupled with the utilization of heatmaps for effective testing strategies.
Participants will gain insights into the responsibilities, challenges, and best practices associated with test management in SAP projects. Additionally, the webinar delves into the significance of heatmaps as a visual aid for identifying testing priorities, areas of risk, and resource allocation within SAP landscapes. Through this session, attendees can expect to enhance their understanding of test management principles while learning practical approaches to optimize testing processes in SAP environments using heatmap visualization techniques
What will you get from this session?
1. Insights into SAP testing best practices
2. Heatmap utilization for testing
3. Optimization of testing processes
4. Demo
Topics covered:
Execution from the test manager
Orchestrator execution result
Defect reporting
SAP heatmap example with demo
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
Key Trends Shaping the Future of Infrastructure.pdfCheryl Hung
Keynote at DIGIT West Expo, Glasgow on 29 May 2024.
Cheryl Hung, ochery.com
Sr Director, Infrastructure Ecosystem, Arm.
The key trends across hardware, cloud and open-source; exploring how these areas are likely to mature and develop over the short and long-term, and then considering how organisations can position themselves to adapt and thrive.
Securing your Kubernetes cluster_ a step-by-step guide to success !KatiaHIMEUR1
Today, after several years of existence, an extremely active community and an ultra-dynamic ecosystem, Kubernetes has established itself as the de facto standard in container orchestration. Thanks to a wide range of managed services, it has never been so easy to set up a ready-to-use Kubernetes cluster.
However, this ease of use means that the subject of security in Kubernetes is often left for later, or even neglected. This exposes companies to significant risks.
In this talk, I'll show you step-by-step how to secure your Kubernetes cluster for greater peace of mind and reliability.
Neuro-symbolic is not enough, we need neuro-*semantic*Frank van Harmelen
Neuro-symbolic (NeSy) AI is on the rise. However, simply machine learning on just any symbolic structure is not sufficient to really harvest the gains of NeSy. These will only be gained when the symbolic structures have an actual semantics. I give an operational definition of semantics as “predictable inference”.
All of this illustrated with link prediction over knowledge graphs, but the argument is general.
Elevating Tactical DDD Patterns Through Object CalisthenicsDorra BARTAGUIZ
After immersing yourself in the blue book and its red counterpart, attending DDD-focused conferences, and applying tactical patterns, you're left with a crucial question: How do I ensure my design is effective? Tactical patterns within Domain-Driven Design (DDD) serve as guiding principles for creating clear and manageable domain models. However, achieving success with these patterns requires additional guidance. Interestingly, we've observed that a set of constraints initially designed for training purposes remarkably aligns with effective pattern implementation, offering a more ‘mechanical’ approach. Let's explore together how Object Calisthenics can elevate the design of your tactical DDD patterns, offering concrete help for those venturing into DDD for the first time!
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024Albert Hoitingh
In this session I delve into the encryption technology used in Microsoft 365 and Microsoft Purview. Including the concepts of Customer Key and Double Key Encryption.
2. Web & Media Group
http://lora-aroyo.org @laroyo
Bulgaria
The Netherlands
Sofia
NYC
Personal
Semantics
3. Web & Media Group
http://lora-aroyo.org @laroyo
Riva del Garda, Italy, 2014
Semantic
Social Life
4. Web & Media Group
http://lora-aroyo.org @laroyo
4
To understand the value of
Semantic Web for e-learning
you have to understand people,
e.g. how they learn, interact &
consume information
5. Web & Media Group
http://lora-aroyo.org @laroyo
5
To understand the value of
Semantic Web for e-learning
you have to understand people,
e.g. how they interact &
consume information
6. Web & Media Group
http://lora-aroyo.org @laroyo
6
To understand the value of Semantic Web
for cultural heritage
you have to understand people, e.g.
how they interact & consume information
7. Web & Media Group
http://lora-aroyo.org @laroyo
7
To understand the value of Semantic Web
for cultural heritage
you have to understand people, e.g.
how they interact & consume information
8. Web & Media Group
http://lora-aroyo.org @laroyo
To understand the value of Semantic Web
for digital humanities, you have to
understand people, e.g.
how they interact & consume information
9. Web & Media Group
http://lora-aroyo.org @laroyo
people are in the center of everything
people & their semantics, i.e. their real-world behavior,
online interactions, information needs, information
consumption habits, personal preferences ...
10. Web & Media Group
http://lora-aroyo.org @laroyo
CrowdTruth team
12. http://lora-aroyo.org @laroyo
50’AI more or less begins
......
80’expert systems
90’knowledge acquisition from experts
00’standards & interoperability
10’big data & large crowds
A long time ago
in a galaxy far, far away …
14. http://lora-aroyo.org @laroyo
Advances in hardware and SDEs
PCs, workstations, Symbolics, Sun
New architectures like the Hypercube
LISP, Prolog, OPS
AI can now BUILD SYSTEMS
Primary focus on experts and rules
What is the knowledge of experts
What is the form of this knowledge?
Graphs, logic, rules, frames
How do experts reason?
Deduction, induction
80’s - empire of the experts
Work on form & process remained
academic
what happened inside the system, to
make the reasoning inside the system
proper and as good as possible
industry forged ahead with ad-hoc
& proprietary systems and actually
tried to build expert systems
Originals of uncertain KR
Fuzzy, probabilistic
15. http://lora-aroyo.org @laroyo
Piero Bonissone and the
DELTA/CATS expert system for
locomotive repair with David Smith, a
locomotive repair expert
Buchanan and Shortliff’s MYCIN project at
Stanford built an huge rule base for medicat
diagnosis working with an extensive team of
medical experts.
18. http://lora-aroyo.org @laroyo
90’s - knowledge acquisition from experts
The 90’s brought [attention for] knowledge acquisition.
Knowing that expert systems by then can functionally work, the focus [in
practice as well as scientific research and technology development] shifted
to the then-bigger challenge of how to acquire knowledge in real-world
scenarios.
It seems natural that after the look inside the systems, then one needed
to pay attention to how actually get the knowledge from the world outside
and frame it into the proper structured knowledge for inside the system.
Dream of the 90’s
26. Web & Media Group
http://lora-aroyo.org @laroyo
27. Web & Media Group
http://lora-aroyo.org @laroyo
the semantic
comfort
zone
28. Web & Media Group
http://lora-aroyo.org @laroyo
One truth: knowledge acquisition for the semantic web
assumes one correct interpretation for every example
All examples are created equal: triples are triples, one is not
more important than another, they are all either true or false
Disagreement bad: when people disagree, they don’t
understand the problem
Experts rule: knowledge is captured from domain experts
One is enough: knowledge by a single expert is sufficient
Detailed explanations help: if examples cause disagreement
- add instructions
Once done, forever valid: knowledge is not updated; new
data not aligned with old
“Truth is a Lie: 7 Myths about Human Annotation”, AI Magazine 2014, L. Aroyo, C. Welty
29. Web & Media Group
http://lora-aroyo.org @laroyo
Use Case:
video archive
enrichment
Search Behavior of Media Professionals at an Audiovisual Archive:
A Transaction Log Analysis (2009).
B. Huurnink, L. Hollink, W. van den Heuvel, M. de Rijke.
30. Web & Media Group
http://lora-aroyo.org @laroyo
Use Case:
video archive
enrichment
Goal:
make the
multimedia content of
Dutch National Video Archive
accessible to large audiences
Comfort Zone Solution:
media professionals watch & annotate videos. Of course!
31. Web & Media Group
http://lora-aroyo.org @laroyo
but ...
Expensive
Doesn’t scale
time-consuming
5 times the video duration
professional vocabulary
experts use a specific vocabulary
that is unknown to general audiences
32. Web & Media Group
http://lora-aroyo.org @laroyo
… and
people search for fragments
experts annotate full videos
not finding
35% of search queries result in not found
33. Web & Media Group
http://lora-aroyo.org @laroyo
Use Case:
real world QA
for Watson
Crowdsourcing ground truth for Question Answering using CrowdTruth (2015).
B Timmermans, L Aroyo, C Welty
34. Web & Media Group
http://lora-aroyo.org @laroyo
Goal:
gather questions
that real people ask
for training & evaluating Watson
Data:
30K Questions + Candidate Answers.
from Yahoo! Answers
Comfort Zone Solution:
ask people if the passage answers the question (Y/N). Simple!
Use Case:
real world QA
for Watson
35. Web & Media Group
http://lora-aroyo.org @laroyo
Contradicting evidence
Is Coral a plant?
• “Coral almost could be considered half-plant [..]”
• “[..] organism, such as a coral, resembling a stony plant.”
Unanswerable questions
• Can I take a pill if you don't have a child yet?
• Is the spelling for being drunk right?
• Is napster black?
Unclear answer type
Is paper animal plant or man made?
Multiple right answers to a question
What is the best university in NY? (subjective)
YES or NO?
36. Web & Media Group
http://lora-aroyo.org @laroyo
Use Case:
medical relation
extraction
for Watson
Crowdsourcing Ground Truth for Medical Relation Extraction (2017).
A Dumitrache, L Aroyo, C Welty
37. Web & Media Group
http://lora-aroyo.org @laroyo
Goal:
gather data to train
Watson to read
medical text & automatically
extract a medical relations KB
Comfort Zone Solution:
having medical experts read & annotate examples
Use Case:
medical relation
extraction
for Watson
38. Web & Media Group
http://lora-aroyo.org @laroyo
ANTIBIOTICS are the first line treatment for
indications of TYPHUS.
treats(ANTIBIOTICS, TYPHUS)? Expert: yes
Patients with TYPHUS who were given ANTIBIOTICS
exhibited side-effects.
treats(ANTIBIOTICS, TYPHUS)? Expert: yes
With ANTIBIOTICS in short supply, DDT was used
during WWII to control the insect vectors of
TYPHUS.
treats(ANTIBIOTICS, TYPHUS)? Expert: yes.
Are these three really all the same???
39. Web & Media Group
http://lora-aroyo.org @laroyo
Use Case:
map music to moods
40. Web & Media Group
http://lora-aroyo.org @laroyo
Use Case:
map music to moods
Goal:
annotate songs with emotional tags
Comfort Zone Solution:
people assign the prevalent mood of a song
41. Cluster 1 Cluster 2 Cluster 3 Cluster 4 Cluster 5 Other
passionate, rollicking, literate, humorous, silly, aggressive, fiery, does not fit into
rousing, cheerful, fun, poignant, wistful, campy, quirky, tense, anxious, any of the 5
confident, sweet, amiable, bittersweet, whimsical, witty, intense, volatile, clusters
boisterous, good-natured autumnal, wry visceral
rowdy brooding
Choose one:
Which is the mood most appropriate
for each song?
Goal:
(Lee and Hu 2012)
1 song - 1 mood???
42. Web & Media Group
http://lora-aroyo.org @laroyo
One truth: knowledge acquisition for the semantic web
assumes one correct interpretation for every example
All examples are created equal: triples are triples, one is not
more important than another, they are all either true or false
Disagreement bad: when people disagree, they don’t
understand the problem
Experts rule: knowledge is captured from domain experts
One is enough: knowledge by a single expert is sufficient
Detailed explanations help: if examples cause disagreement
- add instructions
Once done, forever valid: knowledge is not updated; new
data not aligned with old
“Truth is a Lie: 7 Myths about Human Annotation”, AI Magazine 2014, L. Aroyo, C. Welty
43. Web & Media Group
http://lora-aroyo.org @laroyo
One truth: knowledge acquisition for the semantic web
assumes one correct interpretation for every example
All examples are created equal: triples are triples, one is not
more important than another, they are all either true or false
Disagreement bad: when people disagree, they don’t
understand the problem
Experts rule: knowledge is captured from domain experts
One is enough: knowledge by a single expert is sufficient
Detailed explanations help: if examples cause disagreement
- add instructions
Once done, forever valid: knowledge is not updated; new
data not aligned with old
“Truth is a Lie: 7 Myths about Human Annotation”, AI Magazine 2014, L. Aroyo, C. Welty
Semantic
Comfort Zone
44. Web & Media Group
http://lora-aroyo.org @laroyo
One truth: knowledge acquisition for the semantic web
assumes one correct interpretation for every example
All examples are created equal: triples are triples, one is not
more important than another, they are all either true or false
Disagreement bad: when people disagree, they don’t
understand the problem
Experts rule: knowledge is captured from domain experts
One is enough: knowledge by a single expert is sufficient
Detailed explanations help: if examples cause disagreement
- add instructions
Once done, forever valid: knowledge is not updated; new
data not aligned with old
“Truth is a Lie: 7 Myths about Human Annotation”, AI Magazine 2014, L. Aroyo, C. Welty
Semantic
Comfort Zone
disrupted
45. Web & Media Group
http://lora-aroyo.org @laroyo
46. Web & Media Group
http://lora-aroyo.org @laroyo
interestingly …
47. Web & Media Group
http://lora-aroyo.org @laroyo
• collective decisions of large groups
of people
• a group of error-prone
decision-makers can be surprisingly
good at picking the best choice
• when thumbs up or thumbs down - the
chance of picking the right answer
needs to be > 50%
• the odds that a most of them will pick
the right answer is greater than any of
them will pick it on their own
• performance gets better as size grows
1785
Marquis de Condorcet
“wisdom of crowds”
48. Web & Media Group
http://lora-aroyo.org @laroyo
•asked 787 people to
guess the weight of
an ox
•none got the right
answer
•their collective guess
was almost perfect
1906
Sir Francis Galton
“wisdom of crowds”
49. Web & Media Group
http://lora-aroyo.org @laroyo
WWII Math Rosies
1942: Ballistics calculations and flight trajectories
50. Web & Media Group
http://lora-aroyo.org @laroyo
NASA’s Computer Room
transcribe raw flight data from celluloid film & oscillograph paper
51. Web & Media Group
http://lora-aroyo.org @laroyo
can we harness it?
53. http://lora-aroyo.org @laroyo
Web & Media Group
CrowdTruth
Three basic causes of disagreement: workers,
examples, target semantics
Disagreement is signal, not noise.
It is indicative of the variation in human semantic
interpretation
It can indicate ambiguity, vagueness, similarity,
over-generality, etc, as well as quality
Crowdtruth: Machine-human computation framework for harnessing disagreement
in gathering annotated data (2014)
O Inel, A Dumitrache, l.Aroyo, C. Welty
54. Web & Media Group
http://lora-aroyo.org @laroyo
one truth: multiple truths
all examples are created equal:
each example is unique
disagreement bad: disagreement is good
experts rule: crowd rules
one is enough: the more the better
detailed explanations help:
keep it simple stupid
once done, forever valid:
maintenance is necessary
“Truth is a Lie: 7 Myths about Human Annotation”, AI Magazine 2014, L. Aroyo, C. Welty
55. Web & Media Group
http://lora-aroyo.org @laroyo
changes needed
video archive
enrichment
improve support
for fragment search
time-based annotations
bridging vocabulary gap between
searcher & cataloguer
56. Web & Media Group
http://lora-aroyo.org @laroyo
crowdsourcing
video tagging
two
video tagging pilots
57. Web & Media Group
http://lora-aroyo.org @laroyo
@waisda
http://waisda.nl
engage
crowds
through
continuous
gaming
59. http://lora-aroyo.org @laroyo
Web & Media Group
time-based
bernhard
just “tags”
“On the Role of User-Generated Metadata in A/V Collections”, Riste Gligorov et al. KCAP2011
60. http://lora-aroyo.org @laroyo
Web & Media Group
objects (57%)
westminster abbey
abbey
priester
geestelijken
hek
paarden
tocht
aankomst
koets
kroning
mensenmassa
parade
kroon
regen
“On the Role of User-Generated Metadata in A/V Collections”, Riste Gligorov et al. KCAP2011
61. http://lora-aroyo.org @laroyo
Web & Media Group
persons (31%)
bernhard
juliana
objects (57%)
“On the Role of User-Generated Metadata in A/V Collections”, Riste Gligorov et al. KCAP2011
62. http://lora-aroyo.org @laroyo
Web & Media Group
user vocabulary
8% in professional vocabulary
23% in Dutch lexicon
89% found on Google
locations (7%)
engeland
locations (7%)
persons (31%)
objects (57%)
“On the Role of User-Generated Metadata in A/V Collections”, Riste Gligorov et al. KCAP2011
63. http://lora-aroyo.org @laroyo
Web & Media Group
user vocabulary
8% in professional vocabulary
23% in Dutch lexicon
89% found on Google
locations (7%)
describe mainly short segments
often not very specific
don’t describe programmes as a whole
“On the Role of User-Generated Metadata in A/V Collections”, Riste Gligorov et al. KCAP2011
user vocabulary
8% in professional vocabulary
23% in Dutch lexicon
89% found on Google
64. Web & Media Group
http://lora-aroyo.org @laroyo
crowdsourcing
medical relation
extraction
diversity of opinions
independent perspectives
multitude of contexts
we exposed a richer set of possibilities
that help in identifying, processing
& understanding context
65. Web & Media Group
http://lora-aroyo.org @laroyo
Does this sentence express
TREATS(Antibiotics, Typhus)?
Patients with TYPHUS who were given
ANTIBIOTICS exhibited several side-effects.
With ANTIBIOTICS in short supply, DDT was
used during World War II to control the insect
vectors of TYPHUS.
ANTIBIOTICS are the first line treatment for
indications of TYPHUS. 95%
75%
50%
The crowd results captures the natural ambiguity
66. http://lora-aroyo.org @laroyo
Web & Media Group
What is the relation between the highlighted terms?
He was the first physician to identify the relationship
between HEMOPHILIA and HEMOPHILIC ARTHROPATHY.
Experts Hallucinate
Crowd reads text literally - provide better examples to machine
experts: cause
crowd: no relation
67. http://lora-aroyo.org @laroyo
Web & Media Group
Unclear relationship between the two arguments reflected
in the disagreement
Medical Relation Extraction
68. http://lora-aroyo.org @laroyo
Web & Media Group
Clearly expressed relation between the two arguments reflected in
the agreement
Medical Relation Extraction
69. http://lora-aroyo.org @laroyo
Web & Media Group
Unclear relationship between the two arguments reflected
in the disagreement
Medical Relation Extraction
71. http://lora-aroyo.org @laroyo
Web & Media Group
Learning Curves
(crowd with pos./neg. threshold at 0.5)
above 400 sent.: crowd consistently over baseline & single
above 600 sent.: crowd out-performs experts
72. http://lora-aroyo.org @laroyo
Web & Media Group
Learning Curves Extended
(crowd with pos./neg. threshold at 0.5)
crowd consistently performs better than baseline
74. Web & Media Group
http://lora-aroyo.org @laroyo
Training a Relation Extraction Classifier
F1
Cost per
sentence
CrowdTruth 0.642 $0.66
Expert Annotator 0.638 $2.00
Single Annotator 0.492 $0.08
“wisdom of the crowd”
provides training data that is at least as good
if not better than experts
only with proper analytic framework for
harnessing disagreement from the crowd
75. http://lora-aroyo.org @laroyo
Web & Media Group
map music to moods
Goal:
tag songs with emotional clusters
Comfort Zone Solution:
people assign the prevalent mood of a song
76. Web & Media Group
http://lora-aroyo.org @laroyo
77. Web & Media Group
http://lora-aroyo.org @laroyo
Is this song ….
?Passionate
Rousing
Confident
Boisterous
Rowdy
Literate
Poignant
Wistful
Bittersweet
Autumnal
Brooding
Rollicking
Cheerful
Fun
Sweet
Amiable
Good-natured
Humorous
Silly
Campy
Whimsical
Witty
Wry
Aggressive
Fiery
Tense
Anxious
Intense
Volatile
80. Web & Media Group
http://lora-aroyo.org @laroyo
can indicate
alternative interpretations
Worker Mood-C1 Mood-C2 Mood-C3 Mood-C4 Mood-C5 Other
W10 1 1 1 1 1
Totals 3 5 6 5 2 8
Disagreement as Signal
can indicate
ambiguity in the
categorisation
can indicate
low quality workers
83. http://lora-aroyo.org @laroyo
Take Home Message
People first, experts second
True and False is not enough,
There is diversity in human interpretation
CrowdTruth introduces a spatial representation
of meaning that harnesses disagreement
With CrowdTruth untrained workers can be just as
reliable as highly trained experts