Exploiting disagreement through open ended tasks for capturing interpretation spaces

•

1 like•5,220 views

The document discusses using open-ended crowdsourcing tasks to capture interpretation spaces for multimedia data. It outlines challenges with existing approaches that assume a single ground truth and stimulate agreement. The methodology proposes using open-ended tasks without predefined answers to gather a range of interpretations. Preliminary results show capturing diverse sound interpretations through open-ended tasks to tag sounds. The conclusion is that open-ended tasks do not force agreement but can capture the full interpretation space for more complete multimedia data.

Science

Exploiting disagreement
through open- ended tasks for
capturing interpretation spaces
Doctoral Consortium
By /Benjamin Timmermans @8w

Outline
Introduction
State of the Art
Problem Statement
Methodology
Preliminary Results
Conclusions

For the training, testing and evaluation
of machines we rely on a...
ground "truth"

Crowdsourcing Approach
1-3 annotators
Evaluate workers
Inner-annotator agreement
Use test questions
Predeﬁned answer choices

The CrowdTruth Approach
10-15 annotators
Evaluate the input, annotations and workers
Disagreement-based analytics

Problems with multimedia annotations
Are sparse
Are homogeneous
Do not represent everything that can be heard or seen

Problems with crowdsourcing tasks
Are designed to stimulate agreement
Assumes answers are right or wrong

Closed task
How many beams do you see?
1
2
3
4
5
1 1
2
3
4
5

Open- ended tasks
How many beams do you see?

Gathering the interpretation space of multimedia through
open-ended crowdsourcing tasks
Goal
More efﬁcient crowdsourcing
Higher quality ground truth data
Improved search and discovery of multimedia

Are open-ended crowdsourcing tasks a feasible method for
capturing the interpretation space of multimedia?
Research Question

1. Improving quality evaluation
Comparing Closed and open-ended tasks
Measure worker conﬁdence

2. Improving open- ended task design
Combine constrains with open-ended designs
Showing known annotations
Detecting the distribution of answers

3. Applying the ground "truth"
Compare different contexts
Improve indexing of multimedia

Range of tasks
Passage Justiﬁcation
Passage Alignment
Distributional disambiguation

Sound Interpretations
2.133 short sounds
Top 5000 search terms = 11 mil. searches

Conclusions
There is no ultimate "truth"
Do not stimulate agreement
Capture the interpretation space
Use open-ended crowdsourcing tasks
Evaluation more difﬁcult

Who we are
Lora Aroyo Robert-Jan Sips Chris Welty
Oana Inel Anca Dumitrache Benjamin
Timmermans

Acknowledgements
Supervisor: Dr. Lora Aroyo
Mentor: Dr. Matteo Palmonari

CrowdTruth.org
Benjamin Timmermans
btimmermans.com
b.timmermans@vu.nl
@8w

Big data is having a disruptive impact across the sciences. Human annotation of semantic interpretation tasks is a critical part of big data semantics, but it is based on an antiquated ideal of a single correct truth that needs to be similarly disrupted.We expose seven myths about human annotation, most of which derive from that antiquated ideal of truth, and dispell these myths with examples from our research.We propose a new theory of truth, Crowd Truth, that is based on the intuition that human interpretation is subjective, and that measuring annotations on the same objects of interpretation (in our examples, sentences) across a crowd will provide a useful representation of their subjectivity and the range of reasonable interpretations.

Crowds & Niches Teaching Machines to Diagnose: NLeSC Kick off eHumanities pr...

Lora Aroyo

This document discusses using crowdsourcing to teach machines medical diagnosis by having crowds annotate medical text. It describes several natural language processing tasks needed for medical question answering like entity identification and relation extraction. It also discusses challenges like disagreement between annotators and how the CrowdTruth framework represents disagreement to help machines learn. Experts are motivated to annotate through a game format. The goal is a system that harnesses both crowds and experts to teach machines.

WebSci2013 Harnessing Disagreement in Crowdsourcing

Lora Aroyo

The document discusses harnessing disagreement in crowdsourcing for cognitive computing tasks like relation extraction. Typically, a single gold standard answer is assumed, but the authors argue that annotator disagreement is not just noise but a source of useful information. By capturing and understanding disagreement through frequencies and similarities, machine learning models can be scored based on how well their outputs fit within the space of possible human interpretations. This approach aims to better adapt models to new annotation tasks by tolerating the inherent vagueness and ambiguity of human understanding.

(Presentation Chris) Crowdsourcing & Semantic Web: Dagstuhl 2014

Lora Aroyo

The document discusses measuring quality in crowdsourced semantic interpretation tasks where there is disagreement between annotators. It introduces the concept of the "three sides of CrowdTruth" - representation, workers, and sentences. It shows that sentence quality and relation quality impact measurements of worker quality, and that considering these interdependencies can significantly improve worker metric accuracy in detecting spam annotators. Filtering both low-quality sentences and vague relations best separates high- and low-quality workers in evaluations.

Crowdsourcing & Semantic Web: Dagstuhl 2014 (Presentation Lora)

Lora Aroyo

Chapter 5 – quiz 5 instructions in most cases the topic area

nand15

This document provides instructions for a quiz with multiple choice and short answer questions about online privacy. It states that students must address all parts of each question, find a current event less than 4 years old relevant to one of the topics as a reference, and include an in-text citation and reference for the current event. Responses should be typed under each question. Only the first two questions will be graded. The document provides two sample questions about concerns over individual privacy due to online activities and search engines.

Chapter 8 comparing and contrasting computers and technology

nand15

The document provides guidance on comparing and contrasting topics for writing. It discusses real-world applications of comparison and contrast writing, such as comparing job offers or equipment purchases. The document then outlines how to structure a comparison and contrast essay, including developing a thesis, identifying elements to compare, using a block or point-by-point structure, and concluding logically. Specific examples provided relate to comparing technology options and virtual versus real classrooms.

Buttons on forms and surveys: a look at some research 2012

Caroline Jarrett

The document summarizes several studies on button placement in forms and surveys. The key findings are: 1) Buttons should be aligned with the left-hand side of text boxes as users' eyes naturally scan left to right. 2) A "surprise" layout with the Previous button on the right performed poorly in tests compared to more standard layouts. 3) While unusual layouts can be completed, they take longer and are less preferred by users compared to expected layouts with Previous on the left of Next. The best practice is to avoid non-standard button placements.

The document discusses using machine-crowd annotation workflows to improve event understanding across different domains and collections of data. It proposes using crowdsourcing with a disagreement-based approach to generate ground truth training data for natural language processing tools, and using the outputs of tools to optimize the crowdsourcing process. Preliminary results show the crowd performs nearly as well as experts at annotating events when provided specialized guidelines. A hybrid workflow combining entity extraction, crowdsourcing for events and linking, and cross-domain analysis is proposed to instantiate the methodology with different data types.

Visualization of Disagreement-based Quality Metrics of Crowdsourcing Data

CrowdTruth

Crowdsourcing represents a significant source of data which needs to be analyzed and interpreted. These tasks influence the quality of the output as well as the efficiency of the process. Visualization proved to be an effective way of dealing with large amount of data. In this paper we propose a visualization analytic model in the context of the CrowdTruth framework and CrowdTruth metrics for optimizing the crowdsourcing process and improving its data quality. The requirements for the dynamic, scalable and interactive visualizations were extracted through literature and interviews with users of the framework.

Crowdsourcing Disagreement on Open-Domain Questions

Benjamin Timmermans

The document discusses isotopes, which are variants of the same chemical element that have differing atomic weights due to variations in the number of neutrons in their nuclei. It provides several examples of passages that discuss isotopes and their properties. Specifically, it notes that isotopes of the same element have the same number of protons but different atomic weights, and that isotopes can be used as tracers in biological systems due to their similar chemical properties but different atomic weights. It also discusses how isotopes were discovered in the early 20th century by scientists like Frederick Soddy and Francis Aston.

Utilizing Social Health Websites for Cognitive Computing and Clinical Decisio...

CrowdTruth

Crowdsourced annotations data offers cognitive computing systems insights in lay semantics. This is especially important in health care, where medical terminology is often not aligned with patients `lay' language. However, the general crowd often has limited medical knowledge. Therefore this research investigated the opportunities of social health websites for obtaining ground truth annotations data for cognitive computing systems including clinical decision support systems. By identifying these websites and analyzing their data, it offers a starting point for the future utilization of user-generated health content for cognitive systems. However, the opportunities of social health data are currently limited by various legal regulations. Therefore this paper also dwells on the legal aspects of implementing social health data for cognitive computing systems.

Gamification of crowdsourcing tasks: What motivates a medical expert?

CrowdTruth

The document discusses motivations for medical experts to participate in crowdsourcing tasks to help train cognitive systems like Watson. It describes an initial game called Dr. Detective that medical experts enjoyed playing because it allowed learning new things, but lacked clear scoring and rewards. The document then outlines an improved game design called Dr. Watson that incorporates various motivations identified in literature like personal growth, learning, status and helping patients. Medical experts tested a mockup of Dr. Watson and provided feedback that incorporating topic selection, difficulty personalization, unlockable rewards and tracking Watson's progress could further engage and motivate them to participate in crowdsourcing tasks.

Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)

Lora Aroyo

Towards Better Media Understanding and Searchability

oanainel

Dive+@ICTOpen2017

oanainel

DIVE+ is an exploratory search tool for digital humanities research in CLARIAH Media Suite. It provides an event-centric browser for linked historical data that links objects to events and entities and builds automatic storylines. It integrates access to heterogeneous cultural heritage collections, including over 15 million triples from sources like news broadcasts, scans of radio bulletins, images, and metadata. DIVE+ allows exploring media collections, enriching metadata with historical events, and collecting crowd perspectives to support research.

Dive+ NL eScience symposium 2015

CrowdTruth

The document discusses DIVE+, a tool that aims to address the issues of audiences feeling disconnected from the massive amounts of digital cultural content available by bringing them back into the driving seat of exploration. It does this through an event-centric browser called DIVE that allows linking objects through events and collecting diverse perspectives from crowdsourcing to support different points of view. The goal is to shift museums and archives from being mere inventories to places that engage users through event narratives and an infinity of exploration options in linked open data.

CrowdTruth Games @NLeSc eHumanities day 2015

Lora Aroyo

Boosting Named Entity Extraction through Crowdsourcing

oanainel

This document discusses using crowdsourcing to improve named entity extraction (NE) performance. It finds that current NE tools have low performance and disagreement when used individually or combined. The document proposes using crowdsourcing to correct NE tool outputs and identify ground truth errors. An experiment combines 5 NE tools on two datasets, finding individual tools have low recall but combined output is better. However, combining tools with crowdsourced corrections further improves performance by reducing false positives and negatives.

SXSW2017 @NewDutchMedia Talk: Exploration is the New Search

Lora Aroyo

DIVE Semantic Web Challenge Presentation

Victor de Boer

Kick-off meeting Linkflows project

Vrije Universiteit Amsterdam, IOS Press, Netherlands Institute for Sound & Vision

Genuine semantic publishing

Tobias Kuhn

The document discusses the vision of semantic publishing from 2001 and 2017. In 2001, the vision was that scientists would publish machine-readable semantic representations of their experimental results. However, in 2017 semantic publishing challenges focused on extracting semantics from existing papers rather than changing the publishing process. The document advocates for "genuine semantic publishing" where semantic representations are a primary component of published works, are machine interpretable, cover core claims, and are approved by authors. It also discusses related approaches like research objects, executable papers, and nanopublications.

Europeana GA 2016: Harnessing Crowds, Niches & Professionals in the Digital Age

Lora Aroyo

The document discusses harnessing crowds, niches, and professionals in the digital age. The key points are: - Software is becoming less important as data takes center stage; cultural institutions must know their data and crowds. - Different crowds have different expertise and abilities; nichesourcing can access specialized knowledge. - Crowdsourcing initiatives should be part of an overall strategy and integrated into existing systems. - Novel interactions and user-driven augmentations can empower users and align the digital and physical.

Truth is a Lie - 7 Myths of Human Annotation

Anca Dumitrache

This document appears to be notes from a presentation or paper on using crowdsourcing to collect ground truth data for tasks involving semantic interpretation, where annotator disagreement is common. It discusses developing a disagreement-aware crowdsourcing methodology and online platform to collect gold standard data for tasks like medical relation extraction while capturing levels of annotator disagreement. The goal is to improve machine learning models by factoring disagreement into performance evaluation.

Defining spacial representations for the meaning of sounds

Benjamin Timmermans

Harnessing the Power of Machines & Crowds for Event Extraction

oanainel

Stitch by Stitch: Annotating Fashion at the Rijksmuseum

Lora Aroyo

https://www.rijksmuseum.nl/en/stitch-by-stitch http://annotate.accurator.nl/ Fashion can be found everywhere in museums. Fashion heritage collected over centuries: costumes, accessories, paintings, prints and photographs. But while some clothes and accessories are easily found and identified, others are obscure and require a trained eye to describe. What are we looking at? What kind of sleeve is this? Which materials and techniques have been used? More specific descriptions of the images facilitate better use of digital collections and enable users to wander through them in detail.

2 Studies UX types should know about (Straub UXPA unconference13)

Kath Straub

This document summarizes two studies on usability testing. The first study examines how many users are needed to find most usability problems. It analyzes data from a test of 100 users and finds that testing 5 users can find 55-85% of problems, 10 users find 82-94% of problems, and 15 users find 90-97% of problems. The second study compares problems found through usability testing with 30 users versus expert reviews with 14 experts across 6 websites. It finds that only 14% of problems were identified by both users and experts, with users finding more navigation and content problems and experts finding more consistency and standards problems.

Manual Testing is Dead. Long Live Manual Testing

QASymphony

‘Manual testing’ as a term used to describe testing is extremely confusing. What exactly are they attempting to describe? What do people believe is happening? I believe ‘Manual testing’ displays a lack of consideration of the work, thinking, and effort that goes into testing. However, this depends on who is using the term and why. If testing consists of point and click, then ‘Manual testing’ may be apt. If testing consists of test ideas, prep and experimentation, I don’t believe the term ‘Manual testing’ suits. We need to address this and clarify there is no automation testing or manual testing. There is testing. This talk will be about testing and the need to be able to discuss it more clearly.

Viewers also liked

ESWC - PhD Symposium 2016

oanainel

Visualization of Disagreement-based Quality Metrics of Crowdsourcing Data

CrowdTruth

Crowdsourcing Disagreement on Open-Domain Questions

Benjamin Timmermans

Utilizing Social Health Websites for Cognitive Computing and Clinical Decisio...

CrowdTruth

Gamification of crowdsourcing tasks: What motivates a medical expert?

CrowdTruth

Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)

Lora Aroyo

Towards Better Media Understanding and Searchability

oanainel

Dive+@ICTOpen2017

oanainel

Dive+ NL eScience symposium 2015

CrowdTruth

CrowdTruth Games @NLeSc eHumanities day 2015

Lora Aroyo

Boosting Named Entity Extraction through Crowdsourcing

oanainel

SXSW2017 @NewDutchMedia Talk: Exploration is the New Search

Lora Aroyo

DIVE Semantic Web Challenge Presentation

Victor de Boer

Kick-off meeting Linkflows project

Vrije Universiteit Amsterdam, IOS Press, Netherlands Institute for Sound & Vision

Genuine semantic publishing

Tobias Kuhn

Europeana GA 2016: Harnessing Crowds, Niches & Professionals in the Digital Age

Lora Aroyo

Truth is a Lie - 7 Myths of Human Annotation

Anca Dumitrache

Defining spacial representations for the meaning of sounds

Benjamin Timmermans

Harnessing the Power of Machines & Crowds for Event Extraction

oanainel

Stitch by Stitch: Annotating Fashion at the Rijksmuseum

Lora Aroyo

Viewers also liked (20)

ESWC - PhD Symposium 2016

Visualization of Disagreement-based Quality Metrics of Crowdsourcing Data

Crowdsourcing Disagreement on Open-Domain Questions

Utilizing Social Health Websites for Cognitive Computing and Clinical Decisio...

Gamification of crowdsourcing tasks: What motivates a medical expert?

Truth is a Lie: Rules & Semantics from Crowd Perspectives (RR'2015 Keynote)

Towards Better Media Understanding and Searchability

Dive+@ICTOpen2017

Dive+ NL eScience symposium 2015

CrowdTruth Games @NLeSc eHumanities day 2015

Boosting Named Entity Extraction through Crowdsourcing

SXSW2017 @NewDutchMedia Talk: Exploration is the New Search

DIVE Semantic Web Challenge Presentation

Kick-off meeting Linkflows project

Genuine semantic publishing

Europeana GA 2016: Harnessing Crowds, Niches & Professionals in the Digital Age

Truth is a Lie - 7 Myths of Human Annotation

Defining spacial representations for the meaning of sounds

Harnessing the Power of Machines & Crowds for Event Extraction

Stitch by Stitch: Annotating Fashion at the Rijksmuseum

Similar to Exploiting disagreement through open ended tasks for capturing interpretation spaces

2 Studies UX types should know about (Straub UXPA unconference13)

Kath Straub

Manual Testing is Dead. Long Live Manual Testing

QASymphony

Ofquals reliability of results programme

Ofqual Slideshare

What does collecting better data mean, and how to achieve it?

Ray Poynter

Startup Weekend - Validate Your Idea, Crash Course in User Research

ingridod

This document provides guidance on conducting user research to validate a product idea. It emphasizes that qualitative user research is important to identify user behaviors, needs, and problems in order to reduce the risk of building the wrong product. It recommends understanding the problem from the user's perspective by asking "why" multiple times, conducting ethnographic research by observing users in their environment, and performing interviews without leading questions. The document also provides tips on recruiting test subjects, testing prototypes as early as possible, and tools for remote user research.

7027203.ppt

Praveen Kumar

The document discusses key aspects of data collection and analysis for monitoring and evaluation projects. It covers topics such as qualities of good data, data collection methods including questionnaires, sampling methods, and data analysis techniques. Specifically, it emphasizes that collecting adequate, timely and relevant data is essential for evaluation and that questionnaires must be designed carefully to obtain accurate information and address all relevant variables. It also highlights the importance of representative sampling to make reliable estimates about target populations.

Seeing the Forest for the Trees: Visual Problem Solving as a Design and Usabi...

Christina York

The document discusses using visual problem solving techniques when working with different client groups. It introduces general visualization methodologies and specific tactics for engaging stakeholders. Tactics described include "shock and awe" wallpapering, "splashing the pot" to spread out related information, and creating imperfect visuals to elicit feedback through techniques like "Where's Waldo?" The presentation emphasizes using visual deliverables to communicate outcomes to different client groups.

survey research

Omar Tamimi

This document provides an overview of survey research methods. It defines a survey as a research method that collects information from a sample group using standardized questionnaires or interviews. The key steps in conducting a survey are defined as: 1) defining the objective, 2) constructing the questionnaire, 3) determining the method, 4) testing the questionnaire, 5) administering the questionnaire, 6) analyzing the data, and 7) writing a report. The document also discusses types of survey questions, sampling procedures, data collection techniques, and how to analyze and discuss the results.

Principles of survey research

Omar Tamimi

Investigating Crowdsourcing as an Evaluation Method for (TEL) Recommender Sy...

Christoph Rensing

The document investigates using crowdsourcing as an evaluation method for recommender systems that recommend learning resources. Researchers generated recommendations from two algorithms (AScore and FolkRank) for climate change resources and evaluated them using a crowdsourced questionnaire. The results supported that AScore provided more relevant and novel resources than the baseline FolkRank algorithm, but not more diverse resources. The researchers conclude that crowdsourcing can evaluate recommender systems and plan to further analyze the collected data and improve the crowdsourcing evaluation concept.

Audience Research on a Dime - Nonprofit of Influence

Courtney Clark

You need it. You know you do. Audience research is a vital part of any project, but it’s often the first thing to be cut. “We know our audiences well enough,” they say. “We know what they want.” But is that true? No! Of course not! If we knew what audiences wanted, we’d have an excess of donations, volunteers, newsletter subscribers, and report readers, and we wouldn’t be having conversations about how to get audiences to act or increase awareness. During this session, you’ll learn about: - My favorite lean audience research methods and why they’re awesome - How to convince your boss that audience research is necessary - How to conduct audience research when you have zero resources By the end, you’ll have what you need to do some quick and dirty audience research and convince others that it’s necessary! Presented at the Nonprofit of Influence Conference (hosted by the Colorado Nonprofit Association).

IODA - The Promise & Perils of Narrative Research

Chris Fletcher

The document discusses measuring the impact of an organizational development program at a consulting firm. It outlines the discovery process used, including document review, interviews, an international program review, and workshops. Data was collected using a narrative capture method based on Kirkpatrick and Brinkerhoff frameworks to evaluate how well the training programs helped staff develop the necessary skills and achieved the organization's goals.

Rinse and Repeat : The Spiral of Applied Machine Learning

Anna Chaney

The document describes a four step process for improving machine learning performance through continuous evaluation and retraining. Step 1 involves assessing performance using human judgment. Step 2 optimizes operating thresholds. Step 3 retrains the model with examples from humans. Step 4 redeploys the updated model and repeats the process. The goal is to continuously refine models through an iterative "spiral" approach as performance gains diminish over time.

UI/UX Foundations - Research

Meg Kurdziolek

Research Writing Survey

Aiden Yeh

Validation and mechanism: exploring the limits of evaluation

Alan Dix

Talk at Evaluation, SummerPIT 2019, Aarhus University, 15th August 2019 https://alandix.com/academic/talks/PIT-2019-validation-and-mechanism/ Sometimes evaluation is straightforward. Perhaps our goal is to create a system in a well-understood environment that is fastest to use or with least errors. In this case, and if we believe design choices are effectively independent, then we can run a lab or in-situ study to compare design alternatives. However many things do not fit into this easy-to-evaluate category. Sometimes our goals or more diffuse or long term: sustainability, behavioural change, improving education. Sometimes the thing we wish to 'evaluate' is 'generative' such as toolkits or frameworks used by developers or designers to create systems that then are used by others. In these cases simple post-hoc 'try it and measure it' approaches to evaluation fail, or at best give partial results. However post-hoc evaluation is only one way to validate work – data (quantitative or qualitative) should be combined with an understanding of mechanism, how things work, in order to justify, generalise and innovate.

Writing surveys that work

rebeccaweiss

This document provides an overview of best practices for writing effective surveys and questionnaires. It discusses key concepts like the difference between surveys and censuses, and surveys and questionnaires. It outlines common issues like sampling, design, analysis, question wording, response methods, and question ordering that should be considered. Best practices are presented such as clearly defining the research goal, verifying that a survey is needed, pretesting the questionnaire, and getting feedback from others. The goal is to help people construct surveys that accurately measure constructs and avoid biases.

Writing surveys that work

rebeccaweiss

Peer reviews

Alexander Serebrenik

Usability Testing Basics: What's it All About? at Web SIG Cleveland

Carol Smith

Presented to Web SIG Cleveland on May 21, 2011 at Notre Dame College in South Euclid (Cleveland), Ohio. Learn all you need to get started: - Where you can conduct studies (does it have to be in a lab?) - Types of studies (RITE, think aloud, etc.) - Tips for recruiting participants - Tips for Interacting with participants without biasing the study - Preparing for the study (materials needed, forms, etc.) - Guidance for analyzing the study

Similar to Exploiting disagreement through open ended tasks for capturing interpretation spaces (20)

2 Studies UX types should know about (Straub UXPA unconference13)

Manual Testing is Dead. Long Live Manual Testing

Ofquals reliability of results programme

What does collecting better data mean, and how to achieve it?

Startup Weekend - Validate Your Idea, Crash Course in User Research

7027203.ppt

Seeing the Forest for the Trees: Visual Problem Solving as a Design and Usabi...

survey research

Principles of survey research

Investigating Crowdsourcing as an Evaluation Method for (TEL) Recommender Sy...

Audience Research on a Dime - Nonprofit of Influence

IODA - The Promise & Perils of Narrative Research

Rinse and Repeat : The Spiral of Applied Machine Learning

UI/UX Foundations - Research

Research Writing Survey

Validation and mechanism: exploring the limits of evaluation

Writing surveys that work

Peer reviews

Usability Testing Basics: What's it All About? at Web SIG Cleveland

Recently uploaded

waterlessdyeingtechnolgyusing carbon dioxide chemicalspdf

LengamoLAppostilic

THEMATIC APPERCEPTION TEST(TAT) cognitive abilities, creativity, and critic...

Abdul Wali Khan University Mardan,kP,Pakistan

hematic appreciation test is a psychological assessment tool used to measure an individual's appreciation and understanding of specific themes or topics. This test helps to evaluate an individual's ability to connect different ideas and concepts within a given theme, as well as their overall comprehension and interpretation skills. The results of the test can provide valuable insights into an individual's cognitive abilities, creativity, and critical thinking skills

Micronuclei test.M.sc.zoology.fisheries.

Aditi Bajpai

Unlocking the mysteries of reproduction: Exploring fecundity and gonadosomati...

AbdullaAlAsif1

The pygmy halfbeak Dermogenys colletei, is known for its viviparous nature, this presents an intriguing case of relatively low fecundity, raising questions about potential compensatory reproductive strategies employed by this species. Our study delves into the examination of fecundity and the Gonadosomatic Index (GSI) in the Pygmy Halfbeak, D. colletei (Meisner, 2001), an intriguing viviparous fish indigenous to Sarawak, Borneo. We hypothesize that the Pygmy halfbeak, D. colletei, may exhibit unique reproductive adaptations to offset its low fecundity, thus enhancing its survival and fitness. To address this, we conducted a comprehensive study utilizing 28 mature female specimens of D. colletei, carefully measuring fecundity and GSI to shed light on the reproductive adaptations of this species. Our findings reveal that D. colletei indeed exhibits low fecundity, with a mean of 16.76 ± 2.01, and a mean GSI of 12.83 ± 1.27, providing crucial insights into the reproductive mechanisms at play in this species. These results underscore the existence of unique reproductive strategies in D. colletei, enabling its adaptation and persistence in Borneo's diverse aquatic ecosystems, and call for further ecological research to elucidate these mechanisms. This study lends to a better understanding of viviparous fish in Borneo and contributes to the broader field of aquatic ecology, enhancing our knowledge of species adaptations to unique ecological challenges.

Topic: SICKLE CELL DISEASE IN CHILDREN-3.pdf

TinyAnderson

Describing and Interpreting an Immersive Learning Case with the Immersion Cub...

Leonel Morgado

Current descriptions of immersive learning cases are often difficult or impossible to compare. This is due to a myriad of different options on what details to include, which aspects are relevant, and on the descriptive approaches employed. Also, these aspects often combine very specific details with more general guidelines or indicate intents and rationales without clarifying their implementation. In this paper we provide a method to describe immersive learning cases that is structured to enable comparisons, yet flexible enough to allow researchers and practitioners to decide which aspects to include. This method leverages a taxonomy that classifies educational aspects at three levels (uses, practices, and strategies) and then utilizes two frameworks, the Immersive Learning Brain and the Immersion Cube, to enable a structured description and interpretation of immersive learning cases. The method is then demonstrated on a published immersive learning case on training for wind turbine maintenance using virtual reality. Applying the method results in a structured artifact, the Immersive Learning Case Sheet, that tags the case with its proximal uses, practices, and strategies, and refines the free text case description to ensure that matching details are included. This contribution is thus a case description method in support of future comparative research of immersive learning cases. We then discuss how the resulting description and interpretation can be leveraged to change immersion learning cases, by enriching them (considering low-effort changes or additions) or innovating (exploring more challenging avenues of transformation). The method holds significant promise to support better-grounded research in immersive learning.

Shallowest Oil Discovery of Turkiye.pptx

Gokturk Mehmet Dilci

molar-distalization in orthodontics-seminar.pptx

Anagha Prasad

Oedema_types_causes_pathophysiology.pptx

muralinath2

Remote Sensing and Computational, Evolutionary, Supercomputing, and Intellige...

University of Maribor

在线办理(salfor毕业证书)索尔福德大学毕业证毕业完成信一模一样

vluwdy49

学校原件一模一样【微信：741003700 】《(salfor毕业证书)索尔福德大学毕业证》【微信：741003700 】学位证，留信认证（真实可查，永久存档）原件一模一样纸张工艺/offer、雅思、外壳等材料/诚信可靠,可直接看成品样本，帮您解决无法毕业带来的各种难题！外壳，原版制作，诚信可靠，可直接看成品样本。行业标杆！精益求精，诚心合作，真诚制作！多年品质 ,按需精细制作，24小时接单,全套进口原装设备。十五年致力于帮助留学生解决难题，包您满意。本公司拥有海外各大学样板无数，能完美还原。 1:1完美还原海外各大学毕业材料上的工艺：水印，阴影底纹，钢印LOGO烫金烫银，LOGO烫金烫银复合重叠。文字图案浮雕、激光镭射、紫外荧光、温感、复印防伪等防伪工艺。材料咨询办理、认证咨询办理请加学历顾问Q/微741003700 【主营项目】一.毕业证【q微741003700】成绩单、使馆认证、教育部认证、雅思托福成绩单、学生卡等！二.真实使馆公证(即留学回国人员证明,不成功不收费) 三.真实教育部学历学位认证（教育部存档！教育部留服网站永久可查）四.办理各国各大学文凭(一对一专业服务,可全程监控跟踪进度) 如果您处于以下几种情况： ◇在校期间，因各种原因未能顺利毕业……拿不到官方毕业证【q/微741003700】 ◇面对父母的压力，希望尽快拿到； ◇不清楚认证流程以及材料该如何准备； ◇回国时间很长，忘记办理； ◇回国马上就要找工作，办给用人单位看； ◇企事业单位必须要求办理的 ◇需要报考公务员、购买免税车、落转户口 ◇申请留学生创业基金留信网认证的作用: 1:该专业认证可证明留学生真实身份 2:同时对留学生所学专业登记给予评定 3:国家专业人才认证中心颁发入库证书 4:这个认证书并且可以归档倒地方 5:凡事获得留信网入网的信息将会逐步更新到个人身份内，将在公安局网内查询个人身份证信息后，同步读取人才网入库信息 6:个人职称评审加20分 7:个人信誉贷款加10分 8:在国家人才网主办的国家网络招聘大会中纳入资料，供国家高端企业选择人才

SAR of Medicinal Chemistry 1st by dk.pdf

KrushnaDarade1

Phenomics assisted breeding in crop improvement

IshaGoswami9

As the population is increasing and will reach about 9 billion upto 2050. Also due to climate change, it is difficult to meet the food requirement of such a large population. Facing the challenges presented by resource shortages, climate change, and increasing global population, crop yield and quality need to be improved in a sustainable way over the coming decades. Genetic improvement by breeding is the best way to increase crop productivity. With the rapid progression of functional genomics, an increasing number of crop genomes have been sequenced and dozens of genes influencing key agronomic traits have been identified. However, current genome sequence information has not been adequately exploited for understanding the complex characteristics of multiple gene, owing to a lack of crop phenotypic data. Efficient, automatic, and accurate technologies and platforms that can capture phenotypic data that can be linked to genomics information for crop improvement at all growth stages have become as important as genotyping. Thus, high-throughput phenotyping has become the major bottleneck restricting crop breeding. Plant phenomics has been defined as the high-throughput, accurate acquisition and analysis of multi-dimensional phenotypes during crop growing stages at the organism level, including the cell, tissue, organ, individual plant, plot, and field levels. With the rapid development of novel sensors, imaging technology, and analysis methods, numerous infrastructure platforms have been developed for phenotyping.

The binding of cosmological structures by massless topological defects

Sérgio Sacani

Assuming spherical symmetry and weak field, it is shown that if one solves the Poisson equation or the Einstein field equations sourced by a topological defect, i.e. a singularity of a very specific form, the result is a localized gravitational field capable of driving flat rotation (i.e. Keplerian circular orbits at a constant speed for all radii) of test masses on a thin spherical shell without any underlying mass. Moreover, a large-scale structure which exploits this solution by assembling concentrically a number of such topological defects can establish a flat stellar or galactic rotation curve, and can also deflect light in the same manner as an equipotential (isothermal) sphere. Thus, the need for dark matter or modified gravity theory is mitigated, at least in part.

如何办理(uvic毕业证书)维多利亚大学毕业证本科学位证书原版一模一样

yqqaatn0

原版纸张【微信：741003700 】【(uvic毕业证书)维多利亚大学毕业证】【微信：741003700 】学位证，留信认证（真实可查，永久存档）offer、雅思、外壳等材料/诚信可靠,可直接看成品样本，帮您解决无法毕业带来的各种难题！外壳，原版制作，诚信可靠，可直接看成品样本。行业标杆！精益求精，诚心合作，真诚制作！多年品质 ,按需精细制作，24小时接单,全套进口原装设备。十五年致力于帮助留学生解决难题，包您满意。本公司拥有海外各大学样板无数，能完美还原海外各大学 Bachelor Diploma degree, Master Degree Diploma 1:1完美还原海外各大学毕业材料上的工艺：水印，阴影底纹，钢印LOGO烫金烫银，LOGO烫金烫银复合重叠。文字图案浮雕、激光镭射、紫外荧光、温感、复印防伪等防伪工艺。材料咨询办理、认证咨询办理请加学历顾问Q/微741003700 留信网认证的作用: 1:该专业认证可证明留学生真实身份 2:同时对留学生所学专业登记给予评定 3:国家专业人才认证中心颁发入库证书 4:这个认证书并且可以归档倒地方 5:凡事获得留信网入网的信息将会逐步更新到个人身份内，将在公安局网内查询个人身份证信息后，同步读取人才网入库信息 6:个人职称评审加20分 7:个人信誉贷款加10分 8:在国家人才网主办的国家网络招聘大会中纳入资料，供国家高端企业选择人才

Medical Orthopedic PowerPoint Templates.pptx

terusbelajar5

3D Hybrid PIC simulation of the plasma expansion (ISSS-14)

David Osipyan

Randomised Optimisation Algorithms in DAPHNE

University of Maribor

原版制作(carleton毕业证书)卡尔顿大学毕业证硕士文凭原版一模一样

yqqaatn0

原版纸张【微信：741003700 】【(carleton毕业证书)卡尔顿大学毕业证】【微信：741003700 】学位证，留信认证（真实可查，永久存档）offer、雅思、外壳等材料/诚信可靠,可直接看成品样本，帮您解决无法毕业带来的各种难题！外壳，原版制作，诚信可靠，可直接看成品样本。行业标杆！精益求精，诚心合作，真诚制作！多年品质 ,按需精细制作，24小时接单,全套进口原装设备。十五年致力于帮助留学生解决难题，包您满意。本公司拥有海外各大学样板无数，能完美还原海外各大学 Bachelor Diploma degree, Master Degree Diploma 1:1完美还原海外各大学毕业材料上的工艺：水印，阴影底纹，钢印LOGO烫金烫银，LOGO烫金烫银复合重叠。文字图案浮雕、激光镭射、紫外荧光、温感、复印防伪等防伪工艺。材料咨询办理、认证咨询办理请加学历顾问Q/微741003700 留信网认证的作用: 1:该专业认证可证明留学生真实身份 2:同时对留学生所学专业登记给予评定 3:国家专业人才认证中心颁发入库证书 4:这个认证书并且可以归档倒地方 5:凡事获得留信网入网的信息将会逐步更新到个人身份内，将在公安局网内查询个人身份证信息后，同步读取人才网入库信息 6:个人职称评审加20分 7:个人信誉贷款加10分 8:在国家人才网主办的国家网络招聘大会中纳入资料，供国家高端企业选择人才

Eukaryotic Transcription Presentation.pptx

RitabrataSarkar3

Recently uploaded (20)

waterlessdyeingtechnolgyusing carbon dioxide chemicalspdf

THEMATIC APPERCEPTION TEST(TAT) cognitive abilities, creativity, and critic...

Micronuclei test.M.sc.zoology.fisheries.

Unlocking the mysteries of reproduction: Exploring fecundity and gonadosomati...

Topic: SICKLE CELL DISEASE IN CHILDREN-3.pdf

Describing and Interpreting an Immersive Learning Case with the Immersion Cub...

Shallowest Oil Discovery of Turkiye.pptx

molar-distalization in orthodontics-seminar.pptx

Oedema_types_causes_pathophysiology.pptx

Remote Sensing and Computational, Evolutionary, Supercomputing, and Intellige...

在线办理(salfor毕业证书)索尔福德大学毕业证毕业完成信一模一样

SAR of Medicinal Chemistry 1st by dk.pdf

Phenomics assisted breeding in crop improvement

The binding of cosmological structures by massless topological defects

如何办理(uvic毕业证书)维多利亚大学毕业证本科学位证书原版一模一样

Medical Orthopedic PowerPoint Templates.pptx

3D Hybrid PIC simulation of the plasma expansion (ISSS-14)

Randomised Optimisation Algorithms in DAPHNE

原版制作(carleton毕业证书)卡尔顿大学毕业证硕士文凭原版一模一样

Eukaryotic Transcription Presentation.pptx

Exploiting disagreement through open ended tasks for capturing interpretation spaces

1. Exploiting disagreement through open- ended tasks for capturing interpretation spaces Doctoral Consortium By /Benjamin Timmermans @8w

2. Outline Introduction State of the Art Problem Statement Methodology Preliminary Results Conclusions

3. Introduction

5. How many dogs were in the picture?

6. There is no universal "truth"

7. For the training, testing and evaluation of machines we rely on a... ground "truth"

8. State of the Art

9. Crowdsourcing Approach 1-3 annotators Evaluate workers Inner-annotator agreement Use test questions Predeﬁned answer choices

10.

11. The CrowdTruth Approach 10-15 annotators Evaluate the input, annotations and workers Disagreement-based analytics

12. Problem Statement

13. Problems with multimedia annotations Are sparse Are homogeneous Do not represent everything that can be heard or seen

14. Problems with crowdsourcing tasks Are designed to stimulate agreement Assumes answers are right or wrong

15. Closed task How many beams do you see? 1 2 3 4 5 1 1 2 3 4 5

16. Open- ended tasks How many beams do you see?

17.

18. Gathering the interpretation space of multimedia through open-ended crowdsourcing tasks Goal More efﬁcient crowdsourcing Higher quality ground truth data Improved search and discovery of multimedia

19. Are open-ended crowdsourcing tasks a feasible method for capturing the interpretation space of multimedia? Research Question

20. Methodology

21. 1. Improving quality evaluation Comparing Closed and open-ended tasks Measure worker conﬁdence

22. 2. Improving open- ended task design Combine constrains with open-ended designs Showing known annotations Detecting the distribution of answers

23. 3. Applying the ground "truth" Compare different contexts Improve indexing of multimedia

24. Preliminary Results

25. Gathering training data for IBM Watson

26. Range of tasks Passage Justiﬁcation Passage Alignment Distributional disambiguation

27. Sound Interpretations 2.133 short sounds Top 5000 search terms = 11 mil. searches

28. Sound tag overlap

29. Conclusions There is no ultimate "truth" Do not stimulate agreement Capture the interpretation space Use open-ended crowdsourcing tasks Evaluation more difﬁcult

30. Who we are Lora Aroyo Robert-Jan Sips Chris Welty Oana Inel Anca Dumitrache Benjamin Timmermans

31. Acknowledgements Supervisor: Dr. Lora Aroyo Mentor: Dr. Matteo Palmonari

32. CrowdTruth.org Benjamin Timmermans btimmermans.com b.timmermans@vu.nl @8w

Exploiting disagreement through open ended tasks for capturing interpretation spaces

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (20)

Similar to Exploiting disagreement through open ended tasks for capturing interpretation spaces

Similar to Exploiting disagreement through open ended tasks for capturing interpretation spaces (20)

Recently uploaded

Recently uploaded (20)

Exploiting disagreement through open ended tasks for capturing interpretation spaces