Combining human and computational intelligence for collaborative knowledge creation Elena Simperl Talk at the IEEE International Conference on Intelligent Computer Communication and Processing, Cluj-Napoca, Romania8/27/2011 www.insemtives.eu 1
Insemtives in a nutshell• Many aspects of semantic content authoring naturally rely on human contribution.• Motivating users to contribute is essential for semantic technologies to reach critical mass and ensure sustainable growth.• Insemtives works on – Best practices and guidelines for incentives-compatible technology design. – Enabling technology to realize incentivized semantic applications. – Showcased in three case studies: enterprise knowledge management; services marketplace; multimedia management within virtual worlds. www.insemtives.eu 2
Incentives and motivators• Motivation is the driving • Incentives can be related force that makes humans to both extrinsic and achieve their goals. intrinsic motivations.• Incentives are ‘rewards’ • Extrinsic motivation if assigned by an external task is considered boring, ‘judge’ to a performer for dangerous, useless, undertaking a specific socially undesirable, task. dislikable by the – Common belief (among performer. economists): incentives • Intrinsic motivation is can be translated into a sum of money for all driven by an interest or practical purposes. enjoyment in the task itself.
What is different about semantic systems?• Semantic Web tools vs applications. – Intelligent (specialized) Web sites (portals) with improved (local) search based on vocabularies and ontologies. – X2X integration (often combined with Web services). – Knowledge representation, communication and exchange.
What do you want your users to do?• Semantic applications – Context of the actual application. – Need to involve users in knowledge acquisition and engineering tasks? • Incentives are related to organizational and social factors. • Seamless integration of new features.• Semantic tools – Game mechanics. – Paid crowdsourcing (integrated).• Using results of casual games. http://gapingvoid.com/2011/06/07/pixie-dust-the-mountain-of-mediocrity/
Case studies• Methods applied – Mechanism design – Participatory design – Games with a purpose – Crowdsourcing via MTurk• Semantic content authoring scenarios – Extending and populating an ontology – Aligning two ontologies – Annotation of text, media and Web APIs
Mechanism design in practice• Identify a set of games that represents your situation.• See recommendations in the literature. • Translate what economists do into concrete scenarios. • Assure that the economists’ proposals fit to the concrete situation.• Run user and field experiments. Results influence HCI, social and data management aspects. 8/27/2011 www.insemtives.eu 15
Factors affecting mechanism design Social Nature of good Goal Tasks Structure being produced Communication High High level (about the Medium Variety of Medium Private goodgoal of the tasks) Low Low Hierarchy High High neutralParticipation level Medium Medium (in the definition Specificity of Public good of the goal) Low Low Identification High High Common resource with Low Clarity level Hierarchical Highly specific Low Required skills Club good Trivial/Common More at http://www.insemtives.eu/deliverables/INSEMTIVES_D1.3.1.pdf and http://www.insemtives.eu/deliverables/INSEMTIVES_D1.3.1.pdf 8/27/2011 www.insemtives.eu 16
Mechanism design for Telefonica• Interplay of two alternative games – Principal agent game • The management wants employees to do a certain action but does not have tools to check whether employees perform their best effort. • Various mechanisms can be used to align employees’ and employers’ interests – Piece rate wages (labour intensive tasks) – Performance measurement (all levels of tasks) – Tournaments (internal labour market) – Public goods • Semantic content creation is non-rival and non-excludable • The problem of free riding• Additional problem: what is the optimal time and effort for employees to dedicate to annotation4/14/11 www.insemtives.eu 18
Mechanism design for Telefonica (ii)• Principal agent game • Public goods game – Pay-per-performance – To let users know that their • Points assigned for each contribution was valuable contribution – The portal should be useful – Quality of performance • Possibility to search experts, measurement documents, etc. • Rate user contributions • Possibility to form groups of • Assign quality reviewers users and share contributions – Tournament – The portal should be easy to • Visibility of contributions by use single users • Search for an expert based on contributions • Experiments • Relative standing compared to – Pay-per-tag vs winner-takes- other users it-all for annotation.4/14/11 www.insemtives.eu 19
Knowledge engineering tasks• Granularity of ontology engineering activities is too broad; further splitting is needed• Crowdsource very specific tasks that are (highly) divisible – Labeling (in different languages) – Finding relationships – Populating the ontology – Aligning and interlinking – Ontology-based annotation – Validating the results of automatic methods – … www.insemtives.eu 20
OntoGame API• API that provides several methods that are shared by the OntoGame games, such as: – Different agreement types (e.g. selection agreement). – Input matching (e.g. , majority). – Game modes (multi-player, single player). – Player reliability evaluation. – Player matching (e.g., finding the optimal partner to play). – Resource (i.e., data needed for games) management. – Creating semantic content.• http://insemtives.svn.sourceforge.net/vie wvc/insemtives/generic-gaming-toolkit 8/27/2011 www.insemtives.eu 21
Lessons learned• Tasks which can be subject to games – Definition of vocabulary – Conceptualization • Based on competency questions • Identifying instances, classes, attributes, relationships – Documentation • Labeling and definitions • Localization – Evaluation and quality assurance • Matching conceptualization to documentation – Alignment – Validating the results of automatic methods• But, the approach is per design less applicable because – Knowledge-intensive tasks that are not easily nestable – Repetitive tasks players‘ retention? www.insemtives.eu 24
Lessons learned (ii)• Approach is feasible for mainstream domains, where a knowledge corpus is available• Knowledge corpus has to be large-enough to allow for a rich game experience – But you need a critical mass of players to validate the results• Advertisement is essential• Game design vs useful content – Reusing well-kwown game paradigms – Reusing game outcomes and integration in existing workflows and tools• Cost-benefit analysis