Slides of the presentation given at the 22nd International Conference on the World Wide Web.
URL: http://www2013.org/program/561-reactive-crowdsourcing/
More information on the Crowdsearcher project available at
crowdsearcher.search-computing.com
Answering Search Queries with CrowdSearcher: a crowdsourcing and social netwo...Marco Brambilla
Web users are increasingly relying on social interaction to complete and validate the results of their search activities. While search systems are superior machines to get world-wide information, the opinions collected within friends and expert/local communities can ultimately determine our decisions: human curiosity and creativity is often capable of going much beyond the capabilities of search systems in scouting “interesting” results, or suggesting new, unexpected search directions. Such personalized interaction occurs in most times aside of the search systems and processes, possibly instrumented and mediated by a social network; when such interaction is completed and users resort to the use of search systems, they do it through new queries, loosely related to the previous search or to the social interaction.
In this paper we propose CrowdSearcher, a novel search paradigm that embodies crowds as first-class sources for the information seeking process. CrowdSearcher aims at filling the gap between generalized search systems, which operate upon world-wide information - including facts and recommendations as crawled and indexed by computerized systems – with social systems, capable of interacting with real people, in real time, to capture their opinions, suggestions, emotions. The technical contribution of this paper is the discussion of a model and architecture for integrating computerized search with human interaction, by showing how search systems can drive and encapsulate social systems. In particular we show how social platforms, such as Facebook, LinkedIn and Twitter, can be used for crowdsourcing search-related tasks; we demonstrate our approach with several prototypes and we report on our experiment upon real user communities.
Presentation for: Masterclass 19: Using social media in public engagement for the Public Engagement & Impact Team at The University of Sheffield, 26 November 2014.
Picturing the Social: Talk for Transforming Digital Methods Winter SchoolFarida Vis
This talk highlights the work of the Visual Social Media Lab and the Picturing the Social project. It summarises the key research questions and aims of the project. It highlights the value of interdisciplinarity and working closely with industry in this area. It also focuses on the way in which me might study different types of structures involved in the circulation and the scopic regimes that make social media images more or less visible. It also tries to unpack how we can start to think about APIs as 'method' and looks at the different ways in which we can get access to different kinds of social media image data. Both through public ('free') APIs and ('pay for') firehose data.
Answering Search Queries with CrowdSearcher: a crowdsourcing and social netwo...Marco Brambilla
Web users are increasingly relying on social interaction to complete and validate the results of their search activities. While search systems are superior machines to get world-wide information, the opinions collected within friends and expert/local communities can ultimately determine our decisions: human curiosity and creativity is often capable of going much beyond the capabilities of search systems in scouting “interesting” results, or suggesting new, unexpected search directions. Such personalized interaction occurs in most times aside of the search systems and processes, possibly instrumented and mediated by a social network; when such interaction is completed and users resort to the use of search systems, they do it through new queries, loosely related to the previous search or to the social interaction.
In this paper we propose CrowdSearcher, a novel search paradigm that embodies crowds as first-class sources for the information seeking process. CrowdSearcher aims at filling the gap between generalized search systems, which operate upon world-wide information - including facts and recommendations as crawled and indexed by computerized systems – with social systems, capable of interacting with real people, in real time, to capture their opinions, suggestions, emotions. The technical contribution of this paper is the discussion of a model and architecture for integrating computerized search with human interaction, by showing how search systems can drive and encapsulate social systems. In particular we show how social platforms, such as Facebook, LinkedIn and Twitter, can be used for crowdsourcing search-related tasks; we demonstrate our approach with several prototypes and we report on our experiment upon real user communities.
Presentation for: Masterclass 19: Using social media in public engagement for the Public Engagement & Impact Team at The University of Sheffield, 26 November 2014.
Picturing the Social: Talk for Transforming Digital Methods Winter SchoolFarida Vis
This talk highlights the work of the Visual Social Media Lab and the Picturing the Social project. It summarises the key research questions and aims of the project. It highlights the value of interdisciplinarity and working closely with industry in this area. It also focuses on the way in which me might study different types of structures involved in the circulation and the scopic regimes that make social media images more or less visible. It also tries to unpack how we can start to think about APIs as 'method' and looks at the different ways in which we can get access to different kinds of social media image data. Both through public ('free') APIs and ('pay for') firehose data.
Over the last few years we have observed the emergence of hybrid human-machine information systems which are able to both scale over large amount of data as well as to maintain high-quality data processing intrinsic in human intelligence.
In this talk I will focus on the use of human intelligence at scale by means of crowdsourcing to deal with Big Data problems. We will look specifically on how to deal with the variety in data by means of Human Computation still being able to operate with a large data volume.
First, I will introduce the area of micro-task crowdsourcing also providing an overview of different research challenges that needs to be tackled to enable large-scale hybrid human-machine information systems. Next, I will provide examples of such hybrid systems for entity linking and disambiguation using crowdsourcing and a graph of linked entities as background corpus. I will describe how keyword query understanding can be crowdsourced to build search engines that can answer rare complex queries. Finally, I will present new techniques that allow to improve the quality of crowdsourced information system components by means of push crowdsourcing.
For my final year project I used data analysis techniques to investigate user behavior pattern recognition in respect of similar interests and culture versus offline geographical location. This was an out-of-the-box topic, which I selected due to my love on Data Analysis, in respect of the Social Network Analysis in the Internet era.
Harnessing social signals to enhance a searchIsmail BADACHE
This paper describes an approach of information retrieval which takes into account social signals associated with Web resources to estimate its relevance to a query. We show how these data, which are in the form of actions within social activities (e.g. like, tweet), can be exploited to quantify social properties such as popularity and reputation. We propose a model that combines the social relevance, estimated from these properties, with the conventional textual relevance. We evaluated the effectiveness of our approach on IMDb dataset containing 32706 resources and their social characteristics collected from several social networks. We used also the selected criteria to learn models to determine their effectiveness in information retrieval. Our experimental results are promising and show the interest of integrating social signals in retrieval model to enhance a search.
Immersive Recommendation incorporates cross-platform and diverse personal digital traces into recommendations. Our context-aware topic modeling algorithm systematically profiles users' interests based on their traces from different contexts, and our hybrid recommendation algorithm makes high-quality recommendations by fusing users' personal profiles, item profiles, and existing ratings. The proposed model showed significant improvement over the state-of-the-art algorithms, suggesting the value of using this new user-centric recommendation model to improve recommendation quality, including in cold-start situations.
2014 TheNextWeb-Mapping connections with NodeXLMarc Smith
Slides from a talk at the 2014 TheNextWeb in Amsterdam.
NodeXL social media network analysis of Twitter reveals six common structures in Twitter networks.
Using Behaviour Analysis to Detect Cultural Aspects in Social Web SystemsMatthew Rowe
Presented at:
-Aston Business School, Birmingham, UK. 2011
-Keynote presentation at Detecting and Exploiting Cultural Diversity on the Social Web Workshop, 20th Annual Conference on Information and Knowledge Management 2011
Think Link: Network Insights with No Programming SkillsMarc Smith
Networks are everywhere, but the tools for end users to access, analyze, visualize and share insights into connected structures have been absent. NodeXL, the network overview discovery and exploration add-in for Excel makes network analysis as easy as making a pie chart.
Software Project Management Presentation FinalMinhas Kamal
Software Project Management- ResearchColab
Presented in 4th year of Bachelor of Science in Software Engineering (BSSE) course at Institute of Information Technology, University of Dhaka (IIT, DU).
Over the last few years we have observed the emergence of hybrid human-machine information systems which are able to both scale over large amount of data as well as to maintain high-quality data processing intrinsic in human intelligence.
In this talk I will focus on the use of human intelligence at scale by means of crowdsourcing to deal with Big Data problems. We will look specifically on how to deal with the variety in data by means of Human Computation still being able to operate with a large data volume.
First, I will introduce the area of micro-task crowdsourcing also providing an overview of different research challenges that needs to be tackled to enable large-scale hybrid human-machine information systems. Next, I will provide examples of such hybrid systems for entity linking and disambiguation using crowdsourcing and a graph of linked entities as background corpus. I will describe how keyword query understanding can be crowdsourced to build search engines that can answer rare complex queries. Finally, I will present new techniques that allow to improve the quality of crowdsourced information system components by means of push crowdsourcing.
For my final year project I used data analysis techniques to investigate user behavior pattern recognition in respect of similar interests and culture versus offline geographical location. This was an out-of-the-box topic, which I selected due to my love on Data Analysis, in respect of the Social Network Analysis in the Internet era.
Harnessing social signals to enhance a searchIsmail BADACHE
This paper describes an approach of information retrieval which takes into account social signals associated with Web resources to estimate its relevance to a query. We show how these data, which are in the form of actions within social activities (e.g. like, tweet), can be exploited to quantify social properties such as popularity and reputation. We propose a model that combines the social relevance, estimated from these properties, with the conventional textual relevance. We evaluated the effectiveness of our approach on IMDb dataset containing 32706 resources and their social characteristics collected from several social networks. We used also the selected criteria to learn models to determine their effectiveness in information retrieval. Our experimental results are promising and show the interest of integrating social signals in retrieval model to enhance a search.
Immersive Recommendation incorporates cross-platform and diverse personal digital traces into recommendations. Our context-aware topic modeling algorithm systematically profiles users' interests based on their traces from different contexts, and our hybrid recommendation algorithm makes high-quality recommendations by fusing users' personal profiles, item profiles, and existing ratings. The proposed model showed significant improvement over the state-of-the-art algorithms, suggesting the value of using this new user-centric recommendation model to improve recommendation quality, including in cold-start situations.
2014 TheNextWeb-Mapping connections with NodeXLMarc Smith
Slides from a talk at the 2014 TheNextWeb in Amsterdam.
NodeXL social media network analysis of Twitter reveals six common structures in Twitter networks.
Using Behaviour Analysis to Detect Cultural Aspects in Social Web SystemsMatthew Rowe
Presented at:
-Aston Business School, Birmingham, UK. 2011
-Keynote presentation at Detecting and Exploiting Cultural Diversity on the Social Web Workshop, 20th Annual Conference on Information and Knowledge Management 2011
Think Link: Network Insights with No Programming SkillsMarc Smith
Networks are everywhere, but the tools for end users to access, analyze, visualize and share insights into connected structures have been absent. NodeXL, the network overview discovery and exploration add-in for Excel makes network analysis as easy as making a pie chart.
Software Project Management Presentation FinalMinhas Kamal
Software Project Management- ResearchColab
Presented in 4th year of Bachelor of Science in Software Engineering (BSSE) course at Institute of Information Technology, University of Dhaka (IIT, DU).
This presentation was given to the Tech Change Technology for Monitoring and Evaluation Diploma course on 25th September 2015. It covers:
Why visualise data?
Where to start?
Which tools to use?
It ends with an overview of Kwantu's approach to this area and the technology choices that we've made.
Doing Analytics Right - Building the Analytics EnvironmentTasktop
Implementing analytics for development processes is challenging. As in discussed in the previous webinars, the right analytics are determined by the goals of the organization, not by the available data. So implementing your analytics solutions will require an efficient analytics and data architecture, including the ability to combine and stage data from heterogeneous sources. An architecture that excludes the ability to gain access to the necessary data will create a barrier to deploying your newly designed analytics program, and will force you back into the “light is brighter here” anti-pattern.
This webinar will describe the technical considerations of implementing the data architecture for your analytics program, and explain how Tasktop can help.
Serving tens of billions of personalized recommendations a day under a latency of 30 milliseconds is a challenge. In this talk I'll share our algorithmic architecture, including its Spark-based offline layer, and its Elasticsearch-based serving layer, that enable running complex models under difficult scale constrains and shorten the cycle between research and production.
Sonya Liberman leads the Personalization team @ Outbrain's Recommendations group, developing large-scale machine learning algorithms for Outbrain's content recommendations platform serving tens of billions real-time recommendations a day. She specializes in Information Retrieval, Machine Learning, and Computational Linguistics. Before joining Outbrain, she led the Research and Algorithms @ ConvertMedia (acquired by Taboola). She holds an MSc in Computer Science and a BSc in Computer Science and Computational Biology.
This invited talk was given at PyData Meetup, April 2019
https://www.meetup.com/PyData-Tel-Aviv/
Scaling Your Architecture with Services and EventsRandy Shoup
This session is a deep dive into the modern best practices around asynchronous decoupling, resilience, and scalability that allow us to implement a large-scale software system from the building blocks of events and services, based on the speaker's experiences implementing such systems at Google, eBay, and other high-performing technology organizations. We will outline the various options for handling event delivery and event ordering in a distributed system. We will cover data and persistence in an event-driven architecture. Finally, we will describe how to combine events, services, and so-called 'serverless' functions into a powerful overall architecture. You will leave with practical suggestions to help you accelerate your development velocity and drive business results.
Methods and Challenges for Metaverse Analytics.pdfSafaa Alnabulsi
Which existing methods and analytical approaches can be applied to quantitatively study metaverse?
Which challenges are associated with the quantitative investigation of metaverse and the application of those methods?
Presented at EuroIA17, September 2017; World IA Day NYC, February 2017; Interact, October 2016 (London, UK); earlier versions in 2014 at UXPA Boston (Boston, MA, USA); in 2013 at Interaction S.A. (Recife, Brasil), Intuit (Mountain View, CA, USA), Designers + Geeks (New York, USA); in 2012 at UX Russia (Moscow, Russia), UX Hong Kong (Hong Kong, China), WebVisions NYC (New York, NY, USA); in 2011 at the IA Summit (Denver, CO, USA), UX-LX (Lisbon, Portugal), Love at First Website (Portland, OR, USA).
This is something of a successor to my talk "Marrying Web Analytics and User Experience" (http://is.gd/vK34zS)
Advanced Project Data Analytics for Improved Project DeliveryMark Constable
Data Analytics is already beginning to impact how projects are delivered. We can now automate minute taking and capturing actions, we can use Flow to progress chase, Power BI reduces the burden of reporting.
But we are just scratching the surface. It won’t be long before we can leverage the rich dataset of experience to predict what risks are likely to occur, understand which WBS elements will be susceptible to variance, deduce what the optimum resource profile looks like, define a schedule by leveraging data from those projects that have gone before.
The role of a project professional is about to change dramatically. In this webinar we will explore the challenges and opportunities, and how we should respond. It’s a call-to-action for the community to mobilise, help to reshape project delivery and understand the implications for you and your organisation.
Presenter Martin Paver is a Chartered Project Professional, APM Fellow and Chartered Engineer. In December 2017 he established the London Project Data Analytics meetup which has quickly spread across the UK and expanded to 3000+ members. Martin has major project experience including leading a $billion projects with a team of 220 and a multi-billion PMO with a team of 50. He has a detailed grasp of project management and combines this with a broad understanding of recent developments in the field of data science. He is on a mission to ensure that the project management profession readies itself for a transformed future.
Learning outcomes:
- Understand the implications of advanced data analytics on project delivery
- Understand the scope of which functions it is likely to impact
- Help you to develop a strategy for how you engage with it
- Understand how to leverage the benefits and opportunities that will emerge from it
Presenter:
Martin Paver, CEO & Founder, Projecting Success Ltd
Best Practices in Recommender System ChallengesAlan Said
Recommender System Challenges such as the Netflix Prize, KDD Cup, etc. have contributed vastly to the development and adoptability of recommender systems. Each year a number of challenges or contests are organized covering different aspects of recommendation. In this tutorial and panel, we present some of the factors involved in successfully organizing a challenge, whether for reasons purely related to research, industrial challenges, or to widen the scope of recommender systems applications.
Building Intelligent Workplace Limits and Challenges RIGA COMM 2023 Muntis Rudzitis
Presented in RIGA COMM 2023
In this presentation I briefly cover how I see and try to systematize work, what parts of it can be helped by using modern ML/AI and algorithmic solutions.
Then I show some research work and publications we have done regarding Intelligent workplace concept - a knowledge work environment that tries to help to solve repeatable tasks.
Along the way I show whats possible and whats limited by current ML/AI capabilities, our lessons learned and some tips.
Trustworthy Micro-task Crowdsourcing: Challenges and OpportunitiesAlessandro Bozzon
Micro-task crowdsourcing has become a successful mean to obtain high-quality data from a large crowd of diverse people. In this context, trust between all the involved actors (i.e. requesters, workers, and platform owners) is a critical factor for acceptance and long-term success.
In this talk, I will discuss some problematic aspects of existing micro-task crowdsourcing platforms, where trust is built on fragmented, opaque, and often incomplete knowledge. I will provide examples in which the adoption of open, transparent, and socially-aware trust-building strategies have led to better crowdsourcing performance. I will then conclude with several proposals on how to increase the amount of trust cues available in crowdsourcing platforms, possibly with methods drawn from related disciplines such as user modelling and HCI.
The research team from TU Delft and DAT.Mobility worked on intelligently fusing data about pedestrian flows from different types of sensors (wifi, gps, counting cameras) to estimate crowd densities, travel speeds and flows at different routes of SAIL. Besides that, open social media platforms were crawled and analysed to get insight in demographics of the crowd and crowd sentiments at different hotspots of SAIL during the event. The collected data and state estimates can be used for more advanced and efficient crowd management support in the future. At SAIL 2015 we really focused on testing new sensor technologies, crowd data algorithms and analytics and assess whether they can be made useful and are reliable. This should give us insight in how we can improve crowd management of large events in cities in the future and provide visitors and citizens a more pleasurable experience.
Slides from my presentation at the Amsterdam Data Science seminar on City Analytics.
http://amsterdamdatascience.nl/event/amsterdam-data-science-seminar-city-analytics/
More information about the social glass project: www.social-glass.org
An Introduction to Human Computation and Games With A Purpose - Part IAlessandro Bozzon
Crowdsourcing and human computation are novel disciplines that enable the design of computation processes that include humans as actors for task execution. In such a context, Games With a Purpose are an effective mean to channel, in a constructive manner, the human brainpower required to perform tasks that computers are unable to per- form, through computer games. This tutorial introduces the core research questions in human computation, with a specific focus on the techniques required to manage structured and unstructured data. The second half of the tutorial delves into the field of game design for serious task, with an emphasis on games for human computation purposes. Our goal is to provide participants with a wide, yet complete overview of the research landscape; we aim at giving practitioners a solid understanding of the best practices in designing and running human computation tasks, while providing academics with solid references and, possibly, promising ideas for their future research activities.
A Service-Based Architecture for Multi-domain Search on the WebAlessandro Bozzon
Those slides were presented at the 8th International Conference on Service Computing (ICSOC 2010, San Francisco), and they relates to the research paper "A Service-Based Architecture for Multi-domain Search on the Web" authored by Alessandro Bozzon, Marco Brambilla, Francesco Corcoglioniti, and Salvatore Vadacca
This tutorial, offered at the 10th International Conference on Web Engineering, presents the peculiarities of advanced Web search applications, describes some tools and techniques that can be exploited, and offers a methodological approach to development. The approach proposed in this tutorial is based on the paradigm of Model Driven Development (MDD), where models are the core artifacts of the application life-cycle and model transformations progressively refine models to achieve an executable version of the system. To cope with the process-intensive nature of the main interactions (i.e., content analysis, query management, etc.), we describe the use of Process Models (e.g., BPMN models). Indeed, search-based applications are considered as process- and content-intensive applications, due to the trends towards exploratory search and search as a process visions.
Transcript: Selling digital books in 2024: Insights from industry leaders - T...BookNet Canada
The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more.
Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/
Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...UiPathCommunity
💥 Speed, accuracy, and scaling – discover the superpowers of GenAI in action with UiPath Document Understanding and Communications Mining™:
See how to accelerate model training and optimize model performance with active learning
Learn about the latest enhancements to out-of-the-box document processing – with little to no training required
Get an exclusive demo of the new family of UiPath LLMs – GenAI models specialized for processing different types of documents and messages
This is a hands-on session specifically designed for automation developers and AI enthusiasts seeking to enhance their knowledge in leveraging the latest intelligent document processing capabilities offered by UiPath.
Speakers:
👨🏫 Andras Palfi, Senior Product Manager, UiPath
👩🏫 Lenka Dulovicova, Product Program Manager, UiPath
Elevating Tactical DDD Patterns Through Object CalisthenicsDorra BARTAGUIZ
After immersing yourself in the blue book and its red counterpart, attending DDD-focused conferences, and applying tactical patterns, you're left with a crucial question: How do I ensure my design is effective? Tactical patterns within Domain-Driven Design (DDD) serve as guiding principles for creating clear and manageable domain models. However, achieving success with these patterns requires additional guidance. Interestingly, we've observed that a set of constraints initially designed for training purposes remarkably aligns with effective pattern implementation, offering a more ‘mechanical’ approach. Let's explore together how Object Calisthenics can elevate the design of your tactical DDD patterns, offering concrete help for those venturing into DDD for the first time!
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024Tobias Schneck
As AI technology is pushing into IT I was wondering myself, as an “infrastructure container kubernetes guy”, how get this fancy AI technology get managed from an infrastructure operational view? Is it possible to apply our lovely cloud native principals as well? What benefit’s both technologies could bring to each other?
Let me take this questions and provide you a short journey through existing deployment models and use cases for AI software. On practical examples, we discuss what cloud/on-premise strategy we may need for applying it to our own infrastructure to get it to work from an enterprise perspective. I want to give an overview about infrastructure requirements and technologies, what could be beneficial or limiting your AI use cases in an enterprise environment. An interactive Demo will give you some insides, what approaches I got already working for real.
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024Albert Hoitingh
In this session I delve into the encryption technology used in Microsoft 365 and Microsoft Purview. Including the concepts of Customer Key and Double Key Encryption.
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
Neuro-symbolic is not enough, we need neuro-*semantic*Frank van Harmelen
Neuro-symbolic (NeSy) AI is on the rise. However, simply machine learning on just any symbolic structure is not sufficient to really harvest the gains of NeSy. These will only be gained when the symbolic structures have an actual semantics. I give an operational definition of semantics as “predictable inference”.
All of this illustrated with link prediction over knowledge graphs, but the argument is general.
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered QualityInflectra
In this insightful webinar, Inflectra explores how artificial intelligence (AI) is transforming software development and testing. Discover how AI-powered tools are revolutionizing every stage of the software development lifecycle (SDLC), from design and prototyping to testing, deployment, and monitoring.
Learn about:
• The Future of Testing: How AI is shifting testing towards verification, analysis, and higher-level skills, while reducing repetitive tasks.
• Test Automation: How AI-powered test case generation, optimization, and self-healing tests are making testing more efficient and effective.
• Visual Testing: Explore the emerging capabilities of AI in visual testing and how it's set to revolutionize UI verification.
• Inflectra's AI Solutions: See demonstrations of Inflectra's cutting-edge AI tools like the ChatGPT plugin and Azure Open AI platform, designed to streamline your testing process.
Whether you're a developer, tester, or QA professional, this webinar will give you valuable insights into how AI is shaping the future of software delivery.
Essentials of Automations: Optimizing FME Workflows with ParametersSafe Software
Are you looking to streamline your workflows and boost your projects’ efficiency? Do you find yourself searching for ways to add flexibility and control over your FME workflows? If so, you’re in the right place.
Join us for an insightful dive into the world of FME parameters, a critical element in optimizing workflow efficiency. This webinar marks the beginning of our three-part “Essentials of Automation” series. This first webinar is designed to equip you with the knowledge and skills to utilize parameters effectively: enhancing the flexibility, maintainability, and user control of your FME projects.
Here’s what you’ll gain:
- Essentials of FME Parameters: Understand the pivotal role of parameters, including Reader/Writer, Transformer, User, and FME Flow categories. Discover how they are the key to unlocking automation and optimization within your workflows.
- Practical Applications in FME Form: Delve into key user parameter types including choice, connections, and file URLs. Allow users to control how a workflow runs, making your workflows more reusable. Learn to import values and deliver the best user experience for your workflows while enhancing accuracy.
- Optimization Strategies in FME Flow: Explore the creation and strategic deployment of parameters in FME Flow, including the use of deployment and geometry parameters, to maximize workflow efficiency.
- Pro Tips for Success: Gain insights on parameterizing connections and leveraging new features like Conditional Visibility for clarity and simplicity.
We’ll wrap up with a glimpse into future webinars, followed by a Q&A session to address your specific questions surrounding this topic.
Don’t miss this opportunity to elevate your FME expertise and drive your projects to new heights of efficiency.
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf91mobiles
91mobiles recently conducted a Smart TV Buyer Insights Survey in which we asked over 3,000 respondents about the TV they own, aspects they look at on a new TV, and their TV buying preferences.
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
2. Crowd Control is tough…
• There are several aspects that makes crowd
engineering complicated
• Task design, planning, assignment
• Workers discovery, assessment, engagement
Wednesday, May 15 Reactive Crowdsourcing 2
http://xkcd.com/1060/
3. Crowd Control is tough…
Wednesday, May 15 Reactive Crowdsourcing 3
• There are several aspects that makes crowd
engineering complicated
• Task design, planning, assignment
• Workers discovery, assessment, engagement
• Goal: taming the crowd
• Cost
• Time
• Quality
4. Crowd Control is tough…
Wednesday, May 15 Reactive Crowdsourcing 4
• There are several aspects that makes crowd
engineering complicated
• Task design, planning, assignment
• Workers discovery, assessment, engagement
• Goal: taming the crowd
• Cost
• Time
• Quality
• Motivation!
• Need for higher level abstractions and tools
• CONTROL as first-class citizen
5. Reactive Crowdsourcing
• A conceptual framework for modeling crowdsourcing
computations and control requirements
• Task Design
• Reactive Control Design
• Active Rule programming framework
• Declarative rule language
• A reactive execution environment for requirement
enforcement and reactive execution
• Based on the CrowdSearcher approach
Wednesday, May 15 Reactive Crowdsourcing 5
6. Why Active Rules?
• Crowdsourcing control typically focuses on task data
• Execution results, agreement on truth value, workers performance
• An active rule approach can provide
• Ease of Use: control is easily expressible
• Simple control data structures
• Familiar formalism
• Power: support to arbitrarily complex controls
• Extensibility mechanisms
• Automation: most active rules can be system-generated
• Well-defined semantic
• Flexibility: simple control variants have localized impact on the
rules set
• Control isolation
Wednesday, May 15 Reactive Crowdsourcing 6
7. The CrowdSearcher Approach
• Human-Enhanced data management with social networks
and Q&A systems as crowdsourcing platforms
• Example: search task (WWW2012)
Wednesday, May 15 Reactive Crowdsourcing 7
Human Interaction Management
Social
Networks
Human
Computation
Platforms
Q&A
Search Execution
Engine
raction
ent
Query Interface
Social
Networks
ery Answer
Search Execution
Engine
anInteraction
anagement
Human
Query Interface
Local
Social
Networks
Q&A
Query Answer
Search Execution
Engine
HumanInteraction
Management
SE Access
Interface
Human
Access
Interface
Query Interface
Local
Source
Access
Interface
Social
Networks
Q&A
Crowd-
source
platforms
Query Answer
Search Execution
Engine
HumanInteraction
Management
SE Access
Interface
Human
Access
Interface
Query Interface
Local
Source
Access
Interface
Social
Networks
Q&A
Crowd-
source
platforms
Query Answer
Data Management System
Human Access
Interface
Remote Data
Access
Local Data
Access
Search Execution
Engine
HumanInteraction
Management
SE Access
Interface
Human
Access
Interface
Query Interface
Local
Source
Access
Interface
Social
Networks
Q&A
Crowd-
source
platforms
Query Answer
Search Execution
Engine
HumanInteraction
Management
SE Access
Interface
Human
Access
Interface
Query Interface
Local
Source
Access
Interface
Socia
Networ
Q&A
Crowd
source
platform
Query Answer
Task
Human-Enhanced
Data
Query
Results
8. • A simple abstract model
• A task receives a list of input objects
• Performers execute one or more operations upon them
• The task produces a list of crowd-manipulated objects
• A simple task design and deployment process, based on specific data
structures
• created using model-driven transformations
• driven by the task specification
The Design Process
Wednesday, May 15 Reactive Crowdsourcing 8
I O
9. • A simple abstract model
• A task receives a list of input objects
• Performers execute one or more operations upon them
• The task produces a list of crowd-manipulated objects
• A simple task design and deployment process, based on specific data
structures
• created using model-driven transformations
• driven by the task specification
The Design Process
Task
Specification
Task Planning
Control
Specification
Wednesday, May 15 Reactive Crowdsourcing 9
• Task Spec: task operations, objects, and performers Dimension Tables
10. • A simple abstract model
• A task receives a list of input objects
• Performers execute one or more operations upon them
• The task produces a list of crowd-manipulated objects
• A simple task design and deployment process, based on specific data
structures
• created using model-driven transformations
• driven by the task specification
The Design Process
Task
Specification
Task Planning
Control
Specification
Wednesday, May 15 Reactive Crowdsourcing 10
• Task Spec: task operations, objects, and performers Dimension Tables
• Task Planning: work distribution Execution Table for task monitoring
11. • A simple abstract model
• A task receives a list of input objects
• Performers execute one or more operations upon them
• The task produces a list of crowd-manipulated objects
• A simple task design and deployment process, based on specific data
structures
• created using model-driven transformations
• driven by the task specification
The Design Process
Task
Specification
Task Planning
Control
Specification
Wednesday, May 15 Reactive Crowdsourcing 11
• Task Spec: task operations, objects, and performers Dimension Tables
• Task Planning: work distribution Execution Table for task monitoring
• Control Specification: task control policies Control Mart
13. Task Specification_2/3
Wednesday, May 15 Reactive Crowdsourcing 13
Politician
classifiedParty
lastName
photo
oID
• Input Objects Schema: typed attributes
• Output Attributes (according to task type)
Task
tID
opType
categories
Task Specification Task Planning Control Specification
Task Configuration
Object
Specification
o1
Obama
http://….
?????
14. Task Specification_3/3
Wednesday, May 15 Reactive Crowdsourcing 14
Politician
classifiedParty
lastName
photo
oID
Task Configuration
Object
Specification
Performer
Specification
• Execution platform(s)
• Qualifications, etc.
Task
tID
opType
categories Performer
name
pID
platform
Task Specification Task Planning Control Specification
p1
Alessandro
Facebook
15. Task Planning_1/2
• Organize the task in MicroTasks, and allocate input objects
• μTaskObjectExecution Designed for execution monitoring
• Track performers response
classifiedPartyplatform
μTaskObject
Execution
μtID
startTs
endTs
oID
pID
Wednesday, May 15 Reactive Crowdsourcing 15
Politician
classifiedParty
lastName
photo
oID
Task
tID
opType
categories Performer
name
pID
platform
Splitting
Task Specification Task Planning Control Specification
mt1
O1
…
…
…
…
Facebook
16. Task Planning_2/2
• Assign performers to MicroTasks on platforms
• Pull: dynamic assignment (First come - First served / Choice of the
performer)
• Push: static assignment (Performers’ priority / Performer matching)
classifiedPartyplatform
μTaskObject
Execution
μtID
startTs
endTs
oID
pID
Wednesday, May 15 Reactive Crowdsourcing 16
Politician
classifiedParty
lastName
photo
oID
Task
tID
opType
categories Performer
name
pID
platform
Splitting Assignment
Task Specification Task Planning Control Specification
mt1
O1
P1
Republican
00:00:01
00:00:10
Facebook
17. Control Specification_1/4
Wednesday, May 15 Reactive Crowdsourcing 17
Task Specification Task Planning Control Specification
• Status Variable: tracking task and performers status
classifiedPartyplatform
μTaskObject
Execution
μtID
startTs
endTs
oID
pID
Politician
classifiedParty
lastName
photo
oID
Performer
name
pID
status
platformTask
tID
opType
categories
status Trusted/SpammerCreated/Planned/Closed
18. Control Specification_2/4
Wednesday, May 15 Reactive Crowdsourcing 18
Task Specification Task Planning Control Specification
• Object : tracking objects status
classifiedPartyplatform
μTaskObject
Execution
μtID
startTs
endTs
oID
pID
Politician
classifiedParty
lastName
photo
oID
Performer
name
pID
status
platformTask
tID
opType
categories
status
Object
Control #dem
oID
#eval
#rep
#curAnswer
19. Control Specification_3/4
Wednesday, May 15 Reactive Crowdsourcing 19
Task Specification Task Planning Control Specification
• Object : tracking object responses
• Performer: tracking performer behavior (e.g. spammers)
Performer
Control #right
pID
#eval
#wrong
classifiedPartyplatform
μTaskObject
Execution
μtID
startTs
endTs
oID
pID
Politician
classifiedParty
lastName
photo
oID
Performer
name
pID
status
platformTask
tID
opType
categories
status
Object
Control #dem
oID
#eval
#rep
#curAnswer
20. Control Specification_4/4
• Object : tracking object responses
• Performer: tracking performer behavior (e.g. spammers)
• Task: tracking task status: closing @completion, re-plan
Wednesday, May 15 Reactive Crowdsourcing 20
Task
Control#compObj
tID
#compExec
Performer
Control #right
pID
#eval
#wrong
classifiedPartyplatform
μTaskObject
Execution
μtID
startTs
endTs
oID
pID
Politician
classifiedParty
lastName
photo
oID
Performer
name
pID
status
platformTask
tID
opType
categories
status
Object
Control #dem
oID
#eval
#rep
#curAnswer
Task Specification Task Planning Control Specification
21. Active Rules Language
• Active rules are expressed on the previous data
structures
• Event-Condition-Action paradigm
Wednesday, May 15 Reactive Crowdsourcing 21
22. Active Rules Language
• Active rules are expressed on the previous data
structures
• Event-Condition-Action paradigm
• Events: data updates / timer
• ROW-level granularity
• OLD before state of a row
• NEW after state of a row
Wednesday, May 15 Reactive Crowdsourcing 22
e: UPDATE FOR μTaskObjectExecution[ClassifiedParty]
23. Active Rules Language
• Active rules are expressed on the previous data
structures
• Event-Condition-Action paradigm
• Events: data updates / timer
• ROW-level granularity
• OLD before state of a row
• NEW after state of a row
• Condition: a predicate that must be satisfied (e.g. conditions on
control mart attributes)
Wednesday, May 15 Reactive Crowdsourcing 23
e: UPDATE FOR μTaskObjectExecution[ClassifiedParty]
c: NEW.ClassifiedParty == ’Republican’
24. Active Rules Language
• Active rules are expressed on the previous data
structures
• Event-Condition-Action paradigm
• Events: data updates / timer
• ROW-level granularity
• OLD before state of a row
• NEW after state of a row
• Condition: a predicate that must be satisfied (e.g. conditions on
control mart attributes)
• Actions: updates on data structures (e.g. change attribute
value, create new instances), special functions (e.g. replan)
Wednesday, May 15 Reactive Crowdsourcing 24
e: UPDATE FOR μTaskObjectExecution[ClassifiedParty]
c: NEW.ClassifiedParty == ’Republican’
a: SET ObjectControl[oID == NEW.oID].#Eval+= 1
25. Wednesday, May 15 Reactive Crowdsourcing 25
e: UPDATE FOR ObjectControl
c: (NEW.Rep== 2) or (NEW.Dem == 2)
a: SET Politician[oid==NEW.oid].classifiedParty = NEW.CurAnswer,
SET TaskControl[tID==NEW.tID].compObj += 1
Rule Example
Task
Control#compObj
tID Performer
Control
μTaskObject
Execution
Politician classifiedParty
oID
PerformerTask
Object
Control #dem
oID
#rep
#eval
tIDEvent
26. Wednesday, May 15 Reactive Crowdsourcing 26
e: UPDATE FOR ObjectControl
c: (NEW.Rep== 2) or (NEW.Dem == 2)
a: SET Politician[oid==NEW.oid].classifiedParty = NEW.CurAnswer,
SET TaskControl[tID==NEW.tID].compObj += 1
Rule Example
Task
Control#compObj
tID Performer
Control
μTaskObject
Execution
Politician classifiedParty
oID
PerformerTask
Object
Control #dem
oID
#rep
#eval
tID
Condition
27. Wednesday, May 15 Reactive Crowdsourcing 27
e: UPDATE FOR ObjectControl
c: (NEW.Rep== 2) or (NEW.Dem == 2)
a: SET Politician[oid==NEW.oid].classifiedParty = NEW.CurAnswer,
SET TaskControl[tID==NEW.tID].compObj += 1
Rule Example
Task
Control#compObj
tID Performer
Control
μTaskObject
Execution
Politician classifiedParty
oID
PerformerTask
Object
Control #dem
oID
#rep
#eval
tID
Action
28. Wednesday, May 15 Reactive Crowdsourcing 28
e: UPDATE FOR ObjectControl
c: (NEW.Rep== 2) or (NEW.Dem == 2)
a: SET Politician[oid==NEW.oid].classifiedParty = NEW.CurAnswer,
SET TaskControl[tID==NEW.tID].compObj += 1
Rule Example
Task
Control#compObj
tID Performer
Control
μTaskObject
Execution
Politician classifiedParty
oID
PerformerTask
Object
Control #dem
oID
#rep
#eval
tID
Action
29. Rule Programming Best Practice
• We define three classes of rules
Wednesday, May 15 Reactive Crowdsourcing 29
μTaskObject
Execution
Performer
Control
Object
Control
Task
Control
Politician Performer Task
30. Rule Programming Best Practice
Wednesday, May 15 Reactive Crowdsourcing 30
• We define three classes of rules
• Control rules: modifying the control tables;
μTaskObject
Execution
Performer
Control
Object
Control
Task
Control
Politician Performer Task
31. Rule Programming Best Practice
Wednesday, May 15 Reactive Crowdsourcing 31
• We define three classes of rules
• Control rules: modifying the control tables;
• Result rules: modifying the dimension tables (object, performer, task);
μTaskObject
Execution
Performer
Control
Object
Control
Task
Control
Politician Performer Task
32. Rule Programming Best Practice
Wednesday, May 15 Reactive Crowdsourcing 32
• Top-to-bottom, left-to-right, evaluation
• Guaranteed termination
• We define three classes of rules
• Control rules: modifying the control tables;
• Result rules: modifying the dimension tables (object, performer, task);
μTaskObject
Execution
Performer
Control
Object
Control
Task
Control
Politician Performer Task
33. Rule Programming Best Practice
• We define three classes of rules
• Control rules: modifying the control tables;
• Result rules: modifying the dimension tables (object, performer, task);
• Execution rules: modifying the execution table, either directly or through re-planning
Wednesday, May 15 Reactive Crowdsourcing 33
μTaskObject
Execution
Performer
Control
Object
Control
Task
Control
Politician Performer Task
• Termination must be proven (Rule precedence graph has cycles)
34. Experimental Evaluation
• GOAL: demonstrate the flexibility and expressive power
of reactive crowdsourcing
• 3 experiments, focused on Italian politicians
• Parties: Human Computation affiliation classification
• Law: Game With a Purpose guess the convicted politician
• Order: Pure Game hot or not
• 1 week (November 2012)
• 284 distinct performers
• Recruited through public mailing lists and social networks
announcements
• 3500 Micro Tasks
Wednesday, May 15 Reactive Crowdsourcing 34
35. Politician Affiliation
• Given the picture and name of a politician, specify his/her political
affiliation
• No time limit
• Performers are encouraged to look up online
• 2 set of rules
• Majority Evaluation
• Spammer Detection
Wednesday, May 15 Reactive Crowdsourcing 35
36. Results – Majority Evaluation_1/3
Wednesday, May 15 Reactive Crowdsourcing 36
30 object; object redundancy = 9;
Final object classification as simple majority after 7 evaluations
37. Results - Majority Evaluation_2/3
Wednesday, May 15 Reactive Crowdsourcing 37
Majority @7
Early Majority @3 R @7
-27% executions
-18% precision
%ofCompl.Objects
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
Precision
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
#Executions
0 10 20 30 40 50 60 70 80 90
Final object classification as total majority after 3 evaluations
Otherwise, re-plan of 4 additional evaluations. Then simple majority at 7
38. Results - Majority Evaluation_3/3
Wednesday, May 15 Reactive Crowdsourcing 38
Majority @7
Early Majority @3 R @7
Majority @3 R @5 R @7
-23% executions
+26% precision
+50% precision
%ofCompl.Objects
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
Precision
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
#Executions
0 10 20 30 40 50 60 70 80 90
Final object classification as total majority after 3 evaluations
Otherwise, simple majority at 5 or at 7 (with replan)
39. Results – Spammer Detection_1/2
Wednesday, May 15 Reactive Crowdsourcing 39
New rule for spammer detection without ground truth
Performer correctness on final majority. Spammer if > 50% wrong classifications
Majority @3 R @5 R @7
Majority @3 R @5 R @7 - Spammer Detection
+46% executions
+1.5% precision
%ofCompl.Objects
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
Precision
0
0.1
0.2
0.3
0.4
0.5
0.6
0.7
0.8
#Executions
0 10 20 30 40 50 60 70 80 90
41. Summary
• Results
• An integrated framework for crowdsourcing task design and control
• Well-structured control rules with some guarantees of termination
• Support for cross-platform crowd interoperability
• A working prototype crowdsearcher.search-computing.org
• Forthcoming
• Exploitation of interoperability
• Expertise finding
• Dynamic planning
• Integration with other social-networks and human computation
platforms
Wednesday, May 15 Reactive Crowdsourcing 41
Good afternoonI’m very happy to be here today and discuss with you our work, named reactive crowdsourcing
But first allow me to contextualize a bit its scope. I’m sure we are all very familiar with crowdsourcing and human computation, which is a based on a very neat idea: organize humans to accomplish tasks. Unfortunately, or luckily for us , such organization is a complex matter. When designing systems and experiments, there are several things to consider: how do you split your tasks in microtask, which are the best performers for it, how do you scout them, etc.
It turns out, our ultimate goal is controlling the crowd. We would like to drive people behavior in order satisfy some constraints, which are typically monetary, qualitative, or temporal. Indeed, a lot of work has been devoted to devise algorithms able to minimize one of those costs, and a lot of effort has been recently put in the creation of frameworks and algorithm for crowd management.However, we observed that existing solutions (e.g. crowddb, deco, and others) provide limited and veryspecific control features. We advocate crowd control to be at the very center of task design. And we also advocate the need for methods for the systematicdesign of complex control strategies based on the state of tasks, results and performers.
It turns out, our ultimate goal is controlling the crowd. We would like to drive people behavior in order satisfy some constraints, which are typically monetary, qualitative, or temporal. Indeed, a lot of work has been devoted to devise algorithms able to minimize one of those costs, and a lot of effort has been recently put in the creation of frameworks and algorithm for crowd management.However, we observed that existing solutions (e.g. crowddb, deco, and others) provide limited and veryspecific control features. We advocate crowd control to be at the very center of task design. And we also advocate the need for methods for the systematicdesign of complex control strategies based on the state of tasks, results and performers.
This paper wepropose a conceptual framework and a reactive execution environment for modelling and controlling crowdsourcing computationsWit the ultimate goal of minimizing the effort required for control specification, we propose:a simple task design processA rule specification language,whose properties (e.g., termination) can be easily proved in the context of a well-organized computational framework
But why we decided to go for an approach based on active rules?The choice stemmed from the observation that crowdsourcing control is typically driven by data, like the status of the HIT executions, the worker performance, the current agreement on the truth value of some object. Therefore, it came almost natural for us to turn on a data-driven approach, that proven very effective for the definition of control in several contexts, including database systemsActive rules are actually relatively easy to use, when expressed on well-define data structures. They allow the definition of arbitrary complex control logic, most of which can be easily automated thanks to a well-defined syntax and semantic. Also, they allow for a great flexibilty, since changes in the control logic of the application can be well-isolated into localized changes of the rule set
This work capitalizes on the results of our www2012 paper, where we an approach for data management that integrates SN, QA and traditional HC platform for the execution of human computation and crowdsourcing tasks.This work specifically addressed crowd-enhanced search, a now very popular trend. However, the approach we proposed can be very easily generalized to any kind of human-enhanced data management system, and we believe this crowd-interoperability can be exploited in very interesting ways. LOCAL SOURCE: sorgentidatilocalisfruttate dal Search Execution Engine, magariaccedutedallo Human Interaction Management per configurare / gestirei task. La suaesistenza e’ accessoriarispettoaglialtri, e codificainformazioni applicative specificheICONE DI DX, DALL’ALTO a SX(social networks) Facebook, Twitter, Google +(Q&A systems) StackOverlflow, Yahoo Answers, Quora(HC Platforms) Freebase, Amazon Mechanical Turk, ODesk
The design process we propose is based on a very simple data-driven abstract model for task execution: data in, workers do stuff, data outCoherently with this vision, the task design and deployment process is based upon specific data structures, created in an automated way trough model driven transformations. Unfortunately we don’t have time to go too deep in this matter, so I refer you to the paper for further details.
The process composes of three simple steps. The first, task specification, is devoted to the definition of basic task information, such as the data-driven operations to perform (e.g. classify a picture, add new instances of a given object, etc.), the shape of the actual objects to evaluate, and the charactersticis of the targeted set of performers. All those aspect are encoded in what we call Dimension Tables
Then, the second step is devoted to task planning, i.e. specify how works should be partitioned and distributed among performers. Those planning aspects are encoded into an Execution Table, specifically devoted to task monitoring
Finally, the developer must specify the control logic for the task, and it does so by defining active rules upon control-specific data structures contained in what we call the Control Mart.
To clarify the process, let us show visually how the supporting data structure are created in a simple case study: a picture classification task, where the picture contain picture of politicians, and the task is to specify to which party they belong. We will use color codes to identify data structures types, and we highlight in bold those attributes which depend on the specifically designed tasks. All the other are, somehow, standard. The first dimension table is the Task table, which contains info about the performed operations (a classification, in this case), and the categories to which politicians can be assigned (e.g. republican and democtrats)
Then, we have a dimension table for the actual objects of the task, in this case, politicians. Notice that the classifiedPartyattribute is actually produced by the crowd at the end of the task
Finally, a Performer dimension table represents the population of performer currently available for task execution. The table can be pre-filled, if the performers are known in advance, or empty, in case of openly available tasks.
The second activity of the process deals with the planning of the task and it also composes of two phases. In the first phase, suited algorithms will decide such things as how which objects should appear in each MicroTask? (e.g. ground truth assessment), how many objects in each MicroTask, how often an object should appear in MicroTasks, etc. TheμTaskObjectExecution table keeps track of this organization, and it contains one tuple for each object/microTask association. A performer, when executing a microTask, will fill in the data value required by the specified operation (e.g. the party of a given politician)
The second phase of task planning deals with the assignment of microTasks to performers. This can be done according to several policies (e.g. pull or push). In the example, the assignment is performed in a pull fashion, and attributes are given value on performer arrival
The third part of the process is devoted to the specification of control. Please notice here that I added to an addition color code to denote attributes in the Dimension Tables which serve task control purposes. So for instance, a performer can be classified as trusted or spammer. A task as “created/planned/closed” and so on
Bin addition to status variable, control is defined upon the Control Mart, which composes of three data structures. The first one, called ObjectControl contains control variable related to the object of the task, like the number of evaluation it received, the number of classifications it received for each class, and the current truth value.
The second one, called PerformerControl contains control variable related to the performer of the task, like the number of evaluation she performerd, and the number of times she provided a right or wrong answer (assuming the existance of a ground truth)
Finally, the thirdone, called TaskControl contains control variable related to the task, like the number total number of objects currently evaluated, or the number of executed hits.
The previous data structure provide in a very simple yet complete way the control variables that are needed to define the task control policies. But how can control be specified? We rely on a language based on the classic ECA paradigm
Where events are updated on the data structure values. We decided for a row-level update granularity, so to easily track the before and after states of rows. In the example, the rule triggers when the ClassifiedParty attribute of a tuple in the μTaskObjectExecution table changes
Conditions are expressed as conditions on data attributes (e.g., the value specified by the performer)
And actions are updated performed on the same, or other data structures. Such updates can be done directly or trough special functions, devoted to such operations as replanning’Of course there is no time to show the syntax of the language, but you can find more on the paper.
The second rule is a bit more complex, and it is used to assess the truth value of an object trough majority voting. For instance, here we assume that as soon as a Politician gets 2 evaluation as Rep or Dem, the object can be deemed as completed. triggers when the ObjectControl table updates
It is quite known that active rule programming can be rather subtle and unstable, as the behavior of a set of rules may change dramatically as a consequence of small changes in the rules To simply, and better control rule programming we devise three classes of rules which, as I will show soon, have interesting properties
The first class are named control rules, and are meant to modify control tables. Arrows represents rules triggering on a table (the source of the arrow) and affecting another table (the destination). As you can see, not all the possible source-target couple are adimissible, and I will explain soon why.
The second class of rules modify the dimension tables, and are the one devoted to changing the status of the main task entities (e.g. setting a perforomer as spammer when she makes too many bad classifications)
Note that, since we suggest a very well-defined top-to-bottom, left-to-right semantic, no cycles are allowed, and therefore rules are guardanteed to terminate. Those cycles in the object control – performer control etc. still bases on precise rules also on the attributes of the table
Finally, execution rules are responsivble for the modification of the execution table, and are therefore responsible for modifying the set?distribution/assignment of the currently defined microtasks. Those rules introduce cycles and, therefore, might cause unconvergence (and termination must be verified)
To demonstrate the flexibily and expressive power of reactive crowdsourcing we advised the experiments, conducted during one week of November 2012. We developed three very different scenario, all programmed with our approach. Unfortunately we don’t have enough time to describe them all, so we focus on just one. A classification task similar to the one used as example in the presentation