SlideShare a Scribd company logo
1 of 1
Download to read offline
Expert Crowdsourcing with Flash Teams
Daniela Retelny, Sébastien Robaszkiewicz, Alexandra To, Michael Bernstein
{dretelny, robi, ato1120, msb}@cs.stanford.edu

Crowdsourcing is Plateauing in Complexity Flash Team Examples
Crowd workers struggle to complete open ended and highly Our evaluation of flash teams to date includes the successful completion of eight different flash team examples,
utilizing different team structures and workflows. The flash teams varied in size, complexity and expertise and
interdependent tasks.
they produced outputs that ranged in completion time and quality. The images below depict the team structures,
Despite good intentions, teams of crowd experts are
workflows and output from three napkin sketch, animation and MOOC flash team examples we completed.
typically neither effective nor efficient at these tasks.
Expert crowdsourcing lacks generalizable techniques for
guiding crowds through complex tasks.

Flash Teams
We introduce flash teams, modular team structures that
computationally guide a team of crowd experts reproducibly
and at scale. These modular structures enable crowd expert
teams to grow, adapt, and recombine into larger organizations.
Defining attributes of flash teams:
MODULARITY: self-contained and replicable team
structures composed of crowd experts that are
computationally guided.
ELASTICITY: parameterizable team growth on demand,
allowing teams to grow and shrink as expertise are needed.
PIPELINING: parallelism enabled by defining
interdependencies and handoffs.
Methods:
To explore the feasibility and scope of flash teams, and
understand the effectiveness of modularity, elasticity and
pipelining, we gathered thirteen different flash teams to
complete complex tasks ranging from design to education.

Foundry, a Flash Team Authoring Platform

To encapsulate the foundations of the flash teams formalism and lessen
We recruit experts through oDesk, a crowdsourcing platform the threshold to creating flash teams, we created Foundry, a web platform
that allows requesters to create, reuse and recombine flash team
for expert work that allows requesters to post a request
under a specific expertise category (e.g., Python development, structures and workflows and allows team members to coordinate tasks
and track progress. The image to the right shows a screenshot of Foundry.
sound production, or visual design).

http://hci.stanford.edu

STANFORD HCI GROUP

More Related Content

Viewers also liked

5 Frases Para Começar o Dia INSPIRADO!
5 Frases Para Começar o Dia INSPIRADO! 5 Frases Para Começar o Dia INSPIRADO!
5 Frases Para Começar o Dia INSPIRADO! Bruno Perin
 
Infokids 3 Herramientas Informáticas - Fichas de Aprendizaje 2014
Infokids 3 Herramientas Informáticas - Fichas de Aprendizaje 2014Infokids 3 Herramientas Informáticas - Fichas de Aprendizaje 2014
Infokids 3 Herramientas Informáticas - Fichas de Aprendizaje 2014jorgequiat
 
The Audience Is Always Right
The Audience Is Always RightThe Audience Is Always Right
The Audience Is Always Rightmad blog
 
Almost Famous
Almost FamousAlmost Famous
Almost Famousabbieroad
 
SMART: Staff are Mobile And Ready to Teach
SMART: Staff are Mobile And Ready to TeachSMART: Staff are Mobile And Ready to Teach
SMART: Staff are Mobile And Ready to TeachAndrew Middleton
 
Guia de Recursos de la Producció Agrària Ecològica 2011
Guia de Recursos de la Producció Agrària Ecològica 2011Guia de Recursos de la Producció Agrària Ecològica 2011
Guia de Recursos de la Producció Agrària Ecològica 2011La Tavella
 
Conekta seguridad
Conekta seguridadConekta seguridad
Conekta seguridadconektame
 
Np tv the-social-medium-final-21-nov-2014
Np tv the-social-medium-final-21-nov-2014Np tv the-social-medium-final-21-nov-2014
Np tv the-social-medium-final-21-nov-2014OptimediaSpain
 
15 vhvn-247
15 vhvn-24715 vhvn-247
15 vhvn-247Mèo Mun
 

Viewers also liked (15)

5 Frases Para Começar o Dia INSPIRADO!
5 Frases Para Começar o Dia INSPIRADO! 5 Frases Para Começar o Dia INSPIRADO!
5 Frases Para Começar o Dia INSPIRADO!
 
Infokids 3 Herramientas Informáticas - Fichas de Aprendizaje 2014
Infokids 3 Herramientas Informáticas - Fichas de Aprendizaje 2014Infokids 3 Herramientas Informáticas - Fichas de Aprendizaje 2014
Infokids 3 Herramientas Informáticas - Fichas de Aprendizaje 2014
 
The Audience Is Always Right
The Audience Is Always RightThe Audience Is Always Right
The Audience Is Always Right
 
Almost Famous
Almost FamousAlmost Famous
Almost Famous
 
SMART: Staff are Mobile And Ready to Teach
SMART: Staff are Mobile And Ready to TeachSMART: Staff are Mobile And Ready to Teach
SMART: Staff are Mobile And Ready to Teach
 
RESUME
RESUMERESUME
RESUME
 
Isabel
IsabelIsabel
Isabel
 
Memoria Intégrat-e
Memoria Intégrat-eMemoria Intégrat-e
Memoria Intégrat-e
 
Guia de Recursos de la Producció Agrària Ecològica 2011
Guia de Recursos de la Producció Agrària Ecològica 2011Guia de Recursos de la Producció Agrària Ecològica 2011
Guia de Recursos de la Producció Agrària Ecològica 2011
 
Conekta seguridad
Conekta seguridadConekta seguridad
Conekta seguridad
 
Np tv the-social-medium-final-21-nov-2014
Np tv the-social-medium-final-21-nov-2014Np tv the-social-medium-final-21-nov-2014
Np tv the-social-medium-final-21-nov-2014
 
Formato lista de chequeo
Formato lista de chequeoFormato lista de chequeo
Formato lista de chequeo
 
Internet
InternetInternet
Internet
 
BOSS
BOSSBOSS
BOSS
 
15 vhvn-247
15 vhvn-24715 vhvn-247
15 vhvn-247
 

Similar to Expert Crowdsourcing with Flash Teams | CrowdConf 2013 poster

Transforming Museums: Toolkit for Change
Transforming Museums: Toolkit for ChangeTransforming Museums: Toolkit for Change
Transforming Museums: Toolkit for ChangeElliot Felix
 
case study analysis essay Nursing Writers Hub.docx
case study analysis essay Nursing Writers Hub.docxcase study analysis essay Nursing Writers Hub.docx
case study analysis essay Nursing Writers Hub.docxstirlingvwriters
 
FLEXspace - Flexible Learning Environment Exchange - Version 2.0 Design
FLEXspace - Flexible Learning Environment Exchange - Version 2.0 DesignFLEXspace - Flexible Learning Environment Exchange - Version 2.0 Design
FLEXspace - Flexible Learning Environment Exchange - Version 2.0 DesignBennett King
 
The Jnaapti Virtual Coach Platform
The Jnaapti Virtual Coach PlatformThe Jnaapti Virtual Coach Platform
The Jnaapti Virtual Coach PlatformJnaapti
 
What is an (LXP) Learning Experience Platform
What is an (LXP) Learning Experience PlatformWhat is an (LXP) Learning Experience Platform
What is an (LXP) Learning Experience PlatformAndrewSebastian17
 
10 differences between SAFe and LeSS
10 differences between SAFe and LeSS10 differences between SAFe and LeSS
10 differences between SAFe and LeSSStanislaw Matczak
 
Scaling an Engineering Team
Scaling an Engineering TeamScaling an Engineering Team
Scaling an Engineering TeamDashlane
 
Learning trends white paper final
Learning trends white paper finalLearning trends white paper final
Learning trends white paper finalastdny
 
Agile transfromation
Agile transfromationAgile transfromation
Agile transfromationMax Carlin
 
Salesforce Trailhead for Developers
Salesforce Trailhead for DevelopersSalesforce Trailhead for Developers
Salesforce Trailhead for DevelopersCloud Analogy
 
How a Social Knowledge Graph Improves Remote Working by Capturing Context fro...
How a Social Knowledge Graph Improves Remote Working by Capturing Context fro...How a Social Knowledge Graph Improves Remote Working by Capturing Context fro...
How a Social Knowledge Graph Improves Remote Working by Capturing Context fro...Sabine Seitz
 
Embedded interactive learning analytics dashboards with Elasticsearch and Kib...
Embedded interactive learning analytics dashboards with Elasticsearch and Kib...Embedded interactive learning analytics dashboards with Elasticsearch and Kib...
Embedded interactive learning analytics dashboards with Elasticsearch and Kib...Andrii Vozniuk
 
Sym18 296 Ae (Dbaccess Case Study)
Sym18 296 Ae (Dbaccess Case Study)Sym18 296 Ae (Dbaccess Case Study)
Sym18 296 Ae (Dbaccess Case Study)La Red DBAccess
 
Knowledge Sharing Canvas
Knowledge Sharing CanvasKnowledge Sharing Canvas
Knowledge Sharing CanvasRaphaël Briner
 
Assignment 2 Group-Level of Analysis – Portfolio ProjectDue Dat.docx
Assignment 2 Group-Level of Analysis – Portfolio ProjectDue Dat.docxAssignment 2 Group-Level of Analysis – Portfolio ProjectDue Dat.docx
Assignment 2 Group-Level of Analysis – Portfolio ProjectDue Dat.docxsteviesellars
 
Epic battle of component feudalism vs scaled agile
Epic battle of  component feudalism  vs  scaled agileEpic battle of  component feudalism  vs  scaled agile
Epic battle of component feudalism vs scaled agileAlexey Kovalyov
 

Similar to Expert Crowdsourcing with Flash Teams | CrowdConf 2013 poster (20)

Transforming Museums: Toolkit for Change
Transforming Museums: Toolkit for ChangeTransforming Museums: Toolkit for Change
Transforming Museums: Toolkit for Change
 
case study analysis essay Nursing Writers Hub.docx
case study analysis essay Nursing Writers Hub.docxcase study analysis essay Nursing Writers Hub.docx
case study analysis essay Nursing Writers Hub.docx
 
FLEXspace - Flexible Learning Environment Exchange - Version 2.0 Design
FLEXspace - Flexible Learning Environment Exchange - Version 2.0 DesignFLEXspace - Flexible Learning Environment Exchange - Version 2.0 Design
FLEXspace - Flexible Learning Environment Exchange - Version 2.0 Design
 
E20CSE109-PPT.pdf
E20CSE109-PPT.pdfE20CSE109-PPT.pdf
E20CSE109-PPT.pdf
 
The Jnaapti Virtual Coach Platform
The Jnaapti Virtual Coach PlatformThe Jnaapti Virtual Coach Platform
The Jnaapti Virtual Coach Platform
 
What is an (LXP) Learning Experience Platform
What is an (LXP) Learning Experience PlatformWhat is an (LXP) Learning Experience Platform
What is an (LXP) Learning Experience Platform
 
El futuro del trabajo
El futuro del trabajoEl futuro del trabajo
El futuro del trabajo
 
10 differences between SAFe and LeSS
10 differences between SAFe and LeSS10 differences between SAFe and LeSS
10 differences between SAFe and LeSS
 
Scaling an Engineering Team
Scaling an Engineering TeamScaling an Engineering Team
Scaling an Engineering Team
 
Learning trends white paper final
Learning trends white paper finalLearning trends white paper final
Learning trends white paper final
 
WCECS2009_pp764-767
WCECS2009_pp764-767WCECS2009_pp764-767
WCECS2009_pp764-767
 
Agile transfromation
Agile transfromationAgile transfromation
Agile transfromation
 
Salesforce Trailhead for Developers
Salesforce Trailhead for DevelopersSalesforce Trailhead for Developers
Salesforce Trailhead for Developers
 
How a Social Knowledge Graph Improves Remote Working by Capturing Context fro...
How a Social Knowledge Graph Improves Remote Working by Capturing Context fro...How a Social Knowledge Graph Improves Remote Working by Capturing Context fro...
How a Social Knowledge Graph Improves Remote Working by Capturing Context fro...
 
Embedded interactive learning analytics dashboards with Elasticsearch and Kib...
Embedded interactive learning analytics dashboards with Elasticsearch and Kib...Embedded interactive learning analytics dashboards with Elasticsearch and Kib...
Embedded interactive learning analytics dashboards with Elasticsearch and Kib...
 
Sym18 296 Ae (Dbaccess Case Study)
Sym18 296 Ae (Dbaccess Case Study)Sym18 296 Ae (Dbaccess Case Study)
Sym18 296 Ae (Dbaccess Case Study)
 
Optimizing group collaboration - MeetingSphere services brochure
Optimizing group collaboration - MeetingSphere services brochureOptimizing group collaboration - MeetingSphere services brochure
Optimizing group collaboration - MeetingSphere services brochure
 
Knowledge Sharing Canvas
Knowledge Sharing CanvasKnowledge Sharing Canvas
Knowledge Sharing Canvas
 
Assignment 2 Group-Level of Analysis – Portfolio ProjectDue Dat.docx
Assignment 2 Group-Level of Analysis – Portfolio ProjectDue Dat.docxAssignment 2 Group-Level of Analysis – Portfolio ProjectDue Dat.docx
Assignment 2 Group-Level of Analysis – Portfolio ProjectDue Dat.docx
 
Epic battle of component feudalism vs scaled agile
Epic battle of  component feudalism  vs  scaled agileEpic battle of  component feudalism  vs  scaled agile
Epic battle of component feudalism vs scaled agile
 

More from CrowdFlower

Building Better Models Faster Using Active Learning
Building Better Models Faster Using Active LearningBuilding Better Models Faster Using Active Learning
Building Better Models Faster Using Active LearningCrowdFlower
 
Active Learning and Human-in-the-Loop
Active Learning and Human-in-the-LoopActive Learning and Human-in-the-Loop
Active Learning and Human-in-the-LoopCrowdFlower
 
CrowdFlower NDA Crowds - Secure, exceptional tasking at a massive scale.
CrowdFlower NDA Crowds - Secure, exceptional tasking at a massive scale. CrowdFlower NDA Crowds - Secure, exceptional tasking at a massive scale.
CrowdFlower NDA Crowds - Secure, exceptional tasking at a massive scale. CrowdFlower
 
CrowdFlower Product Webinar - Graphical Editor and Visual Reports
CrowdFlower Product Webinar - Graphical Editor and Visual ReportsCrowdFlower Product Webinar - Graphical Editor and Visual Reports
CrowdFlower Product Webinar - Graphical Editor and Visual ReportsCrowdFlower
 
How Oracle Uses CrowdFlower For Sentiment Analysis
How Oracle Uses CrowdFlower For Sentiment AnalysisHow Oracle Uses CrowdFlower For Sentiment Analysis
How Oracle Uses CrowdFlower For Sentiment AnalysisCrowdFlower
 
Humanizing The Machine
Humanizing The MachineHumanizing The Machine
Humanizing The MachineCrowdFlower
 
Open Data Science Conference 2015
Open Data Science Conference 2015Open Data Science Conference 2015
Open Data Science Conference 2015CrowdFlower
 
Productive Out-of-the-Box | Tooling with Yeoman to Rapidly Develop Ember.js A...
Productive Out-of-the-Box | Tooling with Yeoman to Rapidly Develop Ember.js A...Productive Out-of-the-Box | Tooling with Yeoman to Rapidly Develop Ember.js A...
Productive Out-of-the-Box | Tooling with Yeoman to Rapidly Develop Ember.js A...CrowdFlower
 
Virtual Data Steward: Data Management 3.0
Virtual Data Steward: Data Management 3.0Virtual Data Steward: Data Management 3.0
Virtual Data Steward: Data Management 3.0CrowdFlower
 
The State of Enterprise Crowdsourcing 2013
The State of Enterprise Crowdsourcing 2013The State of Enterprise Crowdsourcing 2013
The State of Enterprise Crowdsourcing 2013CrowdFlower
 
CrowdFlower University Oct. 21 2013
CrowdFlower University Oct. 21 2013CrowdFlower University Oct. 21 2013
CrowdFlower University Oct. 21 2013CrowdFlower
 

More from CrowdFlower (12)

7 Myths of AI
7 Myths of AI7 Myths of AI
7 Myths of AI
 
Building Better Models Faster Using Active Learning
Building Better Models Faster Using Active LearningBuilding Better Models Faster Using Active Learning
Building Better Models Faster Using Active Learning
 
Active Learning and Human-in-the-Loop
Active Learning and Human-in-the-LoopActive Learning and Human-in-the-Loop
Active Learning and Human-in-the-Loop
 
CrowdFlower NDA Crowds - Secure, exceptional tasking at a massive scale.
CrowdFlower NDA Crowds - Secure, exceptional tasking at a massive scale. CrowdFlower NDA Crowds - Secure, exceptional tasking at a massive scale.
CrowdFlower NDA Crowds - Secure, exceptional tasking at a massive scale.
 
CrowdFlower Product Webinar - Graphical Editor and Visual Reports
CrowdFlower Product Webinar - Graphical Editor and Visual ReportsCrowdFlower Product Webinar - Graphical Editor and Visual Reports
CrowdFlower Product Webinar - Graphical Editor and Visual Reports
 
How Oracle Uses CrowdFlower For Sentiment Analysis
How Oracle Uses CrowdFlower For Sentiment AnalysisHow Oracle Uses CrowdFlower For Sentiment Analysis
How Oracle Uses CrowdFlower For Sentiment Analysis
 
Humanizing The Machine
Humanizing The MachineHumanizing The Machine
Humanizing The Machine
 
Open Data Science Conference 2015
Open Data Science Conference 2015Open Data Science Conference 2015
Open Data Science Conference 2015
 
Productive Out-of-the-Box | Tooling with Yeoman to Rapidly Develop Ember.js A...
Productive Out-of-the-Box | Tooling with Yeoman to Rapidly Develop Ember.js A...Productive Out-of-the-Box | Tooling with Yeoman to Rapidly Develop Ember.js A...
Productive Out-of-the-Box | Tooling with Yeoman to Rapidly Develop Ember.js A...
 
Virtual Data Steward: Data Management 3.0
Virtual Data Steward: Data Management 3.0Virtual Data Steward: Data Management 3.0
Virtual Data Steward: Data Management 3.0
 
The State of Enterprise Crowdsourcing 2013
The State of Enterprise Crowdsourcing 2013The State of Enterprise Crowdsourcing 2013
The State of Enterprise Crowdsourcing 2013
 
CrowdFlower University Oct. 21 2013
CrowdFlower University Oct. 21 2013CrowdFlower University Oct. 21 2013
CrowdFlower University Oct. 21 2013
 

Recently uploaded

WomenInAutomation2024: AI and Automation for eveyone
WomenInAutomation2024: AI and Automation for eveyoneWomenInAutomation2024: AI and Automation for eveyone
WomenInAutomation2024: AI and Automation for eveyoneUiPathCommunity
 
Potential of AI (Generative AI) in Business: Learnings and Insights
Potential of AI (Generative AI) in Business: Learnings and InsightsPotential of AI (Generative AI) in Business: Learnings and Insights
Potential of AI (Generative AI) in Business: Learnings and InsightsRavi Sanghani
 
Transcript: New from BookNet Canada for 2024: BNC SalesData and LibraryData -...
Transcript: New from BookNet Canada for 2024: BNC SalesData and LibraryData -...Transcript: New from BookNet Canada for 2024: BNC SalesData and LibraryData -...
Transcript: New from BookNet Canada for 2024: BNC SalesData and LibraryData -...BookNet Canada
 
Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24
Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24
Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24Mark Goldstein
 
Generative Artificial Intelligence: How generative AI works.pdf
Generative Artificial Intelligence: How generative AI works.pdfGenerative Artificial Intelligence: How generative AI works.pdf
Generative Artificial Intelligence: How generative AI works.pdfIngrid Airi González
 
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyesHow to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyesThousandEyes
 
Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...
Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...
Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...panagenda
 
Connecting the Dots for Information Discovery.pdf
Connecting the Dots for Information Discovery.pdfConnecting the Dots for Information Discovery.pdf
Connecting the Dots for Information Discovery.pdfNeo4j
 
Decarbonising Buildings: Making a net-zero built environment a reality
Decarbonising Buildings: Making a net-zero built environment a realityDecarbonising Buildings: Making a net-zero built environment a reality
Decarbonising Buildings: Making a net-zero built environment a realityIES VE
 
Bridging Between CAD & GIS: 6 Ways to Automate Your Data Integration
Bridging Between CAD & GIS:  6 Ways to Automate Your Data IntegrationBridging Between CAD & GIS:  6 Ways to Automate Your Data Integration
Bridging Between CAD & GIS: 6 Ways to Automate Your Data Integrationmarketing932765
 
Microservices, Docker deploy and Microservices source code in C#
Microservices, Docker deploy and Microservices source code in C#Microservices, Docker deploy and Microservices source code in C#
Microservices, Docker deploy and Microservices source code in C#Karmanjay Verma
 
JET Technology Labs White Paper for Virtualized Security and Encryption Techn...
JET Technology Labs White Paper for Virtualized Security and Encryption Techn...JET Technology Labs White Paper for Virtualized Security and Encryption Techn...
JET Technology Labs White Paper for Virtualized Security and Encryption Techn...amber724300
 
A Framework for Development in the AI Age
A Framework for Development in the AI AgeA Framework for Development in the AI Age
A Framework for Development in the AI AgeCprime
 
Infrared simulation and processing on Nvidia platforms
Infrared simulation and processing on Nvidia platformsInfrared simulation and processing on Nvidia platforms
Infrared simulation and processing on Nvidia platformsYoss Cohen
 
Design pattern talk by Kaya Weers - 2024 (v2)
Design pattern talk by Kaya Weers - 2024 (v2)Design pattern talk by Kaya Weers - 2024 (v2)
Design pattern talk by Kaya Weers - 2024 (v2)Kaya Weers
 
Testing tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examplesTesting tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examplesKari Kakkonen
 
Email Marketing Automation for Bonterra Impact Management (fka Social Solutio...
Email Marketing Automation for Bonterra Impact Management (fka Social Solutio...Email Marketing Automation for Bonterra Impact Management (fka Social Solutio...
Email Marketing Automation for Bonterra Impact Management (fka Social Solutio...Jeffrey Haguewood
 
Manual 508 Accessibility Compliance Audit
Manual 508 Accessibility Compliance AuditManual 508 Accessibility Compliance Audit
Manual 508 Accessibility Compliance AuditSkynet Technologies
 
Digital Tools & AI in Career Development
Digital Tools & AI in Career DevelopmentDigital Tools & AI in Career Development
Digital Tools & AI in Career DevelopmentMahmoud Rabie
 
2024 April Patch Tuesday
2024 April Patch Tuesday2024 April Patch Tuesday
2024 April Patch TuesdayIvanti
 

Recently uploaded (20)

WomenInAutomation2024: AI and Automation for eveyone
WomenInAutomation2024: AI and Automation for eveyoneWomenInAutomation2024: AI and Automation for eveyone
WomenInAutomation2024: AI and Automation for eveyone
 
Potential of AI (Generative AI) in Business: Learnings and Insights
Potential of AI (Generative AI) in Business: Learnings and InsightsPotential of AI (Generative AI) in Business: Learnings and Insights
Potential of AI (Generative AI) in Business: Learnings and Insights
 
Transcript: New from BookNet Canada for 2024: BNC SalesData and LibraryData -...
Transcript: New from BookNet Canada for 2024: BNC SalesData and LibraryData -...Transcript: New from BookNet Canada for 2024: BNC SalesData and LibraryData -...
Transcript: New from BookNet Canada for 2024: BNC SalesData and LibraryData -...
 
Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24
Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24
Arizona Broadband Policy Past, Present, and Future Presentation 3/25/24
 
Generative Artificial Intelligence: How generative AI works.pdf
Generative Artificial Intelligence: How generative AI works.pdfGenerative Artificial Intelligence: How generative AI works.pdf
Generative Artificial Intelligence: How generative AI works.pdf
 
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyesHow to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
How to Effectively Monitor SD-WAN and SASE Environments with ThousandEyes
 
Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...
Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...
Why device, WIFI, and ISP insights are crucial to supporting remote Microsoft...
 
Connecting the Dots for Information Discovery.pdf
Connecting the Dots for Information Discovery.pdfConnecting the Dots for Information Discovery.pdf
Connecting the Dots for Information Discovery.pdf
 
Decarbonising Buildings: Making a net-zero built environment a reality
Decarbonising Buildings: Making a net-zero built environment a realityDecarbonising Buildings: Making a net-zero built environment a reality
Decarbonising Buildings: Making a net-zero built environment a reality
 
Bridging Between CAD & GIS: 6 Ways to Automate Your Data Integration
Bridging Between CAD & GIS:  6 Ways to Automate Your Data IntegrationBridging Between CAD & GIS:  6 Ways to Automate Your Data Integration
Bridging Between CAD & GIS: 6 Ways to Automate Your Data Integration
 
Microservices, Docker deploy and Microservices source code in C#
Microservices, Docker deploy and Microservices source code in C#Microservices, Docker deploy and Microservices source code in C#
Microservices, Docker deploy and Microservices source code in C#
 
JET Technology Labs White Paper for Virtualized Security and Encryption Techn...
JET Technology Labs White Paper for Virtualized Security and Encryption Techn...JET Technology Labs White Paper for Virtualized Security and Encryption Techn...
JET Technology Labs White Paper for Virtualized Security and Encryption Techn...
 
A Framework for Development in the AI Age
A Framework for Development in the AI AgeA Framework for Development in the AI Age
A Framework for Development in the AI Age
 
Infrared simulation and processing on Nvidia platforms
Infrared simulation and processing on Nvidia platformsInfrared simulation and processing on Nvidia platforms
Infrared simulation and processing on Nvidia platforms
 
Design pattern talk by Kaya Weers - 2024 (v2)
Design pattern talk by Kaya Weers - 2024 (v2)Design pattern talk by Kaya Weers - 2024 (v2)
Design pattern talk by Kaya Weers - 2024 (v2)
 
Testing tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examplesTesting tools and AI - ideas what to try with some tool examples
Testing tools and AI - ideas what to try with some tool examples
 
Email Marketing Automation for Bonterra Impact Management (fka Social Solutio...
Email Marketing Automation for Bonterra Impact Management (fka Social Solutio...Email Marketing Automation for Bonterra Impact Management (fka Social Solutio...
Email Marketing Automation for Bonterra Impact Management (fka Social Solutio...
 
Manual 508 Accessibility Compliance Audit
Manual 508 Accessibility Compliance AuditManual 508 Accessibility Compliance Audit
Manual 508 Accessibility Compliance Audit
 
Digital Tools & AI in Career Development
Digital Tools & AI in Career DevelopmentDigital Tools & AI in Career Development
Digital Tools & AI in Career Development
 
2024 April Patch Tuesday
2024 April Patch Tuesday2024 April Patch Tuesday
2024 April Patch Tuesday
 

Expert Crowdsourcing with Flash Teams | CrowdConf 2013 poster

  • 1. Expert Crowdsourcing with Flash Teams Daniela Retelny, Sébastien Robaszkiewicz, Alexandra To, Michael Bernstein {dretelny, robi, ato1120, msb}@cs.stanford.edu Crowdsourcing is Plateauing in Complexity Flash Team Examples Crowd workers struggle to complete open ended and highly Our evaluation of flash teams to date includes the successful completion of eight different flash team examples, utilizing different team structures and workflows. The flash teams varied in size, complexity and expertise and interdependent tasks. they produced outputs that ranged in completion time and quality. The images below depict the team structures, Despite good intentions, teams of crowd experts are workflows and output from three napkin sketch, animation and MOOC flash team examples we completed. typically neither effective nor efficient at these tasks. Expert crowdsourcing lacks generalizable techniques for guiding crowds through complex tasks. Flash Teams We introduce flash teams, modular team structures that computationally guide a team of crowd experts reproducibly and at scale. These modular structures enable crowd expert teams to grow, adapt, and recombine into larger organizations. Defining attributes of flash teams: MODULARITY: self-contained and replicable team structures composed of crowd experts that are computationally guided. ELASTICITY: parameterizable team growth on demand, allowing teams to grow and shrink as expertise are needed. PIPELINING: parallelism enabled by defining interdependencies and handoffs. Methods: To explore the feasibility and scope of flash teams, and understand the effectiveness of modularity, elasticity and pipelining, we gathered thirteen different flash teams to complete complex tasks ranging from design to education. Foundry, a Flash Team Authoring Platform To encapsulate the foundations of the flash teams formalism and lessen We recruit experts through oDesk, a crowdsourcing platform the threshold to creating flash teams, we created Foundry, a web platform that allows requesters to create, reuse and recombine flash team for expert work that allows requesters to post a request under a specific expertise category (e.g., Python development, structures and workflows and allows team members to coordinate tasks and track progress. The image to the right shows a screenshot of Foundry. sound production, or visual design). http://hci.stanford.edu STANFORD HCI GROUP