Overview of what makes good systems research for the 2012 NSF Social Computing Systems (SoCS) PI Meeting held at the University of MIchigan, Ann Arbor (Jun 17-19, 2012)
Crowdsourcing & Human Computation Labeling Data & Building Hybrid SystemsMatthew Lease
Tutorial given at SIAM Data Mining Conference (http://www.siam.org/meetings/sdm13/), May 3, 2013. Based on earlier tutorials given jointly with Omar Alonso from Microsoft Bing.
When building digital products and services, we are designing complex systems.We need to think the customer experience through on several channels, figure out the system architecture, gain understanding through data and research, decide what to iterate... - not easy, but fun!
In this keynote talk given at Agile Cambridge 2016, Johanna introduces core systems thinking principles for designing better services, discussed how data and feedback mechanisms help us understand what is going on in a system, and addressed the challenge of bringing about change in a system.
An Introduction to Human Computation and Games With A Purpose - Part IAlessandro Bozzon
Crowdsourcing and human computation are novel disciplines that enable the design of computation processes that include humans as actors for task execution. In such a context, Games With a Purpose are an effective mean to channel, in a constructive manner, the human brainpower required to perform tasks that computers are unable to per- form, through computer games. This tutorial introduces the core research questions in human computation, with a specific focus on the techniques required to manage structured and unstructured data. The second half of the tutorial delves into the field of game design for serious task, with an emphasis on games for human computation purposes. Our goal is to provide participants with a wide, yet complete overview of the research landscape; we aim at giving practitioners a solid understanding of the best practices in designing and running human computation tasks, while providing academics with solid references and, possibly, promising ideas for their future research activities.
Crowdsourcing & Human Computation Labeling Data & Building Hybrid SystemsMatthew Lease
Tutorial given at SIAM Data Mining Conference (http://www.siam.org/meetings/sdm13/), May 3, 2013. Based on earlier tutorials given jointly with Omar Alonso from Microsoft Bing.
When building digital products and services, we are designing complex systems.We need to think the customer experience through on several channels, figure out the system architecture, gain understanding through data and research, decide what to iterate... - not easy, but fun!
In this keynote talk given at Agile Cambridge 2016, Johanna introduces core systems thinking principles for designing better services, discussed how data and feedback mechanisms help us understand what is going on in a system, and addressed the challenge of bringing about change in a system.
An Introduction to Human Computation and Games With A Purpose - Part IAlessandro Bozzon
Crowdsourcing and human computation are novel disciplines that enable the design of computation processes that include humans as actors for task execution. In such a context, Games With a Purpose are an effective mean to channel, in a constructive manner, the human brainpower required to perform tasks that computers are unable to per- form, through computer games. This tutorial introduces the core research questions in human computation, with a specific focus on the techniques required to manage structured and unstructured data. The second half of the tutorial delves into the field of game design for serious task, with an emphasis on games for human computation purposes. Our goal is to provide participants with a wide, yet complete overview of the research landscape; we aim at giving practitioners a solid understanding of the best practices in designing and running human computation tasks, while providing academics with solid references and, possibly, promising ideas for their future research activities.
Talk given at Delft University speaker series on "Crowd Computing & Human-Centered AI" (https://www.academicfringe.org/). November 23, 2020. Covers two 2020 works:
(1) Anubrata Das, Brandon Dang, and Matthew Lease. Fast, Accurate, and Healthier: Interactive Blurring Helps Moderators Reduce Exposure to Harmful Content. In Proceedings of the 8th AAAI Conference on Human Computation and Crowdsourcing (HCOMP), 2020.
Alexander Braylan and Matthew Lease. Modeling and Aggregation of Complex Annotations via Annotation Distances. In Proceedings of the Web Conference, pages 1807--1818, 2020.
Presentation given at the Linguistic Data Consortium (LDC), University of Pennsylvania, April 2019. Based on presentations at the 6th ACM Collective Intelligence Conference, 2018 and the 6th AAAI Conference on Human Computation & Crowdsourcing (HCOMP), 2018. Blog post: https://blog.humancomputation.com/?p=9932.
AI & Work, with Transparency & the Crowd Matthew Lease
Invited talk at the 2019 AAAI Fall Symposium (https://aaai.org/Symposia/Fall/fss19.php) on Artificial Intelligence and Work (https://waim.network/fs19).
Training Seminar - The Data Design ProcessMaxwell Taylor
An introductory talk into the basic, high-level principles of data system design. Presented to OSU's Office of Institutional Analytics and Reporting Dec 2017
Talk given at Delft University speaker series on "Crowd Computing & Human-Centered AI" (https://www.academicfringe.org/). November 23, 2020. Covers two 2020 works:
(1) Anubrata Das, Brandon Dang, and Matthew Lease. Fast, Accurate, and Healthier: Interactive Blurring Helps Moderators Reduce Exposure to Harmful Content. In Proceedings of the 8th AAAI Conference on Human Computation and Crowdsourcing (HCOMP), 2020.
Alexander Braylan and Matthew Lease. Modeling and Aggregation of Complex Annotations via Annotation Distances. In Proceedings of the Web Conference, pages 1807--1818, 2020.
Presentation given at the Linguistic Data Consortium (LDC), University of Pennsylvania, April 2019. Based on presentations at the 6th ACM Collective Intelligence Conference, 2018 and the 6th AAAI Conference on Human Computation & Crowdsourcing (HCOMP), 2018. Blog post: https://blog.humancomputation.com/?p=9932.
AI & Work, with Transparency & the Crowd Matthew Lease
Invited talk at the 2019 AAAI Fall Symposium (https://aaai.org/Symposia/Fall/fss19.php) on Artificial Intelligence and Work (https://waim.network/fs19).
Training Seminar - The Data Design ProcessMaxwell Taylor
An introductory talk into the basic, high-level principles of data system design. Presented to OSU's Office of Institutional Analytics and Reporting Dec 2017
Understanding The Performance Of Addressed AdmailChristine Croft
A review of tests and pilots conducted by Canada Post between 2006 and 2008 with select Clients demonstrating performance of Addressed admail from Brand awareness to sales.
This is the presentation of the Juan Cruz-Benito’s PhD “On data-driven systems analyzing, supporting and enhancing users’ interaction and experience” that was defended on September 3rd, 2018 in the Faculty of Sciences at University of Salamanca Spain. This PhD was graded with the maximum qualification “Sobresaliente Cum Laude”.
A Pragmatic Perspective on Software VisualizationArie van Deursen
Slides of the keynote presentation at the 5th International IEEE/ACM Symposium on Software Visualization, SoftVis 2010. Salt Lake City, USA, October 2010.
An introduction to Force11 and Beyond the PDF meetings presented to the WWW2013 meeting in Rio de Janeiro, Brazil on May 15, 2013. Presenters were: Ivan Herman, W3C; Sweitze Roffel, Elsevier; David De Roure, University of Oxford; and Todd Carpenter, NISO.
Digital Humanities in Practice, DHC 2012Monica Bulger
This paper presents findings of a fieldwork study that explored research practices, challenges, and directions in contemporary digital humanities scholarship. The study was conducted in the period April-October, 2010, as part of two research projects of the Royal Netherlands Academy of Arts and Sciences and the Oxford Internet Institute. The studies included observations, focus groups, and in-depth interviews with digital humanities scholars, policymakers, and funders, with a focus on developers and users of digital resources for humanities research. The study involved 92 participants from over 25 institutions in 5 countries.
Presented by: Monica Bulger, Eric T. Meyer, and Sally Wyatt, with Smiljana Antonijevic
Presentation based on fieldwork research conducted at digital humanities institutions in Europe and the USA; delivered at Click on Knowledge conference in Copenhagen (http://engerom.ku.dk/clickonknowledge/)
DevOps is becoming the new "buzzworld", like "SOA" or "Agile" did in the past years.
In this presentation we will try to separate the idea from the marketing following "DevOps" idea since its inception in the 2008, from its relationship with Agile and other, similar, ideas in other disciplines.
Then we will discuss why these ideas work and how they're applied in an IT context.
CODATA International Training Workshop in Big Data for Science for Researcher...Johann van Wyk
Presentation at NeDICC Meeting on 16 July 2014. Feedback from CODATA International Training Workshop in Big Data for Science for Researchers from Emerging and Developing Countries, Beijing, China, 5-20 June 2014
Normal Labour/ Stages of Labour/ Mechanism of LabourWasim Ak
Normal labor is also termed spontaneous labor, defined as the natural physiological process through which the fetus, placenta, and membranes are expelled from the uterus through the birth canal at term (37 to 42 weeks
The French Revolution, which began in 1789, was a period of radical social and political upheaval in France. It marked the decline of absolute monarchies, the rise of secular and democratic republics, and the eventual rise of Napoleon Bonaparte. This revolutionary period is crucial in understanding the transition from feudalism to modernity in Europe.
For more information, visit-www.vavaclasses.com
Safalta Digital marketing institute in Noida, provide complete applications that encompass a huge range of virtual advertising and marketing additives, which includes search engine optimization, virtual communication advertising, pay-per-click on marketing, content material advertising, internet analytics, and greater. These university courses are designed for students who possess a comprehensive understanding of virtual marketing strategies and attributes.Safalta Digital Marketing Institute in Noida is a first choice for young individuals or students who are looking to start their careers in the field of digital advertising. The institute gives specialized courses designed and certification.
for beginners, providing thorough training in areas such as SEO, digital communication marketing, and PPC training in Noida. After finishing the program, students receive the certifications recognised by top different universitie, setting a strong foundation for a successful career in digital marketing.
A Strategic Approach: GenAI in EducationPeter Windle
Artificial Intelligence (AI) technologies such as Generative AI, Image Generators and Large Language Models have had a dramatic impact on teaching, learning and assessment over the past 18 months. The most immediate threat AI posed was to Academic Integrity with Higher Education Institutes (HEIs) focusing their efforts on combating the use of GenAI in assessment. Guidelines were developed for staff and students, policies put in place too. Innovative educators have forged paths in the use of Generative AI for teaching, learning and assessments leading to pockets of transformation springing up across HEIs, often with little or no top-down guidance, support or direction.
This Gasta posits a strategic approach to integrating AI into HEIs to prepare staff, students and the curriculum for an evolving world and workplace. We will highlight the advantages of working with these technologies beyond the realm of teaching, learning and assessment by considering prompt engineering skills, industry impact, curriculum changes, and the need for staff upskilling. In contrast, not engaging strategically with Generative AI poses risks, including falling behind peers, missed opportunities and failing to ensure our graduates remain employable. The rapid evolution of AI technologies necessitates a proactive and strategic approach if we are to remain relevant.
Francesca Gottschalk - How can education support child empowerment.pptxEduSkills OECD
Francesca Gottschalk from the OECD’s Centre for Educational Research and Innovation presents at the Ask an Expert Webinar: How can education support child empowerment?
1. James
&
Friends’
Systems
How
To
A
Guide
to
Systems
&
Applica3ons
Research!
James Landay
Short-Dooley Professor
Computer Science & Engineering
University of Washington " " "
2012 NSF SoCS PI Meeting
University of Michigan
June19, 2012
2. What
Type
of
Researcher
are
You?
A
-‐
Discoverer
B
-‐
Ques=oner
C
-‐
Maker
4. QuesCons
Answered
What
are
the
key
a6ributes
of
strong
systems
work?
What
are
the
best
techniques
to
evaluate
systems
&
when
do
they
make
sense
to
use?
Which
HCI
techniques
do
not
make
sense
in
systems
research?
How
do
you
disCnguish
good
research
from
bad?
What
are
your
favorite
systems
research
projects
&
why?
What
makes
a
good
social
compuCng
systems
research
project
&
what
are
your
favorites?
5. Key
A6ributes
of
Strong
Systems
Research
Compelling
Target
• “Solves
a
concrete,
compelling
problem
with
demonstrated
need”
Strong
moCvaCon
for
the
problem
w/
need
based
in
users,
costs,
or
tech
issues
• “Solves
a
compelling
set
of
problems
using
a
unifying
set
of
principles”
The
principles
Ce
the
set
of
problems
together
• “Explores
how
people
will
interact
with
computers
in
the
future”
Takes
into
account
technical
&
usage
trends
6. Key
A6ributes
of
Strong
Systems
Research
Technical
Challenge
• “Goes
beyond
rou3ne
so@ware
engineering”
Requires
novel,
non-‐trivial
algorithms
or
configura=on
of
components
Deployed
When
Possible
• “system
is
deployed
&
intended
benefits
&
unexpected
outcomes
documented”
Not
required,
but
gold
standard
for
most
systems
work
8. EvaluaCon
Methods
for
Systems
Research
“it
depends
upon
the
contribu3on”
“match
the
type
of
evalua3on
with
how
you
expect
the
system
to
be
used”
“mul3tude
of
metrics
to
give
you
a
holis3c
view”
9. Idea
EvaluaCon
Overall
value
of
system
or
applica2on
• If
extremely
novel,
the
fact
that
it
works
&
logical
argument
to
explore
“boundaries
of
value”
• Real
world
deployment
(expensive
in
Cme
&
effort)
10. Technical
EvaluaCon
Measure
key
aspects
from
technical
perspec2ve
1) Toolkit
è
expressiveness
(“Can
I
build
it?”)
efficiency
(“How
long
will
it
take?”)
accessibility
(“Do
I
know
how?”)
2) Performance
improvement
è
benchmark
(error,
scale,
effiencey…)
3) Novel
component
è
controlled
lab
study*
*
may
not
generalize
to
real-‐world
condiCons
11. EffecCveness
EvaluaCon
1) Usability
improvement
è
controlled
lab
study*
2) Conceptual
understanding
è
case
studies
w/
a
few
real
external
users
13. HCI
Techniques
That
Don’t
Make
Sense
• Usability
Tests
&
A/B
tests
“can’t
tell
much
about
complex
systems”
• Contextual
Inquiry
“good
for
today,
but
can’t
predict
tomorrow”
• TradiConal
controlled
empirical
studies
“not
meaningful
to
isolate
small
number
of
variables”
15. How
Do
You
Tell
Good
From
Bad?
Good
• “Combines
a
lot
of
exisCng
ideas
together
in
new
ways
…
it
really
is
a
case
of
the
sum
being
greater
than
the
parts”
• “PotenCal
for
impact”
• “Tries
to
solve
an
important
problem
using
novel
technology.
It
is
creaCve
&
raises
new
possibiliCes
for
human-‐computer
interacCon.”
Bad
• “Fails
to
jusCfy
the
problem
it
addresses,
uses
off-‐the-‐shelf
technology,
or
does
not
teach
anything
new
about
how
people
interact
with
computers.”
• “too
many
concepts—true
insight
has
a
simplicity
to
it”
• “a
feature,
but
not
a
product
or
a
business”
17. HYDROSENSE
Froehlich,
Larson,
Fogarty,
Patel
+
crucial
problems,
surprising
how
well
can
do
w/
few
sensors
18. prefab
Dixon
&
Fogarty
+
“compelling,
but
not
obvious
best
way…
pushes
as
far
as
can”
19. Whyline
Ko
&
Myers
+
“based
on
studies
of
how
people
debug
today”
+
“insight
that
almost
all
quesCons
in
form
of
why
or
“whynot”
20. $100
InteracCve
Whiteboard
Johnny
Lee
+
“repurposes
current
tools
in
a
creaCve
way
to
solve
a
problem
that
no
one
would
have
imagined
possible
before
he
did
it”
21. What
Makes
a
Good
Social
CompuCng
System?
• “criteria
above
+
involves
social
interacCon
as
a
main
feature..
Facilitates
new
or
enhanced
forms
of
collaboraCve
parCcipaCon”
• “combines
good
theory
with
good
systems
building”
• “finds
new
ways
of
combining
the
best
of
people
and
computers
together”
• “good
answers
to
why
people
will
parCcipate
at
scale”
• “a
model
of
individual
user
behavior;
a
model
of
aggregated
social
behavior;
use
that
model
to
build
a
novel
system”
• “make
the
system
work
in
the
face
of
malicious
behavior”
22. Soylent
Bernstein,
et.
al.
+
“innovaCve
applicaCons
for
growing
trend
(crowdsourcing)”
+
“led
to
new
ideas
for
how
to
organize
people
&
computers”
+
“contributed
a
general
design
pa6ern
(Find-‐Fix-‐Verify)”
23. Group
Lens
/
Movie
Lens
Riedl,
Herlocker,
Lam,
et.
al.
+
“built
their
own
community
&
used
it
to
develop
a
long
list
of
compelling
research
results”
+
“incorporates
lots
of
social
science
ideas,
led
to
innovaCons
in
collaboraCve
filtering,
and
has
actual
deployment
&
lots
of
use”
24. Many-‐Eyes
Heer,
Viégas,
Wa6enberg
+
“recognized
the
social
nature
of
people’s
relaConships
to
data
visualizaCons
&
provided
a
planorm
for
disseminaCng”
+
“significant
real-‐world
impact
in
introducing
larger
audiences
to
a
variety
of
visualizaCon
techniques”
25. Thanks
to
Contributors
Ben
Bederson,
University
of
Maryland
Ed
H.
Chi,
Google
Research
Saul
Greenberg,
University
of
Calgary
François
GuimbreCère,
Cornell
University
Jeffrey
Heer,
Stanford
University
Jason
Hong,
Carnegie
Mellon
University
Tessa
Lau,
IBM
Research
Dan
Olsen,
Brigham
Young
University