Scholarly Document Processing Research in the Age of AI

Scholarly Document Processing
Research in the Age of AI
Min-Yen Kan
National University of Singapore
Slides @ http://bit.ly/kan-sdp22
17 Oct 2022 3rd SDProc @ COLING 2022 1

Warning: This is a participatory keynote!
…that is, there is a pop quiz. You have been warned! 🤣🤣
Please access the poll at
http://pollev.com/knmnyn
Do skip the name registration

Fast and Slow
Kahneman and Tversky, Thinking Fast and Slow
Daniel Kahneman
System 1 System 2
Fast Slow
Automatic Controlled
Intuitive Analytical
Parallel Serial
Associative Logical

Neural Nets – System 1
Andrew Ng
System 1 System 2
Fast Slow
Parallel Serial
Associative Logical
✏Your Turn: What do you think the
loss function of research should be?

The Age of Accelerations
Friedman, Thank You for Being Late
His three accelerations
• Moore’s law
• Globalization
• Mother Nature
Kurzweil’s “Second half of the
chessboard”
Thomas Friedman

The Age of Accelerations
Friedman, Thank You for Being Late
Our three accelerants (take your pick)
• arXiv
• PapersWithCode
• (Semantic) Scholar
Kurzweil’s “Second half of the
chessboard”
Thomas Friedman

What about System 2?
Are there scholarly problems that
require more analytical, logical,
and sustained thinking?
Absolutely!
System 1 System 2
Fast Slow
Parallel Serial
Associative Logical

A Brief
History of
Science
A fast primer
Photo Credit: 오힘찬 @ WikimediaCommons (CC SA-BY 4.0)

From Astrology to Astronomy
Ptolemy Copernicus

Astronomy 2.0 and 3.0
Galileo and Kepler Newton

Kuhn – challenging accumulative growth
Kuhn, The Structure of Scientific Revolutions
Paradigm Shift
Normal Science
To think about: What age is SDP in now?
Thomas Kuhn

Science is a verb
… in the sense that it is a method (activity) involving the making of hypotheses,
the design of experiments and the analysis of data. But a critical part of the
scientific process is the conversation phase after the experimentation is
done. Scientists share their findings with the broader community through
publications or presentations at meetings. What happens next is a back-and-
forth discussion including a critique of methods or interpretation, and a
comparison with previous findings.
If there are flaws in the experimental design or interpretation, … scientists
need to be willing to hear and respond to feedback. If there are conflicting
results, it may require additional hypothesis making and
experimentation. Only when the conversation runs its course do the conclusions
become a part of accepted scientific understanding.
Steve Savage’s post on Science 2.0

Science in the Age of AI
Video Source: Video by RedEye450 from Pexels

Loss function of research
Beam search analogy
Accelerations make the
gradient steeper
Overload favors System 1
Publish or Perish
Suboptimal local minima

What affordances does AI yield?
Better System 1!
e.g., Neural Architecture
Search (NAS)
Figures from Ren et al. 2021 ACM Comput. Surv. 37(4)

System 1 and 2 work together
One way: System 1 brings data for System 2 to deliberate with
System 2 gives feedback (end-to-end) to System 1
Neither system is perfect but the whole is better than the parts
(multi-view learning)
Let’s connect it back to our societal research loss function

Scholarly
Document
Processing for
System 2
Slowing down
Photo Credits: sizumaru @ Flickr

Challenges for System 2 SDP
1. Discovering Adjacent Possibles (Branch Out)
2. Uncovering Discrepancies (Dive Deep)
3. Finding Provenance (Travel Back)

1. Discovering Adjacent Possibles
Liquid Networks
The Slow Hunch
Serendipity
Exaptation
Steven Johnson
Johnson, Where Good Ideas Come From

Confirmation Bias in Recommender Systems
We train search and recommender systems, but on historical data
This results in confirmation bias (more like this)
But if we want to afford System 2 thinking, we want
serendipitous recommendation (to learn what we don’t know)
Need to capture multimodal evidence and laborious human assessment

Next Gen Platforms
For discoverability:
• Setting exploration criteria
• Reproducible search
• Suggesting alternative paths and terminologies
For discussion, collaboration and crediting:
• “Calm” for Scientists (arXiv off)
• MIT Deliberatorium
• Big Science initiatives
& Toolkits (not everyone wants to do it
globally and publicly)

2. Uncovering Discrepancies
✏Your Turn:
Is coffee bad
for you?

2. Uncovering Discrepancies –
Countering the Streetlight Effect
What happens next is a back-and-forth discussion including a
critique of methods or interpretation, and a comparison with
previous findings.
If there are flaws in the experimental design or interpretation,
… scientists need to be willing to hear and respond to feedback.
Communities do not sufficiently report negative
results
Difficult to organize discrepancies for systematic
exploration, thus we cannot question the
establishment
Related: Davies et al. Promoting inclusive metrics of success and
impact to dismantle a discriminatory reward system in science.
Photo by Guilherme Rossi @ Pexels

Aids for Paradigm Shifts
Systematic reviews for what doesn’t work
“Our techniques improve on Dataset X but less well on Y.
Uncover choices left (un)stated by authors
“We compare against current relevant baselines [1, 2, 3]”
Machine reading of Limitations and Ethical Consideration sections
, 3

https://symplectic.co.uk/guest-blog/research-data-mechanics/
✏Your Turn: What
about citation
half-life? How is it
changing?

3. Finding Provenance
Perhaps surprisingly, citations half-life has lengthened in most fields.
Does this mean that we are finding the right works?
Martín-Martín et al. Back to the past: on the shoulders of an academic search engine giant
Davis and Cochran Cited Half-Life of the Journal Literature

Aids for Finding Provenance
Paraphrase, terminology and simplification services in situ
(stay tuned for Head’s keynote)
Lower the barrier for communication. Platforms for easier means for
discussing problems and knowing of furthering research
Who cares about my research?
(Multi-hop) Trace terms and ideas back to their source

We need to participate in Science!
This is the last activity, I promise!
http://pollev.com/knmnyn
✏Your Turn: Please use
your own judgement to
rank the three
challenges presented

Conclusion: SDP needs to
get involved in Science
Let’s be deliberate about our tools for
science. Care to discuss?
Diversity and inclusion are also important
for holistic progress in science.
Thanks to:
WING members:
George Huang Po-Wei
Yajing Yang
Abhinav Ramesh Kashyap
Muthu Kumar Chandrasekaran
Collaborators:
Min Song
Namhee Kim
and many more previous WING members,
and my family, and
all of you who’ve attended physically and
virtually to listen!
Thank you!
Yanxia Qin
Aminesh Prasad
Kazunari Sugiyama
Juyoung An

Scholarly Document Processing Research in the Age of AI

Recommended

Recommended

More Related Content

Similar to Scholarly Document Processing Research in the Age of AI

Similar to Scholarly Document Processing Research in the Age of AI (20)

Recently uploaded

Recently uploaded (20)

Scholarly Document Processing Research in the Age of AI