This presentation was provided by Kyle Lo of The Allen Institute for AI (AI2) during the NISO hot topic event "Preprints." The virtual conference was held on April 21, 2021.

1. AI-infused interfaces for
reading AI preprints
Kyle Lo
Semantic Scholar Research, Allen Institute for Artificial Intelligence
NISO 2021
April 21, 2021

2. Preprints
High volume. Variable quality (no peer review).
Metadata complications. Faster dissemination of research.
Keeping up-to-date
Passive. Recommendation. Habit. Little at a time.

3. Preprints
High volume. Variable quality (no peer review).
Metadata complications. Faster dissemination of research.
Keeping up-to-date
Passive. Recommendation. Habit. Little at a time.

4. Preprints
High volume. Variable quality (no peer review).
Metadata complications. Faster dissemination of research.
Keeping up-to-date
Passive. Recommendation. Habit. Little at a time.

5. Preprints
High volume. Variable quality (no peer review).
Metadata complications. Faster dissemination of research.
Keeping up-to-date
Passive. Recommendation. Habit. Little at a time.

6. Preprints
High volume. Variable quality (no peer review).
Metadata complications. Faster dissemination of research.
Keeping up-to-date
Passive. Recommendation. Habit. Little at a time.

7. arXiv
The AI research community’s relationship with preprints
Improving the reading experience for arXiv preprints
Keeping up-to-date
Passive. Recommendation. Habit. Little at a time.

8. arXiv
The AI research community’s relationship with preprints
Improving the reading experience for arXiv preprints
Keeping up-to-date
Passive. Recommendation. Habit. Little at a time.

9. arXiv
The AI research community’s relationship with preprints
Improving the reading experience for arXiv preprints
Keeping up-to-date
Passive. Recommendation. Habit. Little at a time.

10. Who we are

11. Allen Institute for AI
● Seattle, WA, USA

12. Allen Institute for AI
● Seattle, WA, USA
● Nonprofit research

13. Allen Institute for AI
● Seattle, WA, USA
● Nonprofit research
● Founded by Paul Allen

14. Allen Institute for AI
● Seattle, WA, USA
● Nonprofit research
● Founded by Paul Allen
● University of Washington

15. Allen Institute for AI
● Seattle, WA, USA
● Nonprofit research
● Founded by Paul Allen
● University of Washington
● AI for the Common Good

16. Semantic Scholar
● Literature search

17. Semantic Scholar
● Literature search
● Keeping up-to-date

18. Semantic Scholar
● Literature search
● Keeping up-to-date
○ Alerts for new papers

19. Semantic Scholar
● Literature search
● Keeping up-to-date
○ Alerts for new papers
○ Recommendations
○ etc.

20. Semantic Scholar Research
● Natural language processing
○ Automatic summarization
○ Question answering
● Computer vision
○ Figure & table extraction
● Human AI interaction
○ Interpretable models
○ Augmented reading interfaces

21. Semantic Scholar Research
● Natural language processing
○ Automatic summarization
○ Question answering
● Computer vision
○ Figure & table extraction
● Human AI interaction
○ Interpretable models
○ Augmented reading interfaces

22. Semantic Scholar Research
● Natural language processing
○ Automatic summarization
○ Question answering
● Computer vision
○ Figure & table extraction
● Human AI interaction
○ Interpretable models
○ Augmented reading interfaces

23. Semantic Scholar Research
● Natural language processing
○ Automatic summarization
○ Question answering
● Computer vision
○ Layout understanding
● Human AI interaction
○ Interpretable models
○ Augmented reading interfaces

24. Semantic Scholar Research
● Natural language processing
○ Automatic summarization
○ Question answering
● Computer vision
○ Layout understanding
● Human AI interaction
○ Augmented reading interfaces

25. AI research ↔ arXiv preprints

26. 13% 21%
11% 16%
2019 - 2020 2020 - 2021
% papers are
arXiv preprints
% viewers read
arXiv preprints
100k most viewed AI papers on
Semantic Scholar

27. 13% 21%
11% 16%
2019 - 2020 2020 - 2021
% papers are
arXiv preprints
% viewers read
arXiv preprints
100k most viewed AI papers on
Semantic Scholar

28. 13% 21%
11% 16%
2019 - 2020 2020 - 2021
% papers are
arXiv preprints
% viewers read
arXiv preprints
100k most viewed AI papers on
Semantic Scholar

29. 13% 21%
11% 16%
2019 - 2020 2020 - 2021
% papers are
arXiv preprints
% viewers read
arXiv preprints
100k most viewed AI papers on
Semantic Scholar

30. Why do AI researchers keep up
with arXiv preprints?

31. A lot of AI papers have an arXiv preprint
Jan - Mar

32. Typical AI conference publication
Submission
deadline Accepted
3-4 months

33. Typical AI conference publication
Submission
deadline Accepted Conference
3-4 months
Paper available
online (open) 🥳

34. Typical AI conference publication
Anonymity
begins
Submission
deadline Accepted Conference
1 month
Paper available
online (open) 🥳

35. Typical AI conference publication
Anonymity
begins
Submission
deadline Accepted Conference
8 month
Paper available
online (open) 🥳

36. 1. arXiv helps get your work out faster
Anonymity
begins
Submission
deadline Accepted Conference
4-5 months
Upload our paper to
arXiv 😎

37. 2. Risk & flag-planting
Anonymity
begins
Submission
deadline Accepted Conference
4-5 months
What if concurrent, similar
work appears? 😨

38. 2. Risk & flag-planting
Anonymity
begins
Submission
deadline Accepted Conference
4-5 months
What if my paper is
rejected & my work
becomes stale? 😭

39. 2. Risk & flag-planting
Anonymity
begins
Submission
deadline
Conference
Upload our paper to
arXiv 😅

40. 3. AI research is fast & work goes stale quickly
% of citations of “recent” papers (<1 year)

41. 4. Incentives for preprinting on arXiv

42. 4. Incentives for preprinting on arXiv
“... papers that were eventually accepted for publication
at top-tier CS conferences, and were posted on arXiv
either before or after the acceptance notification…”

43. 4. Incentives for preprinting on arXiv
“...papers submitted to arXiv before acceptance have, on
average, 65% more citations in the following year
compared to papers submitted after…”

44. 5. arXiv policies for top AI conferences
Q: Is arXiv preprinting allowed?
“Submission of the paper to archival repositories such as arXiv are allowed.” - ICLR 2020
”For arXiv submissions, <date> is the submission deadline” - ACL 2020

45. Q: Does my submission need to cite arXiv papers that are related to my work?
“Consistent with good academic practice, the authors should cite all sources that
inspired and informed their work. This said, asking authors to thoroughly compare their
work with arXiv reports that appeared shortly before the submission deadline imposes
an unreasonable burden.” - CVPR 2020
5. arXiv policies for top AI conferences

46. Lots of reasons to:
Submit papers to arXiv
+
Keep up-to-date with arXiv

47. Improving the reading
experience for arXiv preprints

48. Reading papers is hard!

49. Preprints are read in standard PDF readers

50. Preprints are read in standard PDF readers

51. Publishers enable nice web interfaces

52. Let’s make preprint PDFs more like that!

53. Reading papers is still hard!

54. How much easier can we make reading?

55. How much easier can we make reading?

56. How much easier can we make reading?

57. How much easier can we make reading?

58. How much easier can we make reading?

59. Demo
Link

60. User study with readers of AI preprints
AI researcher
Everything
No definitions
Status quo

61. User study with readers of AI preprints
AI researcher
Everything
Partial
Basic

62. User study with readers of AI preprints

63. Results - Less time spent looking for answers
Partial
Everything
Basic

64. Results - Less thrashing about in the PDF
Partial
Everything
Basic

65. Results - More confidence answers are correct
Partial
Everything
Basic
Unsure Confident

66. Made possible with LaTeX sources from arXiv

67. Only possible with LaTeX sources from arXiv

68. Only possible with LaTeX sources from arXiv

69. Only possible with LaTeX sources from arXiv

70. Conclusion
Reading preprints is part of normal research workflow
Invest in tools to help with all aspects of doing research
beyond search & discovery
Open data can be unexpectedly helpful for tool development

71. Thanks!
kylel@allenai.org
@kylelostat
Head, Andrew, Kyle Lo, Dongyeop Kang, Raymond Fok, Sam Skjonsberg, Daniel S. Weld and Marti A. Hearst.
“Augmenting Scientific Papers with Just-in-Time, Position-Sensitive Definitions of Terms and Symbols.”
ArXiv (2020). To appear at CHI 2020.