Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Challenges for ACL
ACL Presidential Address 2017
Joakim Nivre
(A)CL is booming!
0
300
600
900
1200
1500
2008 2009 2010 2011 2012 2013 2014 2015 2016 2017
ACL submissions
(A)CL is booming!
0
300
600
900
1200
1500
2008 2009 2010 2011 2012 2013 2014 2015 2016 2017
ACL submissions
92
93
94
95
96
2008 2009 2010 20...
The future is bright!
Challenges for ACL
Equity and Diversity
Publishing and Reviewing
Good Science
Can you spot the mistake?
ACL Presidents 2009–2017
Only four of them are
wearing funny hats?
Can you spot the mistake?
ACL Presidents 2009–2017
Can you spot the mistake?
ACL Presidents 2009–2017
Has Jack Lemmon really 

been ACL president?
Can you spot the mistake?
ACL Presidents 2009–2017
Can you spot the mistake?
ACL Presidents 2009–2017
I didn’t spot it three years ago!
A biased world?
A biased world?
The Matthew/Matilda effect in science
Robert K. Merton, Science, 1968
Margaret W. Rossiter, Social Studies...
A biased world?
The Matthew/Matilda effect in science
Robert K. Merton, Science, 1968
Margaret W. Rossiter, Social Studies...
A biased world?
The Matthew/Matilda effect in science
Robert K. Merton, Science, 1968
Margaret W. Rossiter, Social Studies...
A biased world?
The Matthew/Matilda effect in science
Robert K. Merton, Science, 1968
Margaret W. Rossiter, Social Studies...
Why does it matter?
Why does it matter?
Equity
“The acceptance or rejection of claims entering the lists of science is not to depend on
the pe...
Why does it matter?
Equity
“The acceptance or rejection of claims entering the lists of science is not to depend on
the pe...
What is ACL doing?
What is ACL doing?
A new nominating committee
What is ACL doing?
Revised selection criteria for ACL fellows
A new nominating committee
What is ACL doing?
Revised selection criteria for ACL fellows
Promoting a large and diverse pool of nominations
A new nomi...
What is ACL doing?
Revised selection criteria for ACL fellows
Promoting a large and diverse pool of nominations
Resources ...
Talk about it!
The ACL Publishing Model
CL TACL ACL NAACL EMNLP
264
182
328
3926
• We like conferences
The ACL Publishing Model
• We like conferences
• We use anonymous peer review (almost everywhere)
The ACL Publishing Model
• We like conferences
• We use anonymous peer review (almost everywhere)
• And then there is this thing called arXiv …
The...
Peer review
• First used by PhilosophicalTransactions of the Royal Society, London, 1665
• Standard in scientific journals ...
Peer review
• First used by PhilosophicalTransactions of the Royal Society, London, 1665
• Standard in scientific journals ...
Odds multipliers from double-blind to single-blind review:
• Famous authors: 1.66
• Top universities: 1.61
• Top companies...
• 701 reviewers participated in a survey in June 2017
• 30% claimed they could identify the authors of a paper
• 17% contr...
0
300
600
900
1200
1500
2008 2009 2010 2011 2012 2013 2014 2015 2016 2017
ACL submissions
Are we overusing peer review?
• ...
Are we overusing peer review?
• Increased volumes lead to reviewer fatigue and lower quality
• Conferences lose their role...
Are we overusing peer review?
• Increased volumes lead to reviewer fatigue and lower quality
• Conferences lose their role...
FAST
Speeds up scientific advances?
Threatens scholarly thoroughness?
OPEN
Anyone can publish, anyone can read and discuss
Papers can be revised with version control
PEER REVIEW
Openness undermines double-blind review
Should preprints be cited in peer reviewed work?
ACL Survey on Preprint Publishing and Reviewing
• Run during three weeks in June 2017
• 623 complete responses
• Full repo...
22% upload to preprint servers always or often
• More likely user: graduate student, male
• Less likely user: academic res...
88% consider double-blind reviewing important
• 65% consider it more important than preprint publishing
• 9% consider prep...
87% would submit to ACL if preprints were banned
• 5% would probably stop submitting to ACL
ACL Survey on Preprint Publish...
ACL Survey on Preprint Publishing and Reviewing
0 75 150 225 300
No reviewing Open reviewing Single-blind reviewing
Status...
ACL Survey on Preprint Publishing and Reviewing
0 90 180 270 360
Lobby preprint servers to allow papers to be anonymously ...
Conclusions – Survey
Conclusions – Survey
• Strong support for double-blind reviewing in community
Conclusions – Survey
• Strong support for double-blind reviewing in community
• Weak support for completely banning prepri...
Conclusions – Survey
• Strong support for double-blind reviewing in community
• Weak support for completely banning prepri...
Conclusions – Survey
• Strong support for double-blind reviewing in community
• Weak support for completely banning prepri...
Conclusions – Survey
• Strong support for double-blind reviewing in community
• Weak support for completely banning prepri...
Conclusions – Survey
• Strong support for double-blind reviewing in community
• Weak support for completely banning prepri...
• More detailed analysis of the survey
• Invited quick-fire position statements
• Open discussion about publishing and revi...
Good Science
Good Science
92
93
94
95
96
2008 2009 2010 2011 2012 2013 2014 2015 2016 2017
UAS on WSJ
Good Science
92
93
94
95
96
2008 2009 2010 2011 2012 2013 2014 2015 2016 2017
UAS on WSJ
“Measurement as a virtue in itsel...
Good Science
“Lots of numbers with very small differences”
92
93
94
95
96
2008 2009 2010 2011 2012 2013 2014 2015 2016 201...
Good Science
“Lots of numbers with very small differences”
“What are the research questions?”
92
93
94
95
96
2008 2009 201...
Experimental Science
Experimental Science
• Experiments are run to test hypotheses
Experimental Science
• Experiments are run to test hypotheses
• Hypotheses are tentative theoretical explanations
morpholo...
Experimental Science
• Experiments are run to test hypotheses
• Hypotheses are tentative theoretical explanations
morpholo...
Reproducible Science
Replicability
• Repeating the same experiment with the same result
• Necessary to establish the reliability of measurement...
Replicability
• Repeating the same experiment with the same result
• Necessary to establish the reliability of measurement...
Replicability
• Repeating the same experiment with the same result
• Necessary to establish the reliability of measurement...
Replicability
• Repeating the same experiment with the same result
• Necessary to establish the reliability of measurement...
Everything is connected …
Everything is connected …
Publishing and reviewing
• Our traditional publishing model has a bias towards fast science
• Re...
Everything is connected …
Publishing and reviewing
• Our traditional publishing model has a bias towards fast science
• Re...
Keep up the good work!
Joakim Nivre - 2017 - Presidential Address ACL 2017: Challenges for ACL
Joakim Nivre - 2017 - Presidential Address ACL 2017: Challenges for ACL
Joakim Nivre - 2017 - Presidential Address ACL 2017: Challenges for ACL
Joakim Nivre - 2017 - Presidential Address ACL 2017: Challenges for ACL
Joakim Nivre - 2017 - Presidential Address ACL 2017: Challenges for ACL
Joakim Nivre - 2017 - Presidential Address ACL 2017: Challenges for ACL
Joakim Nivre - 2017 - Presidential Address ACL 2017: Challenges for ACL
Joakim Nivre - 2017 - Presidential Address ACL 2017: Challenges for ACL
Joakim Nivre - 2017 - Presidential Address ACL 2017: Challenges for ACL
Joakim Nivre - 2017 - Presidential Address ACL 2017: Challenges for ACL
Joakim Nivre - 2017 - Presidential Address ACL 2017: Challenges for ACL
Upcoming SlideShare
Loading in …5
×

Joakim Nivre - 2017 - Presidential Address ACL 2017: Challenges for ACL

3,239 views

Published on

Presidential Address to the ACL by Professor Joakim Nivre (Uppsala Universitet) for ACL 2017. 31 Jul 2017. Vancouver, Canada

License: CC BY 4.0

Published in: Education

Joakim Nivre - 2017 - Presidential Address ACL 2017: Challenges for ACL

  1. 1. Challenges for ACL ACL Presidential Address 2017 Joakim Nivre
  2. 2. (A)CL is booming!
  3. 3. 0 300 600 900 1200 1500 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 ACL submissions (A)CL is booming!
  4. 4. 0 300 600 900 1200 1500 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 ACL submissions 92 93 94 95 96 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 UAS on WSJ (A)CL is booming!
  5. 5. The future is bright!
  6. 6. Challenges for ACL Equity and Diversity Publishing and Reviewing Good Science
  7. 7. Can you spot the mistake? ACL Presidents 2009–2017
  8. 8. Only four of them are wearing funny hats? Can you spot the mistake? ACL Presidents 2009–2017
  9. 9. Can you spot the mistake? ACL Presidents 2009–2017 Has Jack Lemmon really 
 been ACL president?
  10. 10. Can you spot the mistake? ACL Presidents 2009–2017
  11. 11. Can you spot the mistake? ACL Presidents 2009–2017 I didn’t spot it three years ago!
  12. 12. A biased world?
  13. 13. A biased world? The Matthew/Matilda effect in science Robert K. Merton, Science, 1968 Margaret W. Rossiter, Social Studies of Science, 1993
  14. 14. A biased world? The Matthew/Matilda effect in science Robert K. Merton, Science, 1968 Margaret W. Rossiter, Social Studies of Science, 1993 Nepotism and sexism in peer-review Christine Wennerås and Agnes Wold, Nature, 1997
  15. 15. A biased world? The Matthew/Matilda effect in science Robert K. Merton, Science, 1968 Margaret W. Rossiter, Social Studies of Science, 1993 Nepotism and sexism in peer-review Christine Wennerås and Agnes Wold, Nature, 1997 Awards and prizes in the US, 1990s and 2000s Anne E. Lincoln, Stephanie Pincus, Janet Bandows Koster 
 and Phoebe S. Leboy, Social Studies of Science, 2012
  16. 16. A biased world? The Matthew/Matilda effect in science Robert K. Merton, Science, 1968 Margaret W. Rossiter, Social Studies of Science, 1993 Nepotism and sexism in peer-review Christine Wennerås and Agnes Wold, Nature, 1997 Awards and prizes in the US, 1990s and 2000s Anne E. Lincoln, Stephanie Pincus, Janet Bandows Koster 
 and Phoebe S. Leboy, Social Studies of Science, 2012 Nationality, ethnicity, religion, gender, sexual orientation, …
  17. 17. Why does it matter?
  18. 18. Why does it matter? Equity “The acceptance or rejection of claims entering the lists of science is not to depend on the personal or social attributes of their protagonist; […] race, nationality, religion, class, and personal qualities are as such irrelevant.” Robert K. Merton. Science and Technology in a Democratic Order. 
 Journal of Legal and Political Sociology, 1: 115–126, 1942.
  19. 19. Why does it matter? Equity “The acceptance or rejection of claims entering the lists of science is not to depend on the personal or social attributes of their protagonist; […] race, nationality, religion, class, and personal qualities are as such irrelevant.” Diversity “Decades of research by organizational scientists, psychologists, sociologists, economists and demographers show that socially diverse groups (that is, those with a diversity of race, ethnicity, gender and sexual orientation) are more innovative than homogeneous groups.” Robert K. Merton. Science and Technology in a Democratic Order. 
 Journal of Legal and Political Sociology, 1: 115–126, 1942. Katherine W. Phillips. How Diversity Makes Us Smarter. 
 Scientific American, October 1, 2014.
  20. 20. What is ACL doing?
  21. 21. What is ACL doing? A new nominating committee
  22. 22. What is ACL doing? Revised selection criteria for ACL fellows A new nominating committee
  23. 23. What is ACL doing? Revised selection criteria for ACL fellows Promoting a large and diverse pool of nominations A new nominating committee
  24. 24. What is ACL doing? Revised selection criteria for ACL fellows Promoting a large and diverse pool of nominations Resources for promoting diversity and preventing bias A new nominating committee
  25. 25. Talk about it!
  26. 26. The ACL Publishing Model
  27. 27. CL TACL ACL NAACL EMNLP 264 182 328 3926 • We like conferences The ACL Publishing Model
  28. 28. • We like conferences • We use anonymous peer review (almost everywhere) The ACL Publishing Model
  29. 29. • We like conferences • We use anonymous peer review (almost everywhere) • And then there is this thing called arXiv … The ACL Publishing Model
  30. 30. Peer review • First used by PhilosophicalTransactions of the Royal Society, London, 1665 • Standard in scientific journals from the mid-20th century • Meant to guarantee scientific quality – “organized scepticism”
  31. 31. Peer review • First used by PhilosophicalTransactions of the Royal Society, London, 1665 • Standard in scientific journals from the mid-20th century • Meant to guarantee scientific quality – “organized scepticism” Double-blind peer review • Popularized by sociology journals in the 1950s • Less common in natural sciences and engineering • Meant to reduce author bias – the Matthew/Matilda effect
  32. 32. Odds multipliers from double-blind to single-blind review: • Famous authors: 1.66 • Top universities: 1.61 • Top companies: 2.10
  33. 33. • 701 reviewers participated in a survey in June 2017 • 30% claimed they could identify the authors of a paper • 17% contributed 196 unique guesses Exact Partial Institution Incorrect Aurélie Névéol Karën Fort EMNLP 2017 Reviewer Survey on Double-Blind Reviewing Report available at: 
 https://www.aclweb.org/portal/ 41%16%38%5% Rebecca Hwa
  34. 34. 0 300 600 900 1200 1500 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 ACL submissions Are we overusing peer review? • Increased volumes lead to reviewer fatigue and lower quality • Conferences lose their role as a discussion forum for new ideas • Scientific gatekeeping should be left to journals?
  35. 35. Are we overusing peer review? • Increased volumes lead to reviewer fatigue and lower quality • Conferences lose their role as a discussion forum for new ideas • Scientific gatekeeping should be left to journals? Ideas to consider • Prescreening to reduce reviewer load (and increase quality) • Journal-style reviewing with rolling deadline for long papers • Abstract submission for short papers/posters
  36. 36. Are we overusing peer review? • Increased volumes lead to reviewer fatigue and lower quality • Conferences lose their role as a discussion forum for new ideas • Scientific gatekeeping should be left to journals? Ideas to consider • Prescreening to reduce reviewer load (and increase quality) • Journal-style reviewing with rolling deadline for long papers • Abstract submission for short papers/posters How to combine our model with preprints on arXiv?
  37. 37. FAST Speeds up scientific advances? Threatens scholarly thoroughness?
  38. 38. OPEN Anyone can publish, anyone can read and discuss Papers can be revised with version control
  39. 39. PEER REVIEW Openness undermines double-blind review Should preprints be cited in peer reviewed work?
  40. 40. ACL Survey on Preprint Publishing and Reviewing • Run during three weeks in June 2017 • 623 complete responses • Full report available at: https://www.aclweb.org/portal/ Jennifer Foster Marti Hearst Shiqi Zhao
  41. 41. 22% upload to preprint servers always or often • More likely user: graduate student, male • Less likely user: academic researcher, female 27% cite preprints often or very often • Frequent users are more likely to cite often ACL Survey on Preprint Publishing and Reviewing Do you upload preprints? Do you cite preprints? Never Seldom Sometimes Often Always 8%14%25%19%34% 11%16%27%30%15%
  42. 42. 88% consider double-blind reviewing important • 65% consider it more important than preprint publishing • 9% consider preprint publishing more important ACL Survey on Preprint Publishing and Reviewing Which is more important? Double-blind Both Preprint Neither 9%65% 23%
  43. 43. 87% would submit to ACL if preprints were banned • 5% would probably stop submitting to ACL ACL Survey on Preprint Publishing and Reviewing Would you still submit? Definitely yes Probably yes Maybe Probably not Definitely not 70% 17% 8% 4%
  44. 44. ACL Survey on Preprint Publishing and Reviewing 0 75 150 225 300 No reviewing Open reviewing Single-blind reviewing Status quo Discourage preprints Ban preprints How would you like to see ACL’s reviewing model working in the future?
  45. 45. ACL Survey on Preprint Publishing and Reviewing 0 90 180 270 360 Lobby preprint servers to allow papers to be anonymously uploaded Make available author guidelines for citing preprint papers Have a separate track at ACL for preprint papers Journal-style reviewing with a rolling deadline (with CL and/or TACL) The following are some actions suggested by members of the community. 
 Please indicate which, if any, you would like to see implemented.
  46. 46. Conclusions – Survey
  47. 47. Conclusions – Survey • Strong support for double-blind reviewing in community
  48. 48. Conclusions – Survey • Strong support for double-blind reviewing in community • Weak support for completely banning preprints
  49. 49. Conclusions – Survey • Strong support for double-blind reviewing in community • Weak support for completely banning preprints • Temporarily anonymous preprints is worth looking into
  50. 50. Conclusions – Survey • Strong support for double-blind reviewing in community • Weak support for completely banning preprints • Temporarily anonymous preprints is worth looking into • Guidelines for citing preprints are needed
  51. 51. Conclusions – Survey • Strong support for double-blind reviewing in community • Weak support for completely banning preprints • Temporarily anonymous preprints is worth looking into • Guidelines for citing preprints are needed • Many are concerned about reviewing quality
  52. 52. Conclusions – Survey • Strong support for double-blind reviewing in community • Weak support for completely banning preprints • Temporarily anonymous preprints is worth looking into • Guidelines for citing preprints are needed • Many are concerned about reviewing quality • We have to work both long-term and short-term
  53. 53. • More detailed analysis of the survey • Invited quick-fire position statements • Open discussion about publishing and reviewing ACL Business Meeting Wednesday, August 2, 13:00-14:30
  54. 54. Good Science
  55. 55. Good Science 92 93 94 95 96 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 UAS on WSJ
  56. 56. Good Science 92 93 94 95 96 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 UAS on WSJ “Measurement as a virtue in itself”
  57. 57. Good Science “Lots of numbers with very small differences” 92 93 94 95 96 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 UAS on WSJ “Measurement as a virtue in itself”
  58. 58. Good Science “Lots of numbers with very small differences” “What are the research questions?” 92 93 94 95 96 2008 2009 2010 2011 2012 2013 2014 2015 2016 2017 UAS on WSJ “Measurement as a virtue in itself”
  59. 59. Experimental Science
  60. 60. Experimental Science • Experiments are run to test hypotheses
  61. 61. Experimental Science • Experiments are run to test hypotheses • Hypotheses are tentative theoretical explanations morphological segmentation facilitates syntactic parsing system A outperforms system B on data set C
  62. 62. Experimental Science • Experiments are run to test hypotheses • Hypotheses are tentative theoretical explanations morphological segmentation facilitates syntactic parsing system A outperforms system B on data set C • Validating hypotheses requires repeated testing
  63. 63. Reproducible Science
  64. 64. Replicability • Repeating the same experiment with the same result • Necessary to establish the reliability of measurements • Enables benchmarking and “fair” comparisons – fast science Reproducible Science
  65. 65. Replicability • Repeating the same experiment with the same result • Necessary to establish the reliability of measurements • Enables benchmarking and “fair” comparisons – fast science Reproducibility • Repeating a similar experiment with a similar result • Necessary to establish the validity of hypotheses • Requires repeated testing and comparative analysis – slow science Reproducible Science
  66. 66. Replicability • Repeating the same experiment with the same result • Necessary to establish the reliability of measurements • Enables benchmarking and “fair” comparisons – fast science Reproducibility • Repeating a similar experiment with a similar result • Necessary to establish the validity of hypotheses • Requires repeated testing and comparative analysis – slow science Reproducible Science We need both!
  67. 67. Replicability • Repeating the same experiment with the same result • Necessary to establish the reliability of measurements • Enables benchmarking and “fair” comparisons – fast science Reproducibility • Repeating a similar experiment with a similar result • Necessary to establish the validity of hypotheses • Requires repeated testing and comparative analysis – slow science Reproducible Science We need both! We need diversity!
  68. 68. Everything is connected …
  69. 69. Everything is connected … Publishing and reviewing • Our traditional publishing model has a bias towards fast science • Reinforced by increasing reviewer loads • Accelerated by preprint publishing
  70. 70. Everything is connected … Publishing and reviewing • Our traditional publishing model has a bias towards fast science • Reinforced by increasing reviewer loads • Accelerated by preprint publishing Equity and diversity • We need diversity and innovation in research and publishing • A more inclusive and diverse community is more likely to give us that
  71. 71. Keep up the good work!

×