0
Data Science of the
KDD ‘14 Review Process
Jure Leskovec (Stanford) and
Wei Wang (UCLA)
Joint work with
Jason Hirshman and...
KDD 2014 Research Track
Statistics
KDD 2014 Program
Largest KDD program ever:
• 151 research papers (20% growth over KDD’13)
• 43 industry & govt. papers (30...
KDD 2014 Research Track
• 1036 submissions from 2600 authors
– 42% increase over KDD ’13
• 151 papers:
– Acceptance rate
1...
KDD Reviewing Process
46 Senior PC members + 340 PC members
• 2971 reviews in total
(Rough) Acceptance rule:
• Raw review ...
Submissions per Country
Acceptance Rate per Country
Acceptance by Subject Area
Predicting Paper Acceptance
Features Used Accuracy
Random Guessing 0.50
Paper Abstract 0.57
Author Status (Past paper coun...
Predicting Paper Acceptance
from the Review Text
Features Used
Paper:
Accepted?
Review:
Score > 0?
Random Guessing 0.50 0....
I’m submitting a paper:
What correlates with acceptance?
Academia + Industry Papers do Better
Submissions per Author: 5 is best!
No benefit in submitting >5 papers!
Having more authors (seems to) help
It is the most experienced author
that matters!
What insights can we gain on
the review process?
Most reviews are Weak Rejects
More granularity is needed at the
Weak Reject / Weak Accept level
Reviewagreeswiththefinaloutcome
Review length is a good determinant
of a review’s influence/quality
Reviewagreeswiththefinaloutcome
Shorter reviews are used for
clear accepts and rejects
Never review co-author’s papers
The Curse of the Review
Submission Deadline
Over 50% reviews submitted in the last 5 days
Over 20% reviews submitted in the last 24 hours
10% of reviews
submitted late
Ratings increase near the deadline
Weak Rejects
increase while
Rejects decrease
Reviews submitted late are less likely
to agree with final outcome
Late reviews are shorter
Review quality drops: Accuracy of
predicting score from review text
Conclusions
• To get your papers accepted to KDD:
– Collaborate in multidisciplinary teams
– Have a senior author on board...
Upcoming SlideShare
Loading in...5
×

Data Science view of the KDD 2014

10,177

Published on

KDD is a premier conference that brings together researchers and practitioners from data science, data mining, knowledge discovery, large-scale data analytics, and big data.

These slides give some statistics about the KDD program and present data science view of the paper review process: 1100 submissions, 3000 reviews, and 150 accepted papers.

Published in: Data & Analytics
0 Comments
20 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
10,177
On Slideshare
0
From Embeds
0
Number of Embeds
6
Actions
Shares
0
Downloads
126
Comments
0
Likes
20
Embeds 0
No embeds

No notes for slide
  • Country of the paper is given by the mode author nationality.
    Only countries with more than 10 submissions are shown (except South Korea, which had 0 acceptance)
  • Subject areas were based on a field that authors tagged their papers with.
    Only subject areas with more than 50 submissions are shown.
  • On balanced dataset (we subsampled negative examples)
  • On balanced dataset (we subsampled negative examples)
  • Academic: All authors of paper are affiliated with a university
    Industry: All authors of paper are affiliated with industry
    Mixed: Paper has authors affiliated with both universities and industry.
  • Transcript of "Data Science view of the KDD 2014"

    1. 1. Data Science of the KDD ‘14 Review Process Jure Leskovec (Stanford) and Wei Wang (UCLA) Joint work with Jason Hirshman and David Zeng (Stanford)
    2. 2. KDD 2014 Research Track Statistics
    3. 3. KDD 2014 Program Largest KDD program ever: • 151 research papers (20% growth over KDD’13) • 43 industry & govt. papers (30% growth) • 26 workshops (75% growth) • 11 tutorials (83% growth) Program highlights: • Paper spotlights early morning (8:15am) • Oral presentations (Mon-Wed) • Posters at the reception (Tue night)
    4. 4. KDD 2014 Research Track • 1036 submissions from 2600 authors – 42% increase over KDD ’13 • 151 papers: – Acceptance rate 14.6% 0 200 400 600 800 1000 1200 2000 2005 2010 2015 KDD year Numberofsubmissions
    5. 5. KDD Reviewing Process 46 Senior PC members + 340 PC members • 2971 reviews in total (Rough) Acceptance rule: • Raw review score AND Standardized review score AND Raw meta-review AND Standardized meta-review score ≥ Weak Accept • 110 papers matched (immediate accepts) • Remaining papers were discussed with meta-reviewers and final decisions were made
    6. 6. Submissions per Country
    7. 7. Acceptance Rate per Country
    8. 8. Acceptance by Subject Area
    9. 9. Predicting Paper Acceptance Features Used Accuracy Random Guessing 0.50 Paper Abstract 0.57 Author Status (Past paper counts) 0.64 Author Status (DBLP graph connectivity) 0.61 Author Status (Counts + Graph) 0.65 Reviewer (Similarity, Graph distance to authors) 0.60 All (Abstract, Author Status, and Reviewer) 0.65
    10. 10. Predicting Paper Acceptance from the Review Text Features Used Paper: Accepted? Review: Score > 0? Random Guessing 0.50 0.50 Review Text 0.68 0.72 Review Text + Numeric Score (Novelty, Presentation) 0.77 0.77 Human Reading of Review Text 0.88 0.73
    11. 11. I’m submitting a paper: What correlates with acceptance?
    12. 12. Academia + Industry Papers do Better
    13. 13. Submissions per Author: 5 is best!
    14. 14. No benefit in submitting >5 papers!
    15. 15. Having more authors (seems to) help
    16. 16. It is the most experienced author that matters!
    17. 17. What insights can we gain on the review process?
    18. 18. Most reviews are Weak Rejects
    19. 19. More granularity is needed at the Weak Reject / Weak Accept level Reviewagreeswiththefinaloutcome
    20. 20. Review length is a good determinant of a review’s influence/quality Reviewagreeswiththefinaloutcome
    21. 21. Shorter reviews are used for clear accepts and rejects
    22. 22. Never review co-author’s papers
    23. 23. The Curse of the Review Submission Deadline
    24. 24. Over 50% reviews submitted in the last 5 days Over 20% reviews submitted in the last 24 hours 10% of reviews submitted late
    25. 25. Ratings increase near the deadline Weak Rejects increase while Rejects decrease
    26. 26. Reviews submitted late are less likely to agree with final outcome
    27. 27. Late reviews are shorter
    28. 28. Review quality drops: Accuracy of predicting score from review text
    29. 29. Conclusions • To get your papers accepted to KDD: – Collaborate in multidisciplinary teams – Have a senior author on board – Do not submit more than 5 papers • To improve KDD community standards: – Avoid Weak Reject/Weak Accept scores – Write longer and clearer reviews – Submit reviews early!
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×