This document summarizes the reviewing process for ICSE 2016. It outlines the roles and responsibilities of the Program Board and Program Committee. The Program Committee conducts the main paper reviews while the Program Board oversees the process and discussion. All papers will receive at least 3 reviews and an opportunity for author rebuttal. The Program Board will drive online discussions, conduct additional reviews, and make final acceptance decisions at an in-person meeting in December.

1. ICSE 2016 Reviewing Process
Laurie and Willem

2. Welcome
• Thanks for serving!
• 28 Program Board members
– In charge of +-22 papers
– Full review for +-6 of those
• 89 Program Committee members
– +-16 papers to review

3. General Philosophy
• Program Committee does the main reviewing
• Program Board keeps process on track and leads
the discussions
• PB then communicates PC decisions at the face-
to-face meeting
– Extra PB member does a checks&balances review
• Main purpose of the meeting is to determine
where the bar is for acceptance
– For some problem cases that had no consensus during
the PC reviewing period, decisions will be made
• Chairs sit back and relax!

4. Author view
• All papers get at least 3 reviews
– Some will get 4 (extra from PB member)
• All papers will get a chance at a rebuttal
– Right after PC reviewing, i.e. will come about 3
days after the start of online discussion
• All papers will get a summary of the decision
• Author feedback will be solicited

5. Desk Rejects
28/8 1/9
Bidding
6/9
Assignments
Rebuttals
23/10 (3 days)
Online Discussion
23/10
PC Reviewing
11/9
PB Overseeing
20/11
2nd Reader
Assignment
24/11-6/12
PB Review
9-10/12
PB Meeting
15/12
Notifications
Overall Process

6. Desk Rejects
28/8 1/9
Bidding
6/9
Assignments
Rebuttals
23/10 (3 days)
Online Discussion
23/10
PC Reviewing
11/9
PB Overseeing
20/11
2nd Reader
Assignment
24/11-6/12
PB Review
9-10/12
PB Meeting
15/12
Notifications
Program Board Involvement

7. Desk Rejects
28/8 1/9
Bidding
6/9
Assignments
Rebuttals
23/10 (3 days)
Online Discussion
23/10
PC Reviewing
11/9
PB Overseeing
20/11
2nd Reader
Assignment
24/11-6/12
PB Review
9-10/12
PB Meeting
15/12
Notifications
Program Committee Involvement
Online
Discussion

8. Program Board Actions Overview
• Bidding
• Monitor Reviewing [Chairs will as well]
– Make sure reviews are on time
• Drive online discussion
• PB Reviews
– Full review, i.e. filling in the reviewing form
• PB Meeting

9. Program Board Actions
Online Discussion
• Drive online discussion
– Summarize views (including rebuttal) and kick off the conversation
– Eliminate clear rejects first
• No support is a reject
– Try to find consensus
• No consensus is fine at this point
– Consensus is based on all 3 reviewers and the PB member’s opinion
• i.e. PB member is allowed an opinion, but it doesn’t carry more weight than
the PC
• Write a discussion summary at the end of the discussion period
– Select one of two outcomes: reject or move on
• Either consensus accept or no consensus moves on
– Rejection summaries will be included in author feedback
• Summaries of papers moving on will be superseded by edited summaries later
in the process

10. Program Board Actions
Phase 2
• PB Reviews
– Full review, i.e. filling in the reviewing form
• PB review is visible to PC and might lead to more
online discussion
– If there is disagreement between the consensus or if
there was no consensus before another brief online
discussion is recommended
• PB Meeting
– PB member summarizes the paper and the
reviews/discussion, 2nd Reader comments briefly and
the complete PB discusses whether the paper is above
the bar for the meeting
– Edit the final summary that will be included in the
feedback to the authors

11. Program Committee Actions
• Bidding
– This is important and without it you might get things you
don’t like or can’t even review properly
• Reviewing
– Intermediate deadline when 1/2 the papers will be done
• This feature saves everyone pain
• Online Discussions
– Mandatory Participation
– Check rebuttals and amend reviews where required
– Check that PB member summaries are accurate
– Be available for last minute discussions early December

12. Paper Flow
Desk
Reject
Online
Discussion
Online
Discussion
PC Reviewing
PB Reviewing
PB Meeting
500
30
470
16 per PC (PB)
300
170
6 per PB
2030
120
60??60??

13. PB Meeting
• Raleigh, North Carolina
• 9 and 10 December 2015
– 2 full days
• Sheraton Downtown Raleigh
• Raleigh-Durham Airport (RDU)
– Direct flights from all major US hubs
– London