The paper review process, and how to critique a paper

Objectives:

• To humanize all players in the paper review process, from the paper authors, to the reviews, to the area chairs, etc.
• To understand how to review a paper constructively: what can the authors do to realistically improve the score you give them as a reviewer?
• To understand the benefits and shortcomings of conference and journal reviewing.

Pre-class work

• Skim:
  • The ins and outs of reviewing
  • How NOT to review a paper The tools and techniques of the adversarial reviewer
  • Mistakes Reviewers Make
  • Dear Area Chairs (ACs)
• Answer:
  • From these readings, in one sentence what makes a “good” review vs. a “bad” review?
  • Select one of the following three papers and review them, as per the instructions below:
    • Paper 1: Tailoring Language Generation Models under Total Variation Distance (link)
    • Paper 2: CodeBPE: Investigating Subtokenization Options for Large Language Model Pretraining on Source Code (link)
    • Paper 3: Knowledge-Consistent Dialogue Generation with Language Models and Knowledge Graphs (link)
  • Complete the simplified review form (below) for the paper you chose

Tip: We selected three papers that have recently been accepted to decrease the chance students have already seen them. We specifically selected papers on OpenReview with different scores – high, medium, and low – and linked to the anonymized versions of them in the pre-class work. Lastly, we ensured students spread evenly across each paper.

In class [slides]

1. [10min] Students socialize
2. [30min] Overview of the review process and what makes a “good” (constructive and respectful) paper review.
3. [20min] Break students into groups to fill out the meta-review form (below) for their assigned paper, given reviews from OpenReview. We assign each group a paper that the students did not read to simulate the experience of being an AC / meta-reviewer.
4. [20min] As a whole class: ask each group whether they accepted / rejected the paper and why. We highlight places of disagreement amongst groups that evaluated the same papers and use that to shape the conversation.
5. [20min] As a group, we discuss challenges of reviewers, meta-reviewers, area chairs, etc.
6. [5min] Take in-class survey
   • What did you take away from today’s class?

Simplified Review Form

1. Summary Of The Paper
2. Strength and Weaknesses
3. Clarity, Quality, Novelty and Reproducibility
4. Recommendation: (choose one)
   • 10: Top 5% of accepted papers, seminal paper
   • 9: Top 15% of accepted papers, strong accept
   • 8: Top 50% of accepted papers, clear accept
   • 7: Good paper, accept
   • 6: Marginally above acceptance threshold
   • 5: Marginally below acceptance threshold
   • 4: Ok but not good enough - rejection
   • 3: Clear rejection
   • 2: Strong rejection *1: Trivial or wrong
5. Confidence: (choose one)
   • 5: You are absolutely certain about your assessment. You are very familiar with the related work and checked the math/other details carefully.
   • 4: You are confident in your assessment, but not absolutely certain. It is unlikely, but not impossible, that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work.
   • 3: You are fairly confident in your assessment. It is possible that you did not understand some parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
   • 2: You are willing to defend your assessment, but it is quite likely that you did not understand central parts of the submission or that you are unfamiliar with some pieces of related work. Math/other details were not carefully checked.
   • 1: Your assessment is an educated guess. The submission is not in your area or the submission was difficult to understand. Math/other details were not carefully checked.

Meta-Review Form

1. Summary, Strengths And Weaknesses: A concise description of the submission’s main content (scientific claims and findings) based on your own reading and reviewers’ characterization, including the paper’s strengths and weaknesses. Ideally, this description should contain both what is discussed in the submission and what is missing from the submission. Also summarize the reviews and what stood out for you as an AC.
2. Justification for Why Not Higher Score: Why did you choose to not give a higher score? Which key factors influenced your decision? Give the authors actionable and useful feedback.
3. Justification for Why Not Lower Score: Why did you choose not to give a lower score? Which key factors influenced your decision? Give the authors actionable and useful feedback.