Critiquing Research Ideas

1 Purpose of this Assignment

As you probably know by now, research continually undergoes what is known as peer review, whereby other people in your discipline, and in the broader field of computer science, are called upon to evaluate your work.

Although the process of peer review is not perfect (in fact, as you can read in one of the references below, it is often very imperfect!), it is one of the most important (and most common) ways you will be evaluated as a researcher. As the term “peer” implies, you will be called upon to review the work of other researchers. In this assignment, you will write a review of each other’s research project proposals from Assignment 2. The purpose of this assignment is to give you practice writing reviews of each other’s work.

In practice, you will often be given papers, proposals and projects to review whose subject matter falls outside your immediate area of expertise. When this happens, you are still expected to give as thorough a review as possible but indicate that you are not an expert in the area (many review forms ask for a self-assessment of expertise). This assignment will require you to read some project proposals outside of your immediate area, as well. Do your best! (A good reviewer will often go beyond reading the actual paper or proposal itself and familiarize him or herself with the related subject material.)

2 Problem

You may find some of the following resources helpful:

• We are Sorry to Inform You...{target=”_blank” rel=”noopener”}
• An Evaluation of the Ninth SOSP Submissions{target=”_blank” rel=”noopener”}

3 Task

The TAs will post the assignments of papers to groups shortly; you will meet in a group of three students that is different from your original group.

Write project reviews. For each set of projects that you are asked to review, you should (1) rate your level of expertise in the general subject area on a score of 1-5; and (2) rate each project from 1-5 according to the following categories:

• Novelty: How unique or new is the idea? Has the work been placed in the context of the related previous work?

• Potential impact/Technical merit: How likely is the idea to advance its particular field signif- icantly? How well thought out are the ideas? Is there an evaluation plan?

• Clarity/Presentation: Are the ideas clearly presented? Are there spelling or grammatical errors?

In addition to providing scores for each of these, you are welcome to provide specific comments on each of those three categories. You should also provide detailed comments on the writeup, suggesting changes to the presentation, approach/technique, etc.

Select the best paper from your batch. Think of your small group as a kind of “mini program committee”; the three of you should come to a consensus about the best project proposal among those that were given to you. You should nominate this project for the best project award and provide a short defense of your selection (imagine yourself arguing at a meeting trying to defend your selected project!).