Recognizing Good Ideas

1 Purpose of this Assignment

This assignment is the first in a series of assignments where you will take the steps needed to do great research. In this assignment, you will perform one of the most common tasks that we as researchers perform to find a research problem: you will read the most recent proceedings from the top conference in your area, select two papers that you think represent “good research”, and present one idea (for each paper) on how to extend that research (sometimes hints are presented in the papers themselves!).

2 Problem

One of the first tasks that a researcher must do—not to mention one of the most challenging ones—is finding the right problem or problems to work on. What makes a great research problem?

First, it’s important to realize that there’s not necessarily a “right” answer to this question, and that research is often a matter of taste. Second, it’s often difficult to tell whether a research problem is truly great at first cut (see today’s reading for examples where reviewers “got it wrong” the first time around).

Nevertheless, while there are no hard and fast criteria, great research problems tend to share many of the following characteristics. Here are some of our thoughts on what makes good research; when perusing conference proceedings, it may help you to keep some of the following criteria in mind when selecting papers:

• The problem is difficult. Life is too short to solve easy problems.

• Solving the problem creates new knowledge. Recognizing the difference between research and “a simple matter of engineering” is important. Many problems are difficult, but if an army of programmers could solve the problem with what we know today, it’s probably not research.

• The problem changes conventional thinking. While there is value in confirming conventional wisdom, great research typically offers surprising results, creates a new way of thinking about or approaching problems.

• The problem has a solution. Not every paper needs to solve a problem completely (in fact, you’ll go looking for future work as part of this assignment). Still, while posing a good question is paramount, the solution or finding presented in a paper should represent a significant advance beyond the previous body of knowledge. As a Ph.D. student, since you ultimately need to graduate, it’s good to pick problems that have some solution!

Needless to say, it’s difficult when first starting out on a problem do determine whether it has these characteristics, but it is often patently clear when a problem does not have them.

3 Task

Read conference proceedings. Each research group should select two papers from their respec- tive conference proceedings. Please use the following conference proceedings as references:

• Architecture: International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS)
• Artificial Intelligence: International Joint Conference on Artificial Intelligence
• Cognitive Science: Cognitive Science Society Annual Meeting (CogSci)
• Computational Science and Engineering: SIAM Annual Meeting
• Cryptography: CRYPTO
• Databases: ACM SIGMOD/PODS
• Graphics: ACM SIGGRAPH
• HCI: CHI
• Learning Science and Technology: International Conference of the Learning Sciences,
• Machine Learning: Neural Information Processing Systems (NIPS)
• Networking: Proceedings of ACM SIGCOMM
• Programming Languages: ACM Programming Language Design and Implementation
• Robotics: Robotics Science and Systems
• Security: IEEE Symposium on Security and Privacy
• Software Engineering: ACM SIGSOFT Symposium on the Foundations of Software Engineering (FSE)
• Systems: ACM Symposium on Operating Systems Principles
• Theory: ACM Symposium on Theory of Computing
• Vision: Computer Vision and Pattern Recognition (CVPR)

Select and defend papers. After selecting your two papers, write a one-page summary of each paper that provides (1) A short summary of the paper; (2) Why you think it satisfies the criteria for good research; (3) A short problem statement for a possible follow-on research problem.  Use the criteria of novelty, impact, and clarity to defend your selection, as well as the questions from the post on cultivating research taste