How to read a research paper?

References

• How to Read a Paper; S. Keshav; ACM SIGCOMM Computer Communication Review; 2007.
• How to Read an Engineering Research Paper; William G. Griswold.

We recommend the students use the three-pass approach by S. Keshav and read the paper while thinking about the questions listed in an article by William G. Griswold. We highly encourage you to read these two articles as they contain more content such as note-taking tips. We provide an adapted and succinct version of how to read a hardware security paper for the SHD course below.

The Three-Pass Approach

The first pass:

• Read the title, abstract, introduction, section and subsection titles, related work, and conclusion.
• After this pass, you should be able to answer the following questions:
  • Category: What type of paper is this? Is it an attack paper, a defense paper, or an analysis paper?
  • Context (relevance to related work): Which other papers is it related to? If it is an attack paper, what are the other existing attacks that target the same threat model? If it is a defense paper, what are the other existing defenses that share a similar security goal as this paper?
  • Contributions: What are claimed as the paper’s main contributions?

The second pass:

• Read the paper with greater care, but ignore details such as proofs. Look carefully at the figures, diagrams and other illustrations in the paper. The goal is to grasp the key content of the paper.
• As you read through the paper, attempt to answer the following questions (from Griswold’s article and adapted for hardware security papers):
  1. What are the motivations for this work? For a security research paper, there is an expectation that the paper either discovers a new security threat or solves a security problem that no one else has published in the literature.
     • To derive the motivation of the paper, you need to consider why the security problem has not been discovered or why it does not have a trivial solution.
     • There is also an implication that previous attacks or solutions to the problem are inadequate. What are the previous attacks/solutions and why are they inadequate?
     • Finally, the motivation and statement of the problem are distilled into a research question, the question that the paper sets out to answer.
  2. What is the proposed attack or defense?
     • This is also called the idea of the paper. This is the proposed answer to the research question.
     • There should also be an answer to the question of why it is believed that this attack/defense will work, and be better than previous work.
     • There should also be a discussion about how the attack/defense is achieved (designed and implemented) or is at least achievable.
  3. What is the work’s evaluation of the proposed solution?
     • An idea alone is usually not adequate for the publication of a research paper. The evaluation is the concrete engagement of the research question.
     • What argument, implementation, and/or experiment makes the case for the value of the ideas? What benefits or problems are identified?
  4. What is your analysis of the identified problem, idea and evaluation?
     • Is this a good idea? What flaws do you perceive in the work? What are the most interesting points made? What are the most controversial ideas or points made?
     • For work that has practical implications, you also want to ask: Is this attack/defense really going to work, who can pull off the attack, and who would want the defense solution?
  5. What are the contributions?
     • The contributions in a paper may be many and varied. Beyond the insights on the research question, a few additional possibilities include: ideas, software, experimental techniques, or an area survey.
  6. What are future directions for this research?
     • Not only what future directions do the authors identify, but what ideas did you come up with while reading the paper? Sometimes these may be identified as shortcomings or other critiques in the current work.
  7. What questions are you left with?
     • What questions would you like to raise in an open discussion of the work? What do you find confusing or difficult to understand? By taking the time to list several, you will be forced to think more deeply about the work.
  8. What is your take-away message from this paper?
     • Sum up the main implication of the paper from your perspective. This is useful for very quick review to refresh your memory. It also forces you to try to identify the essence of the work.

The third pass (optional for this course):

• The key to the third pass is to attempt to virtually re-implement the paper: that is, making the same assumptions as the authors, re-create the work. By comparing this re-creation with the actual paper, you can easily identify not only a paper’s innovations, but also its hidden failings and assumptions. This is usually needed when you are reviewing the paper as a program committee member.
• You should identify and challenge every assumption in every statement. During this pass, you should also jot down ideas for future work.

How will we run the discussion session?

During each paper discussion session, we will keep track of the time for each paper. If you run out of time, you will be interrupted and end up not finishing your presentation. So make sure to practice your presentation ahead of time.

• Presentation: 12 min
  • Paper summary (motivation, proposed idea, evaluation): ~8 min
  • Paper critiques (strengths, weaknesses, your thoughts, future work): ~4 min
• Class Q&A: 2 min
• Class vote: 1 min

Timing is very tight – please come to class on time!

How will your presentation be graded?

Your presentation will be graded based on the clarity of the articulation of the paper content and your critiques of the paper. Your presentation will be rigorously graded by four persons: the instructor and the three TAs. The detailed rubric is shown below.

For each item in the rubric, every grader will give a score between 0-5. For example, when grading for “Clearly articulate motivation”, a score of 0 will be given if the presenter does not mention the motivation of the paper, a score of 5 will be given if the presenter precisely and concisely describes the motivation to the point.

We will calculate the average score across the four graders for each item and then compute the weighted sum score as the final score. If a paper discussion is led by two presenters, the two presenters will get the same final score.