How to read a research paper?
We believe learning how to read a research paper is an important component of this class. There exists many materials describing how to read a paper. The following two recommended readings discussed a “three-pass approach”, and talked about what questions you should think about while reading a research paper. If you have never read a research paper before, we suggest you go through these two articles first to get a rough idea of how to approach a paper. They also provide useful note-taking tips.
References
We provide an adapted and succinct version of how to read a hardware security paper for the SHD course below.
The first pass:
For your first pass, read the following items and skip the others:
- Title and abstract
- The introduction section
- Section and subsection titles (just the titles, not the content)
- Related work section and the conclusion.
After this pass, you should be able to answer the following questions:
- Category: What type of paper is this? Is it an attack paper, a defense paper, or an analysis paper?
- Context (background and related work):
- We might have discussed some background materials in class. Try to think about whether there exists any background gap for the other students in the class to understand this paper: what background materials we have NOT covered which you have to include in your presentation.
- Get an idea about existing work that this paper relates to. This step is to get your prepared for critical thinking of this paper. If it is an attack paper, what are the other existing attacks that target the same threat model? If it is a defense paper, what are the other existing defenses that share a similar security goal as this paper?
- Contributions: What are claimed as the paper’s main contributions?
The second pass:
Read the paper with greater care, but ignore details such as proofs. The goal is to grasp the main content of the paper. Here are a few tips on how to zoom in.
- Look carefully at the figures, diagrams, and other illustrations in the paper. If you can thoroughly understand the important figures in the main section, you are on the right track. If you find things are fuzzy, you will then need to read the text that explain these figures/diagrams multiple times to decipher the content.
- A useful tip is to look for videos related to this paper online (sometimes conferences post lightning talks or full talks of the accepted papers on Youtube or conference websites). Observe how the authors explain their work. This will help you understand the key contributions better. Moreover, seeing how the idea being presented in a different format other than pure text will help you structure and design your presentation slides. Just be aware that, conference presentations usually focus on highlighting the contributions, and rarely talk about limitations of the work.
As you read through the paper, attempt to answer the following questions (from Griswold’s article and adapted for hardware security papers):
-
What are the background and motivations for this work? Put the work into context, try to relate to materials that we have covered in class, fill any knowledge gap that is needed for your classmates to understand the work.
-
What is the proposed idea of this paper?
Here it is very important for you to distinguish between high-level ideas and implementation details. Within the limited amount of presentation time, you need to pick the key components of the idea proposed by this paper and focus on the most important things to share with your fellow classmates. -
What is the work’s evaluation of the proposed solution? What argument, implementation, and/or experiment makes the case for the value of the ideas? It should be roughly easy to identify the evaluation section and read through what the authors did to validate their idea.
-
What are future directions for this research? What questions are you left with? What is your take-away message from this paper?
- We recommend the students try to connect the paper with their own background (if any)
- We like to hear critiques from the students. Last year, a team did their presentation by almost toating the paper for 10 min. We (the course staffs and the students) all enjoyed that presenation a lot. Unfortunately, we cannot share the recording of that particular presentation with you.
The third pass (optional for this course):
- The key to the third pass is to attempt to virtually re-implement the paper: that is, making the same assumptions as the authors, re-create the work. By comparing this re-creation with the actual paper, you can easily identify not only a paper’s innovations, but also its hidden failings and assumptions. This is usually needed when you are reviewing the paper as a program committee member.
- You should identify and challenge every assumption in every statement. During this pass, you should also jot down ideas for future work.
How will we run the discussion session?
During each paper discussion session, we will keep track of the time for each paper. If you run out of time, you will be interrupted and end up not finishing your presentation. So make sure to practice your presentation ahead of time.
- Presentation: 10 min
- Paper summary (motivation, proposed idea, evaluation): ~7 min
- Paper critiques (strengths, weaknesses, your thoughts, future work): ~3 min
- Class Q&A: 2 min
Timing is very tight – please come to class on time!
How will your presentation be graded?
Now comes what you care the most, i.e., how we will grade the presentation. First, let’s clarify the grading procedure. Your presentation will be rigorously graded by five persons: the instructor and the four TAs. Every judge’s score weights equally. The average score will be your final grade.
Second, we look for three aspects of the presentation.
- Your understanding of the paper, based on the content of your slides.
- The clarity of the explanation for technical concepts and details, based on the design of your slides.
- The oral presentation quality.
The judges will give letter grades for items 1 and 2. The letter grades are S, A, B, D.
- S: spectacular, award level
- A: good, but not impressive
- B: fair, can be improved
- D: I do not think the presenters put much efforts into it…
The final grade is computed by multiplying the score for the slide content and the score for the design of slides. As such, if you put enough efforts in understanding the paper and putting together the slides, you should get a satisfying score.
You may notice that we do not count item 3 (the oral presentation quality) into your grade. The reason is that we want the paper discussion session to be a welcoming environment for the students to practice public speech and learn from each other. We understand that some students may be shy, and international students might have some language barriers. Therefore, the course staff have made the decision to not include the oral part into grading. Instead, we will give out some small prizes (which has nothing to do with course grades) to award excellent presentations.
Content of Slides (Understanding the paper)
We have the following requirements for the content of the slides.
- Background and motivation: the presentation should include sufficient background for the audience to understand the technical content. Try to connect to the materials that we have covered in class. Fill in any knowledge gap.
- Key contribution: the presentation should provide an intuitive high-level summary of the key contributions, assited with illustrative examples/explanations to guide the audience to thoroughly understand the proposed techniques. A good presentation should pick out the most important things and spend sufficient time going through to convey the concepts or techniques to the audience.
- Evaluation: summarize the main evaluation results, explain why the authors can get such results, and highlights important takeaway messages.
- Critiques: comment on the strengths and weaknesses of the paper, focusing on the technical aspects.
❌ What should you avoid when summarizing a paper?
We have collected a few bad examples so you can also check to makes sure to avoid these pitfalls.
Background and motivation:
- Delve into the main contribution directly without providing the background of the work. The audience is left clueless about why the authors even need to pay the efforts to write this paper.
- Forget to explain some important terminology or acronym, since you are too familiar with them as your spend a few hours reading the paper. But your classmates have not read the paper, so they will be confused throughout the presentation if you repeately use terms they can only guess the meaning of.
Key contribution:
- In the slide, copy the concise and high-level description the authors put in the introduction section or the first paragraph of the main section, without providing extra explanation of what this sentence mean.
- Try to cover content from every section in the paper. This is not feasible. Note these technical papers are usually dense, and no one can have the capability to digest a 12-page paper within 10 minutes. Often, less is more. So you need to spend a lot of time carefully thinking and picking which sections should be included in your presentation.
- Refer to undefined terms. Sometimes the authors may want to come up with fancy names or acronomy for some components of their design or technique. You should not indulge yourself to use them freely, meaning mentioning them without providing a definition, since your classmates will get lost. Overall, the articulation of the proposed technique should be self-contained.
- Show a pseudocode or a figure without explaining what the setup is and what this code/figure does. A useful thumb of rule is that, anything you decide to include in the slides, you should talk through them so the audience can fully grasp what they mean, rather than “oh… the only impression I have is that there are some code…”
Evaluation:
- Go through every figure/plot in the evaluation section. This is not a good idea, since not all the evaluation results matter equally. Most of these papers went through a tough peer review process, where the reviewers may suggest extra experiments for the authors to further validate the design to make the paper more complete. You need to pick the important ones to present.
- Only present the results without explaining why the authors obtain the results, how the results support the contributions of the papers, and what lessons we can learn from these results.
Critiques:
- The paper is not well written. This is not a very useful comment for the audience since they will not read the paper. Better to focus on technical comments.
Design of Slides (Explaining the paper)
We provide the following excellent examples for your inspiration. Both teams won the TA-chosen presentation awards last year. What you can easily see from the videos is that both teams leveraged figures and animations to assist the concepts they try to convey.
- LLC is Practical by Baltasar Dinis and Rem Yang (slides, video)
- Flip Feng Shui by Daniël Trujillo (slides, video)
To help you prepare your presentation, we also provide a few bad examples crafted by the TAs.