Let’s talk about a meta paper today. This paper presents an interesting perspective on how to read a paper. While this paper is aimed at graduate students and researchers, we’ll try to adapt the takeaways for practitioners :)

How to read a paper, ACM SIGCOMM Computer Communication Review, July 2007 by S Keshav. 121 citations.

https://dl.acm.org/doi/abs/10.1145/1273445.1273458↗

Motivation

At the outset, we need to answer the question why am I reading this paper? Common answers include being aware of recent developments, review and appraise as part of service, deeply understand a concept and apply to own research. This determines the depth of our study.

The paper proposes a three pass approach to the study. With each pass, try to revisit the objective and observe if its met. Second, given the volume of new research, it is crucial to eliminate the content that is not relevant. The earlier we determine this, the efficient our study becomes.

Solution

First pass is a 10-15min exercise. Scan through the section headings, abstract and conclusion. Capture five aspects: category, context, contributions, correctness and clarity. Category and Context broadly determine the relevance with subject and problem statement. Contributions and Correctness help answer if this paper is important in its field. Clarity is a function of exposition quality and depth of the authors.

Second pass is 1hr exercise. Study the diagrams and read through the concepts. Skip the detailed proofs. Goal is to understand the flow of thought, reasoning and an ability to summarize the paper with evidence to someone else. Author suggests an active reading i.e. noting down references, background and any comments.

Third pass is a 2-3hr effort. Try to recreate the paper by questioning the assumptions and statements. Study the premise and conclusions deeply. Two key outcomes of this pass: a) note down the future ideas, and b) call out any interesting tools you observe. Both of these add to your repertoire for research.

Algorithm to find papers

1. Use academic search engines like Google Scholar or Citeseer. I’d like to add Microsoft Academic to the mix for its nice taxonomy.
2. Find key researchers in an area by looking at top references.
3. Find key conferences by looking at the recent publications of these researchers.
4. Now look into the conference proceedings and find the top papers.

When I started reading papers, I would commit myself to a third pass always. As a practitioner, this doesn’t payoff well and time constraints decrease the motivation to keep up the habit. The constraints on each pass e.g. boundaries of exploration and time limit are a fantastic tool. This objectivity is a definite antidote to the perfectionist in me.

My experience on the algorithm to find good papers is a bit different. I stumbled upon good blog posts looking for engineering challenges. Observe which topics and papers they talk about. Rest of the algorithm matches - find the conferences the papers were submitted, look at references and find more key people. There’s another interesting source: look for graduate level courses on the topic. Most of the good universities publish their Paper Reading Lists on the course page. These turn out to be a treasure.