10.1.1.94.4999

Literature search for computer science Todd Veldhuizen Chalmers Technical University June 17, 2005

1 Why read the literature? • Because sometimes a year of hard work can save a week of reading. • Increase the likelihood that someone will care about your research: Not duplicate an already known result. Identify communities of people who will appreciate what you are doing. Be able to give a convincing treatment of your research problem: ∗ Using standard terminology and notations; ∗ Comparing your contribution to related work; ∗ Avoiding obvious ‘newbie’ mistakes ∗ Not getting crushed by some crusty professor who claims to have published your result twenty years ago.

2 Why you shouldn’t read the literature • By reading the literature, you may accept the prevailing wisdom unquestioningly, and unconsciously close avenues of investigation that might otherwise have occurred to you. Many research ﬁelds arise from a real problem or question that needs to be solved. In the early stages, many approaches are considered. After a while, one approach usually wins out. This approach becomes the paradigm; people cease asking if it is the right answer; the solution becomes the research question. The original question is often forgotten. Nobody notices that time has changed the question, or that developments in other areas make the question irrelevant or change the criteria or suggest new solutions.

3 Why you shouldn’t read the literature • As a newcomer, you are in a unique position to see the problem clearly from a fresh perspective, without the calcification. Harish-Chandra (Cole Prize in Algebra, 1954): I have often pondered over the roles of knowledge or experience, on the one hand, and imagination or intuition, on the other, in the process of discovery. I believe that there is a certain fundamental conflict between the two, and knowledge, by advocating caution, tends to inhibit the flight of imagination. Therefore, a certain naivete, unburdened by conventional wisdom, can sometimes be a positive asset.

4 Why you shouldn’t read the literature • A conflict; need a balance of enlightenment and ignorance in order to do original work. • A good strategy when embarking on a new piece of research is to explore first, and read second. Try to think of as many possible solutions and approaches as you can. Evaluate them and flesh out the most promising. Then do some literature search. Chances are good that if it’s a problem that’s been encountered before, some of the avenues of investigation you’ve identified have been tried. But maybe others haven’t.

5 What is a new idea? • The computer science literature contains (as of 2005) some 2 000 000 publications, depending on how you count.1 CiteSeer lists 750548 authors of papers in computer science (but many of these are not actively publishing, or are from other fields entirely). • The literature is so vast, and growing so quickly, that claims of “Nobody has done this before” are essentially not falsifiable; you should be very mistrustful of such claims.2 • Unless you are using a new theorem or approach that would have been 1 This is from The Collection of Computer Science Bibliographies, which has some duplication but also much omission. 2 A hypothesis is falsifiable if it admits the possibility of making an observation that refutes it.

6 unknown to someone a decade ago, you should assume the problem has been tackled before, and aim to do it diﬀerently, better, etc.

7 The structure of the CS literature Conference papers Workshop papers Journal papers Technical reports Tutorials Handbook papers Survey papers Textbooks and monographs

8 (Jim Parrott)

9 Computer science literature is atypical In many science disciplines, conferences are very informal, papers are accepted based on incomplete sketches or abstracts of often incomplete work. In computer science: • Conferences: Many of these are high-prestige with very low acceptance rates (e.g., 10-20%); these are comparable to journal papers in terms of the level of rigour required. Often work is expected to be highly polished and complete. Papers are routinely rejected because they “seem premature.”

10 Selection process favours rigorous, incremental work that can be completely conveyed in 10-25 pages, and whose import can be quickly grasped by experts Due to space limitations, conference papers are rarely self-contained and presume that readers already know the relevant background. Result: many conferences (in programming languages) are increasingly perceived as boring and/or inaccessible • Workshops: More preliminary and innovative papers; some people feel this is where research really happens. Often no published proceedings, or perhaps selected papers published as a special issue of a journal or LNCS volume • Technical reports:

11 A mixture of rejected papers, longer versions of conference papers, papers in submission to conferences/journals that the authors want to make available The equivalent of preprints in math/physics. Ironically, computer scientists are slower to adapt to new publishing mechanisms than mathematicians and physicists. • Journals: Often these are longer versions of papers previously published at conferences, that contain more background, details, etc. Serves an archival purpose: a polished, self-contained piece of work that will be available to future generations of researchers. Often the early sections of these papers are quite accessible. • Tutorials, handbook articles, surveys, encyclopedia articles

12 Useful when trying to get a grasp of an unfamiliar ﬁeld Tutorials often come from summer schools, and aim to teach the reader material that may be unfamiliar to them. Handbook articles come from “The Handbook of X” (where X is Theoretical Computer Science, Logic in Computer Science, Artiﬁcial Intelligence, Mathematical Logic, etc.) — much more common for theory/mathematics, or disciplines that exhibit a high degree of “community spirit.” Similar to tutorials, but often aimed at a higher level and more in-depth. Surveys collect and categorize the published work on a topic. ACM CSUR (Computing Surveys) is an entire journal devoted to surveys of topics in computer science; it is an excellent journal to browse to get a feeling for what is going on in the discipline.3 3 In fact, if you made the commitment to read one survey article a week, you’d soon be one of the best informed computer scientists around. Very few computer scientists keep up with what is going on outside their specialty.

13 Encyclopedia articles are similar to handbook articles, but aim more at quick reference than exhaustive exposition. e.g., The Stanford Encyclopedia of Philosophy • Textbooks and Monographs Textbook: a very long tutorial (more or less) Monograph: a very long journal paper (more or less)

14 Researchers • Professors Pre-tenure Tenured • Graduate students • Researchers at industrial or national laboratories (e.g., AT&T, IBM Research, INRIA, Argonne, NERSC, ...)

15 Peer Review A process by which research is evaluated by other researchers before being published. For example, at a conference: The program chair(s) appoint a program committee of researchers in the ﬁeld. It is the duty of this committee to select the best papers submitted for inclusion in the conference program. Researchers submit their papers by a given deadline date. Typically, each paper is reviewed by 3-4 members of the program committee and evaluated. Some program committee members may pass the paper along to experts not on the committee for their appraisal. Reviewers write a referee’s report indicating their opinion of the paper: its strengths and weakness, suggestions for improvement, and so on. There is then a period of discussion during which the committee argues about which papers to accept and which to reject.

16 Peer Review: Good points • Keeps obviously wrong, dilettantish, half-baked research out of the literature. • Provides authors with feedback and (usually constructive) criticism to help them improve their work and its presentation. • Ensures that published research meets some kind of minimal standard. But: this does not mean published research is interesting, or that research meeting the standard is publishable.

17 Peer Review: Bad points • There is no reward for making research look easy. Results are often presented in the most complicated way such that expert reviewers can appreciate it, but gives them a feeling the research is nontrivial. People speak openly of the need to ‘play the game’ (i.e., make it look hard). Reviewers are usually experts, and often impatient with in-depth background or tutorial-style material, so this part of papers tends to minimal. Papers are written to pass review — the needs of the intended audience (e.g., grad students and non-experts) are secondary.

18 Peer Review: Herd science • Enforces a tendancy toward herd science: papers that support prevailing wisdom are more likely to be accepted (conﬁrmation bias). Enrico Fermi: “Never underestimate the joy people derive from hearing something they already know. ” Thomas Gold’s The Inertia of Scientiﬁc Thought: sobering reading about the tendancy of scientists to travel in herds.4 • Research that explores alternatives to the accepted approach is more heavily criticized 4 Gold won the 1985 Royal Astronomical Society Gold Medal.

19 Peer Review: Incremental bias • Particularly with conferences, there is a pronounced bias toward papers whose import can be quickly grasped. Research that is challenging, revolutionary, or very innovative is often dismissed simply because the reviewers don’t “get it” in the limited time they have to review the paper. A maxim: any suﬃciently advanced research is indistinguishable from bullshit. • Some more progressive conferences are starting to set up special tracks for innovative research, to combat this trend (e.g., the OOPSLA Onwards! track)

20 Kinds of research: Problem-oriented Seeks to solve a problem or understand a phenomenon Usually an objective standard of what constitutes success: proof, experimental results that yield insight on the phenomenon, performance on benchmark problems, usability studies, etc. Generally open to a wide array of approaches – anything is valuable, if it works or yields insight More in tune with what we think of as “science” (experimental veriﬁcation, replicability, falsiﬁcation, etc.)

21 Kinds of research: Paradigm-oriented Starts from a presumption that a given idea or paradigm is true/valuable, and tries to explore, further develop, justify, ﬁnd applications, etc. Often have a community of researchers who band together based on similar interests; conferences with titles like Aspect-Oriented Software Development, International Conference on Functional Programming, C++ Templates Workshop, Object-Oriented Programming Languages and Systems, etc. Often not receptive to studies that question, attack, or propose alternatives to the paradigm. Often lack an objective criterion of success; papers are judged based on whether they are “novel” or “interesting,” rather than how well they perform on some benchmark problems. Nonetheless a lot of interesting ideas come out of such communities.

22 When doing problem-oriented research, you frequently need ideas from several such communities, and the fact that these areas have been thoroughly explored by paradigm-oriented researchers is very helpful.

23 Problem-oriented research: a process 1. Identify the problem. Write a problem statement that avoids a pre- emptive selection of approaches or solutions. Understand the context in which the problem must be solved. 2. Identify constraints any solution must satisfy, and an objective criterion that a successful solution must meet. 3. Enumerate possible alternative solutions, and judge them according to the criterion and constraints. 4. Select the best solution. Implement and evaluate it. For paradigm-oriented research: reverse the above process. Start with the best solution, and work backwards to ﬁnd a problem it solves.

24 Fads and trends Four stages of acceptance: i) this is worthless nonsense; ii) this is an interesting, but perverse, point of view; iii) this is true, but quite unimportant; iv) I always said so. • Paradigm-oriented communities seem particularly susceptible to fads, fashions, memes, bandwagons, etc. • With the beneﬁt of hindsight, the amount of research that gets published on a fad is sometimes hugely out of proportion to its eventual impact. • Sometimes bandwagon-hopping is healthy: when a new technique/approach appears, it’s unclear what it’s good for and how it compares to other approaches. A lot of “fad” papers help answer useful questions of the form: “Is X good for Y?”

25 Anatomy of a programming languages fad • 2005 Martin Minton, a mid-career academic, writes a paper about a new approach to software development he calls Foozles. Old-timers grumble that Foozles were well-known in Lisp, under a diﬀerent name, and that it is all laid out clearly in Erik Sandewall’s 1978 Lisp survey. • 2006 Students of Martin Minton release a module for the popular software development environment Equinox that makes writing Foozles easy and fun. • 2007 Phil Wadler writes a beautiful paper about Foozles with a clever title. Everyone reads it. • 2008 Foozle-Oriented Software Development conference held, with moderate success. Successive iterations will grow in popularity, with

26 ever more complex tool and language support for Foozles proposed and implemented. The book Mastering Foozles: Foozle-Oriented Software Development in Java appears, aimed at real-world programmers. • 2009 Franz H¨gler, the noted type theorist, publishes the acclaimed o paper A Theory of Foozles that presents the λfooz calculus. • 2011 First ever Foundations of Foozles conference held. Now Foozles are on a rigorous foundation, but also weirder and hard to understand. The λfooz calculus is acknowledged as a good start, but supplanted by foo/Z the λf ∞z and π calculi. Francoise Boisbleu proves that under a certain formulation, Foozles are a categorical dual to Aspects, which gets everyone terribly excited. • 2012 Waning interest in Foozles; they are increasingly marginalized by the newly-popular Nimble Arrows, which itself is about to be overtaken

27 by Magic Mixin Markups (M 3) • 2014 First-ever usability study of Foozles with real programmers suggests that their beneﬁts are outweighed by the code maintenance problems they create. Study politely criticized by young academics who wrote their dissertations on Foozles. The paper is rejected from FOSD VIII (Foozle- Oriented Software Development); reviewers remark that although the paper presents a promising research direction, the sample size is too small to draw conclusions, the study “feels preliminary,” and authors are criticized for insuﬃciently explaining what they mean by “Fisher’s test”. A short version is published at PPIG and cited heavily by proponents of the M 3 paradigm. • 2025 The 800-page tome Handbook of Foozle Theory is published in hardcover (retail price $ 265 to compensate for the small print run). Now largely divorced from its roots in programming languages, Foozle Theory

28 has evolved into a deep and complex subject that lies at the conjunction of category theory, term rewriting, and topology. An MSRI workshop is held. Papers on Foozle Theory still start with the sentence, “Foozles are increasingly recognized as a promising approach to managing program complexity, and an elegant theory has been developed to explain why [1,3,17,24,25,32,43,57,58,59,60]” even though most practitioners gave up on Foozles a decade ago.

29 Prematurity • Prematurity = the phenomenon in science of discoveries being made “before their time”, and unappreciated and/or neglected. Eventually, “the time is right,” and the discovery is suddenly relevant. Sometimes there is a rediscovery, and the original discoverer is forgotten. Sometimes there are periodic rediscoveries over decades, each quickly forgotten until the community is suddenly receptive. • Prematurity in Scientiﬁc Discovery: On Resistance and Neglect, ed. Ernest B. Hook, University of California Press, 2002. http://www. ucpress.edu/books/pages/9245/9245.ch02.html

30 Generalizations and Applications For any idea, you should be thinking about both its: • Applications– where can the ideas be applied? (This will help you ﬁnd people who will care about your research, similar problems that might give you ideas of how to solve your problem, etc.) • Generalizations– how can the idea be made more abstract? What theory applies? I ﬁnd it helpful to think of an idea as the center of a tree that extends upwards (to generalizations/abstractions) and downwards (to applications). By exploring up and down this tree, you will get insight into the problem, and turn up literature that is related by analogy to your problem.

31 Its generalizations An Idea Its applications

32 Example: Graph algorithm parallelization Idea: To parallelize a graph algorithm (in a SPMD setting) you must decide how to distribute the graph representation over the processors. Applications: parallel breadth-ﬁrst search, parallel ﬁnite element methods, parallel shortest path, etc. (These in turn have their own specialized applications: parallel shortest path for graphs where the edge weights are a distance metric, parallel shortest path for planar graphs with Euclidean distance metrics, ...) Generalizations: spectral partition methods, metrical task systems, competitive algorithms, edge cutset problems, load balancing, metalevel deliberation, approximation algorithms, ... By doing forward bibliographic search on a problem’s generalizations, you

33 can turn up problems that are analogous: data replication for distributed ﬁle systems, page migration for distributed shared memory systems, data replication for distributed databases, ...

34 Online search tools for computer science http://osl.iu.edu/~tveldhui/tools/

35 Google Scholar http://scholar.google.com/ • Harvests publications from lots of other sources (CiteSeer, ACM, IEEE, etc.) • Excellent to find an individual paper by title/authors: if it is online somewhere, changes are good Google Scholar will find it. • Pretty good at keyword searching. Indexes the text of the papers, which is very useful. • Bad: Lots of crosstalk with other disciplines (physics, medicine, biology) (e.g., try searching for “aspects” or “unification”)

36 Example: change impact

37 CiteSeer http://citeseer.ist.psu.edu/ http://citeseer.csail.mit.edu/ • Crawls the web and ﬁnds papers. Therefore heavily biased towards recently published papers (i.e., 1997-present) • Has two search modes: a keyword-type search (not very good), and a citation search, which is amazingly useful • Citation search will weight search results by how heavily cited they are. • Good for ﬁnding pivot papers (heavily cited papers) about a particular topic, and for browsing the citation graph forward/backwards.

38 • Often excruciatingly slow and/or too busy to answer queries. • Good for answering questions like, “What is so-and-so best known for?” and “Has so-and-so published anything with impact in the last ten years?” and “How cited is so-and-so?” • To search for a person “Paul W. McCartney,” have to do something like “(Paul McCartney) or (P McCartney) or (Paul W McCartney) or (P W McCartney)” to account for the ways people might cite papers. • Example: (Alan Turing) or (A Turing) or (A M Turing) or (Alan M Turing) Look at the bar graph at the bottom — note the war years spent at Bletchley. (Turing was convicted for homosexuality in 1952, and committed suicide in 1954.)

39 • Fun to use as a tool to predict when someone got tenure (do you see a steady build-up then sharp drop in cited papers in the graph?) • But note there is a diﬀerence between doing great research that will stand the test of time, and getting cited a lot. e.g., the most cited papers of 1999 include topics like CORBA, C++, Jini, XML, etc. These topics are ephemeral– in twenty years, hardly anyone will care. Lots of profound research doesn’t get recognized at ﬁrst. Some people who get cited heavily are brilliant innovators, some are brilliant communicators, and some are brilliant at picking bandwagons. A lot of really successful ideas get incorporated into textbooks, and then people cite those instead of the original research. • CiteSeer’s list of most cited papers are worth browsing. If you’re looking for worthwhile papers to read, this is a good start. I used to organize a

40 reading group (“High-Impact Papers in Computer Science”) where each week we’d pick a paper from the CiteSeer top cited papers list, read and discuss it.

41 Real research What is worth What is doing publishable You want to be here (I hope) careerists pragmatic frustrated scientists innovaters

42 Collection of Computer Science Bibliographies http://liinwww.ira.uka.de/bibliography/index.html • You have the paper, but you need the bibtex. • Search for ”The First Few Words of the Title” and you’ll probably ﬁnd it. • Beware of the CiteSeer bibtex entries: they are almost never in a suitable form for citing in papers. • Indexes many papers whose fulltext is not available online, so has larger coverage than CiteSeer or Google Scholar.

43 ACM Guide http://portal.acm.org/guide.cfm • Combines bibliography data from many sources • Full-text access to ACM publications (but, annoyingly, will not even link to others like IEEE, Springer, Elsevier, etc.) • Good for browsing the citation graph insofar as it lies in ACM publications • Fairly awful search capabilities compared to google or citeseer • Gives BibTeX entries, but screws up accents (get HTML escapes like é — ugh)

44 UMI Dissertations http://wwwlib.umi.com/dissertations/search • Good for ﬁnding dissertations on topics close to yours. • Restrict subject area to “computer science” • Dissertations often contain well-written introduction and survey sections. If you can ﬁnd one in your area, it may be a perfect starting point.

45 DBLP http://www.informatik.uni-trier.de/~ley/db/ • 610,000 publications • Bibliographic (no fulltext) • More complete coverage than CiteSeer (e.g., citeseer only has about 30% of the articles in DBLP for 2000; less than 10% for decades before that) • Very complete for database theory!

46 Searching the literature

47 Develop your background knowledge To search the literature effectively, you have to know what questions to ask (using the terminology of the field). Bad: How can I figure out by looking at a program whether it will stop or not, and can I make programs simpler so it’s easy to see they will always stop. Good: What is known about automatic termination analysis? Are there subrecursive languages that guarantee termination? To know what questions to ask, you need background in the fields that impinge on your research question. (To know what fields are related, you need a broad background— read surveys!)

48 Develop your background knowledge Identify all the research fields that seem even vaguely relevant to your question. Ask knowledgeable people what fields are involved. Look for and skim: Textbooks Tutorials Summer school proceedings Handbook/encyclopedia articles Introductory chapters of theses/dissertations Surveys These will introduce you to standard terminology in the fields, which will allow you to search the literature with appropriate keywords. Think about generalizations and applications.

49 Finding tutorials, etc. A good strategy is to search for your subject area and one of these keywords: Search terms: tutorial, summer school, handbook, dissertation, thesis, survey, introduction Example: Find a tutorial on Galois connections. Example: Find a survey paper (or three) on uniﬁcation. Also try pulling up a few research papers on the topic and skim the ﬁrst paragraphs; often people will point to relevant background material.

50 Things to collect while you search As you search the literature, you should be building sets of: • Effective search terms and terminology; • Key papers; • Key researchers. If this analogy helps, think of literature search strategies (looking for papers, people, search terms, etc.) as closure operators; you are trying to find the least fixpoint generated by an initial set of papers/terms.

51 The citation graph A graph in which publications are vertices, and edges are citations. You need to browse this graph to ﬁnd relevant work. Pivotal/canonical papers: these are heavily connected papers that either • Everyone cites, because they established the ﬁeld or made an important contribution or made something understandable; • Cite a lot of papers, because it is a survey or the authors did an unusually thorough assessment of related work. These papers are gold mines.

52 Search engines that weight papers according to citations are good at ﬁnding these (CiteSeer, Google Scholar) Example: Data structure selection

53 Backward bibliography search Given a paper in a ﬁeld, search backward through the papers it cites, and the papers those papers cite, etc. Can be done manually: given the paper, skim its bibliography and try to identify promising papers. Better: use an online tool that has links to previous papers (ACM digital library, CiteSeer) Example:

54 Forward bibliography search Given a key paper in a ﬁeld, search forward to ﬁnd papers that cite it. ACM, Google Scholar, CiteSeer are all good for this. Example: Data structure selection

55 By researcher Identify key researchers working in an area. Do bibliographic queries on their name. (DBLP is good for this). Visit their homepages and look through their recent publications — there is usually a lag of a year or more for papers to be published and indexed by bibliographical databases.

56 Ask around Find researchers knowledgeable in the relevant areas (ideally, people you know) and ask them if they can point you to relevant work.

57 Reading a paper Kinds of ways you will read a paper: * For gestalt, look for concepts and big ideas, ignore details (skimming) http://www.ucc.vt.edu/stdysk/skimming.html http://web2.uvcs.uvic.ca/elc/studyzone/570/pulp/hemp1.htm http://helios.hampshire.edu/~apmNS/design/RESOURCES/HOW_ READ.html * When you’re exploring the literature, don’t feel you need to read papers from beginning to end. Skim them, reject them as irrelevant, ﬂing them aside. Invest eﬀort in reading in proportion to their salience.

58 * Look for key ideas and concepts. Don’t get mired in details.

59 Learning progression It’s useful to revisit papers after a while. When you are trying to understand a new area, nothing will make sense to you. It is a waste of time to try to read a paper in detail, because it will be gibberish and you’ll just get frustrated. Instead, skim over a bunch of papers and try to get a feeling for the material. Look for key concepts and build an informal glossary for yourself (in a notebook, on a wiki, etc.) Make notes of unfamiliar terms and try to find definitions of these in textbooks, tutorials, etc. After a while, the field will gradually come into focus, and papers that were impenetrable a while before will start to make sense.

60 It’s best if you can space this process over time, so that understanding can seep in gradually. Nobody can master a new ﬁeld in a couple of weeks. Once the ﬁeld comes into focus, you’ll have a better feeling of what papers are relevant and deserve closer reading, and which are not.

61 Organizing your bibliography Do whatever works for you; here’s what I do. Whenever I read a paper, I put its bibtex in my repository, and save a digital copy of the paper (if I have one). • I use BibTeX for my bibliography information. A while ago I settled on the following standard format for bibtex tags: FirstAuthor:Venue:Year. Example: Donald P. Moore, Library loading with alternate routine selection,

62 Communications of the CACM 4(11), 1961. becomes Moore:CACM:1961 Michael O. McCracken, Allan Snavely and Allen Malony, Performance Modeling for Dynamic Algorithm Selection, Iternational Conference on Computational Science 2003. becomes McCracken:ICCS:2003 • A copy of the paper is saved to documents/2003/McCracken:ICCS:2003.pdf • Whenever possible, I include the abstract in the bibtex, and possibly add “summary,” “annote,” “comment” or “keywords” ﬁelds so I can ﬁnd the paper later by keyword searching: @article{Shaw:CACM:1977, author = {Mary Shaw and William A. Wulf and Ralph L. London}, title = {Abstraction and verification in Alphard: defining and s tion and generators},

63 journal = {Commun. ACM}, volume = {20}, number = {8}, year = {1977}, issn = {0001-0782}, pages = {553--564}, doi = {http://doi.acm.org/10.1145/359763.359782}, publisher = {ACM Press}, comment = { generics, pioneered the idea of iterators (here call } }

64 Some useful tools • bibfind: A perl script you can find online. Like grep, but especially for bibtex files. I use a script “bibgrep”: for f in $BIBINPUTS/*.bib; do bibfind "$*" $f done • viewpaper McCracken:ICCS:2003: another script that looks for documents/2003/McCracken:ICCS:2003.[ps,pdf,html] and pops up an appropriate viewer. Otherwise tries to find a DOI in the bibtex. • bibtex2html: A C program that turns our bibtex files into web pages,

65 indexed by authors, keywords, year, etc. I hacked it up to insert links to PDF/PS versions of papers when possible.

66 Essential readings Richard W. Hamming, You and Your Research. Thomas Gold, The inertia of scientiﬁc thought. References

10.1.1.94.4999

by Tran Diep Hue Man on Jul 26, 2010

Accessibility

Categories

Tags

Upload Details

Usage Rights

1 Embed 15

Statistics

10.1.1.94.4999 — Presentation Transcript