Course Description, Goals, and Objectives

This course introduces concepts, practices, and tools for conducting effective research. You will learn how to read technical papers, interpret published research, assess the research frontier, select research topics, devise research questions and hypotheses, propose and plan research activities, analyze experimental results, and report those results. 

The course is structured around five activities: (1) In-class lectures on basic research strategies and techniques; (2) In-class activities that apply course concepts; (3) Reading and in-class discussions of technical papers in computer science; (4) An individual semester-long empirical research project; and (5) Review and feedback on other student’s projects. The course requires significant reading, reviewing, and writing. Students are expected to participate actively in class activities and to provide meaningful comments on the work of other students.  

The course is taught in two versions. One version — COMPSCI 602 — is intended for PhD students. This course will help PhD students accelerate their research and increase their changes of successful completion of a dissertation. The other version — COMPSCI 590RM — is intended for MS students and undergraduates. This course provides a grounding in research methods that will aid entry into research-oriented industrial positions and later PhD studies. For MS and undergraduate students considering PhD studies, this course will help inform and accelerate that direction. For undergraduates, this course can also be used to satisfy the 499Y requirement for Departmental and Multidisciplinary Honors students whose theses or projects have a substantial empirical component.

Schedule and course topics

Course Introduction (Week 1) — Goals; Course content; Mechanics; Activities: paper responses, project, reviewing, discussion; Grading; Practical advice.

The Nature of Science and Technology (Week 2) — Motivation for studying research methods; The scientific content of CS; Characterizing CS research in terms of system, task, environment, and phenomena; Frameworks for understanding connections between research and development; Pasteur’s quadrant; Technologies and scientific phenomena; The physical, design, and intentional stances; Diax’s rake; Description vs. explanation; Empiricism.

Finding and Characterizing the Frontier (Week 3) — Examples of science-driven vs. technology-driven investigation; Strategies for investigation; Decomposition and localization; Reductionism; Assessing the frontier and the Medawar zone; Identifying useful papers; Goals and pathologies of scholarly publishing; Types of research papers; How to read a paper.

Formulating Research Questions and Hypotheses (Week 4) — Research questions and their utility; How to formulate good research questions; Hypotheses and their utility; Types of hypotheses; The falsifiability criterion; The utility of entertaining multiple working hypotheses.

The Fundamental Challenges of Research (Week 5) — The intrinsic challenges of research; Confirmation holism; Individual and community biases; Scientific misconduct; Why most published research findings are false.

Sources of Evidence and Assessing Statistical Evidence (Week 6) — Sources of evidence: observation, experiment, simulation, and proof; Internal and external validity; Tactics for research programming; The logic of statistical hypothesis tests; Bias and variance; Statistical and substantive significance; Standard deviation vs. standard error; Confidence intervals; Multiple comparisons; Interpreting and reporting hypothesis tests.

Causation and Mechanism (Week 7) — Causation vs. association; Value of causal inferences; Inferring causality; Berkson’s paradox; Abductive inference; Causal designs; Randomization; Instrumental variable designs; Propensity score designs; Mechanistic hypotheses; Differentiating simple causation and mechanism; Computational mechanisms; Challenges of discovering mechanisms: complexity, vestiges, spandrels, supervenience.

Tactics for Experimentation (Week 8) — Key concepts in experimental design; Generic and specific challenges of research design; Specific opportunities and challenges of research design in CS; Tips for using factorial designs; Decomposition and localization; Ablation studies.

Personal Productivity and Having Good Ideas (Week 9) — Typical responses to productivity challenges: work harder, work efficiently, work smarter; The problem of cognitive span; Psychological challenges; Systems and practical tips for managing projects and next actions; Strategies for having, recording, and using good ideas.

Writing and Reviewing Papers (Week 10) — Goals and utility of writing papers; Structure of research papers in CS; Authorship responsibility and decisions; Conceptual frames for writing; Practical writing tips; Writing myths; Characteristics of good graphics; Goals and utility of peer review; Aspects of reviewing papers and proposals; Practical reviewing tips; Common errors, psychological issues, and ethical issues in reviewing; Double-blind review; Variance in reviewing; Future directions for peer review; Effectively using reviews.

Managing Ideas and Proposing Research (Week 11) — Typical review criteria; Heilmeier’s questions; Top reasons that proposals are rejected; Assessing, reducing, and managing research risk; Research portfolios; Research metrics; Campbell’s Law; Recognizing “cranks”; The Gartner Hype Curve; Strategies for selecting projects; Starting and abandoning projects; Managing the “sunk cost” problem.

Fostering Professional Relationships (Week 12) — Learning the culture of science; Strategies for success in graduate school; Managing the relationship with your advisor; Managing professional relationships throughout your career; Making the most of scientific meetings.

Navigating the Scientific Enterprise (Week 13) — Alternative research organizations; Considerations for selecting where to work; Typical motivations for moving organizations; The CS research enterprise; Who funds R&D and why it is funded; Trends in R&D funding.

Course Wrap-up (Week 14)