{"id":185,"date":"2025-11-19T19:26:29","date_gmt":"2025-11-19T19:26:29","guid":{"rendered":"https:\/\/groups.cs.umass.edu\/jensen\/?page_id=185"},"modified":"2025-11-19T19:37:41","modified_gmt":"2025-11-19T19:37:41","slug":"research-methods","status":"publish","type":"page","link":"https:\/\/groups.cs.umass.edu\/jensen\/teaching\/research-methods\/","title":{"rendered":"Research Methods for Empirical Computer Science"},"content":{"rendered":"\n

Course Description, Goals, and Objectives<\/strong><\/p>\n\n\n\n

This course introduces concepts, practices, and tools for conducting effective research. You will learn how to read technical papers, interpret published research, assess the research frontier, select research topics, devise research questions and hypotheses, propose and plan research activities, analyze experimental results, and report those results. <\/p>\n\n\n\n

The course is structured around five activities: (1) In-class lectures on basic research strategies and techniques; (2) In-class activities that apply course concepts; (3) Reading and in-class discussions of technical papers in computer science; (4) An individual semester-long empirical research project; and (5) Review and feedback on other student’s projects. The course requires significant reading, reviewing, and writing. Students are expected to participate actively in class activities and to provide meaningful comments on the work of other students.\u00a0\u00a0<\/p>\n\n\n\n

The course is taught in two versions. One version \u2014 COMPSCI 602 \u2014 is intended for PhD students. This course will help PhD students accelerate their research and increase their changes of successful completion of a dissertation. The other version \u2014 COMPSCI 590RM \u2014 is intended for MS students and undergraduates. This course provides a grounding in research methods that will aid entry into research-oriented industrial positions and later PhD studies. For MS and undergraduate students considering PhD studies, this course will help inform and accelerate that direction. For undergraduates, this course can also be used to satisfy the 499Y requirement for Departmental and Multidisciplinary Honors students whose theses or projects have a substantial empirical component.<\/p>\n\n\n\n

Schedule and course topics<\/strong><\/p>\n\n\n\n

Course Introduction<\/em><\/strong> (Week 1) \u2014 Goals; Course content; Mechanics; Activities: paper responses, project, reviewing, discussion; Grading; Practical advice.<\/p>\n\n\n\n

The Nature of Science and Technology<\/em><\/strong> (Week 2) \u2014 Motivation for studying research methods; The scientific content of CS; Characterizing CS research in terms of system, task, environment, and phenomena; Frameworks for understanding connections between research and development; Pasteur\u2019s quadrant; Technologies and scientific phenomena; The physical, design, and intentional stances; Diax\u2019s rake; Description vs. explanation; Empiricism.<\/p>\n\n\n\n

Finding and Characterizing the Frontier<\/em><\/strong> (Week 3) \u2014 Examples of science-driven vs. technology-driven investigation; Strategies for investigation; Decomposition and localization; Reductionism; Assessing the frontier and the Medawar zone; Identifying useful papers; Goals and pathologies of scholarly publishing; Types of research papers; How to read a paper.<\/p>\n\n\n\n

Formulating Research Questions and Hypotheses<\/em><\/strong> (Week 4) \u2014 Research questions and their utility; How to formulate good research questions; Hypotheses and their utility; Types of hypotheses; The falsifiability criterion; The utility of entertaining multiple working hypotheses.<\/p>\n\n\n\n

The Fundamental Challenges of Research<\/em><\/strong> (Week 5) \u2014 The intrinsic challenges of research; Confirmation holism; Individual and community biases; Scientific misconduct; Why most published research findings are false.<\/p>\n\n\n\n

Sources of Evidence and Assessing Statistical Evidence<\/em><\/strong> (Week 6) \u2014 Sources of evidence: observation, experiment, simulation, and proof; Internal and external validity; Tactics for research programming; The logic of statistical hypothesis tests; Bias and variance; Statistical and substantive significance; Standard deviation vs. standard error; Confidence intervals; Multiple comparisons; Interpreting and reporting hypothesis tests.<\/p>\n\n\n\n

Causation and Mechanism<\/em><\/strong> (Week 7) \u2014 Causation vs. association; Value of causal inferences; Inferring causality; Berkson\u2019s paradox; Abductive inference; Causal designs; Randomization; Instrumental variable designs; Propensity score designs; Mechanistic hypotheses; Differentiating simple causation and mechanism; Computational mechanisms; Challenges of discovering mechanisms: complexity, vestiges, spandrels, supervenience.<\/p>\n\n\n\n

Tactics for Experimentation<\/em><\/strong> (Week 8) \u2014 Key concepts in experimental design; Generic and specific challenges of research design; Specific opportunities and challenges of research design in CS; Tips for using factorial designs; Decomposition and localization; Ablation studies.<\/p>\n\n\n\n

Personal Productivity and Having Good Ideas<\/em><\/strong> (Week 9) \u2014 Typical responses to productivity challenges: work harder, work efficiently, work smarter; The problem of cognitive span; Psychological challenges; Systems and practical tips for managing projects and next actions; Strategies for having, recording, and using good ideas.<\/p>\n\n\n\n

Writing and Reviewing Papers<\/em><\/strong> <\/em>(Week 10)<\/strong> \u2014 Goals and utility of writing papers; Structure of research papers in CS; Authorship responsibility and decisions; Conceptual frames for writing; Practical writing tips; Writing myths; Characteristics of good graphics; Goals and utility of peer review; Aspects of reviewing papers and proposals; Practical reviewing tips; Common errors, psychological issues, and ethical issues in reviewing; Double-blind review; Variance in reviewing; Future directions for peer review; Effectively using reviews.<\/p>\n\n\n\n

Managing Ideas and Proposing Research<\/em><\/strong> (Week 11) \u2014 Typical review criteria; Heilmeier\u2019s questions; Top reasons that proposals are rejected; Assessing, reducing, and managing research risk; Research portfolios; Research metrics; Campbell\u2019s Law; Recognizing \u201ccranks\u201d; The Gartner Hype Curve; Strategies for selecting projects; Starting and abandoning projects; Managing the \u201csunk cost\u201d problem.<\/p>\n\n\n\n

Fostering Professional Relationships<\/em><\/strong> (Week 12) \u2014 Learning the culture of science; Strategies for success in graduate school; Managing the relationship with your advisor; Managing professional relationships throughout your career; Making the most of scientific meetings.<\/p>\n\n\n\n

Navigating the Scientific Enterprise<\/em><\/strong> (Week 13) \u2014 Alternative research organizations; Considerations for selecting where to work; Typical motivations for moving organizations; The CS research enterprise; Who funds R&D and why it is funded; Trends in R&D funding.<\/p>\n\n\n\n

Course Wrap-up<\/strong> (Week 14)<\/p>\n","protected":false},"excerpt":{"rendered":"

Course Description, Goals, and Objectives This course introduces concepts, practices, and tools for conducting effective research. You will learn how to read technical papers, interpret published research, assess the research frontier, select research topics, devise research questions and hypotheses, propose and plan research activities, analyze experimental results, and report those results.  The course is structured … Continue reading “Research Methods for Empirical Computer Science”<\/span><\/a><\/p>\n","protected":false},"author":16,"featured_media":0,"parent":10,"menu_order":0,"comment_status":"closed","ping_status":"closed","template":"","meta":{"footnotes":""},"class_list":["post-185","page","type-page","status-publish","hentry","hfeed"],"_links":{"self":[{"href":"https:\/\/groups.cs.umass.edu\/jensen\/wp-json\/wp\/v2\/pages\/185","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/groups.cs.umass.edu\/jensen\/wp-json\/wp\/v2\/pages"}],"about":[{"href":"https:\/\/groups.cs.umass.edu\/jensen\/wp-json\/wp\/v2\/types\/page"}],"author":[{"embeddable":true,"href":"https:\/\/groups.cs.umass.edu\/jensen\/wp-json\/wp\/v2\/users\/16"}],"replies":[{"embeddable":true,"href":"https:\/\/groups.cs.umass.edu\/jensen\/wp-json\/wp\/v2\/comments?post=185"}],"version-history":[{"count":1,"href":"https:\/\/groups.cs.umass.edu\/jensen\/wp-json\/wp\/v2\/pages\/185\/revisions"}],"predecessor-version":[{"id":186,"href":"https:\/\/groups.cs.umass.edu\/jensen\/wp-json\/wp\/v2\/pages\/185\/revisions\/186"}],"up":[{"embeddable":true,"href":"https:\/\/groups.cs.umass.edu\/jensen\/wp-json\/wp\/v2\/pages\/10"}],"wp:attachment":[{"href":"https:\/\/groups.cs.umass.edu\/jensen\/wp-json\/wp\/v2\/media?parent=185"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}