This is the third and final part of the Test stage of Designing Thinking. The goal here is to (1) analyze the data that you collected in your experiment, (2) to write up your experiment report, and (3) write up final project level conclusions & recommendations, and reflect deeply on the design process that you followed. Please note that only the first component of this milestone (analysis) is done as a group, and the rest is done individually.
Part A serves as a contained report of the experiment that your team conducted for Test – Part 2.
Step 1: Analyze the data (group)
Based on what you have learned in class to date, conduct: (1) the planned quantitative statistical analyses, (2) any additional quantitative summarization, and (3) your qualitative summary of the data, which looks for themes, key representative examples, or any particularly interesting outlier responses. Where appropriate, you should be triangulating the quantitative and qualitative data.
From this point onward, you will be working individually.
Step 2: Write up your experiment report (individual)
Write up your individual report. The content of the experiment report must follow the outline given in Saul Greenberg’s outline for experiment reports, which can be found here [pdf]. Note that that outline serves as a general template for writing up experiments, for example experiments that you might conduct for your graduate research and submit to an HCI conference.
Here is some additional guidance for a subset of the specific sections of the report (the numbering below matches the numbering of the sections in the Greenberg report):
Abstract:
Refer to the examples of research papers from the course readings for examples of good abstracts.
2. Description of the experiment:
Some of what you include in this section can be cut and pasted directly from your team’s earlier milestone (Test – Part 1, with changes reflected from piloting). You should change the tense from future tense to past tense to reflect what you actually did. Divergence in the experiment methods from Test – Part 1 must be clearly marked, so that the reader can easily determine what has changed since that milestone. The remaining sections should be written entirely in your own words.
3. Results:
Think carefully about how to present your results for maximum visual impact. Use plots/graphs whenever it makes sense; these are usually easier for the reader to understand, and often have more impact. Include mention of any outliers, any interesting demographic differences, and more generally any surprises. Make sure that your statistics are clearly reported, including degrees of freedom and p values.
4.4. Limitations:
You should be able to identify several limitations to the experiment you conducted (and you may have predicted some of them already in your earlier milestone). The following questions may help guide you in this process. Were there any threats to validity (of any of the different forms discussed in class)? To what extent did your prototype support the needs of the experiment? Were there procedural breakdowns that occurred during study execution? Did your hypotheses play out as you expected? If not, can it be explained by some problem in the way the experiment was run? Were the participants biased in any way (e.g., were any of them classmates, or were they the same as participants you used in an earlier stage of the project)?
5. Conclusions:
Summarize what you learned from the experiment. You may find it helpful to reflect on your original research questions which should either be strongly implied or explicitly stated in the Introduction of the report. You can summarize the key insights gained from the experiment in terms of the key strengths and weakness of your interface (or interfaces, if you compared more than one prototype), the relative importance of these strengths and weaknesses as you have learned them from users, and how your view of the situation changed from prior to the evaluation. It is useful to reiterate the key positive characteristics of the current interface(s), as well as to note key deficiencies.
Note that we ask you to stop short of making explicit recommendations here: these will be the focus of Part B.
6. References:
You will likely already have some relevant references from earlier milestones that can be used in your experiment report. Those references were gathered before you designed your experiment. You will likely find it valuable to add additional references relevant to your specific experiment.
A.1 Experiment Report (individual)
Length: up to 10 pages (+ Appendices)
Part B is about broader project considerations, beyond just the experiment.
Step 3: Formulate final overall project conclusions and design recommendations.
Your project is broader than what your experiment was likely able to evaluate. Your goal here is to synthesize overall project-level conclusions and recommendations.
B.1 Report Conclusions: Discuss what you can conclude about your original research problem, the quality of your overall design concept, and its specific user interface design to address that problem. If you were able to assess this within your experiment, please simply refer the reader back to your Experiment Conclusions (Part A).
B.2. Report Recommendations: Decide what your findings throughout the entire project mean in terms of the next logical design step. In the real world, the resulting action from any of these conclusions would obviously depend on many other factors as well.
Length: up to 1 page for B.1 and B.2
Step 4: Reflect on your overall design process and your experience designing an interactive system.
The goal of this step is to first reflect openly on the design process you followed throughout this project. What aspects of the design thinking process worked well for your project? What aspects did not work so well? This part is free-form, and there is no specific right answer. We are interested in your honest thoughts.
Here are some questions to get you started:
B.3 Report your reflections as described above.
Length: up to 1 page
Step 5: Propose project ideas for next year’s class.
Having experience the process of a design thinking / human-centered design course project, what could be good project topics for next year’s teams to consider. Remember that that they should not be too vague, but also not too narrow, and that the timeframe for making progress on the topic following the methodology is about 3 months.
B.4 List 2-3 project ideas in one sentence each.
Length: 3 bullet points.
Step 6: Apply design thinking to a research project topic of your choice.
The goal of this step is to start you thinking beyond the team project that you tackled this term, towards the research project that you will eventually be working on for your Masters or PhD degree (if you will not be working on a research project for your degree, you can make one up that you would like to work on). We recognize that you may not yet know the details of your graduate research project, but you likely roughly know the research problem space. Start brainstorming how you might apply what you have learned in this course to addressing that problem.
B.5. Report your research problem and how you will apply design thinking methods. You are asked to: (1) describe the research problem, as succinctly as you are able, and then (2) briefly propose two distinct ways that design thinking methods might be useful. For the later, you need to be as specific as possible; e.g., name the specific method that you propose to use and clarify how you might use it, with which sample of users, if any. For each, you should justify why it is appropriate for your research problem. You can feel free to consult with your (prospective) graduate supervisor/advisor to help you with (1).
Length: up to one page.
Report submission:
For this final milestone, each student will individually submit one pdf document that includes all the required components for both Part A and Part B.
Include an appendix that identifies each team member’s contributions to the data analysis — each team member should have an appendix with the identical content. You many optionally include additional appendices for content that does not fit into the main sections of the report, e.g., supplementary analysis, if any (e.g., extra plots for which you didn’t have room in the report but feel should be included for completeness).
The pdf document should be named “Test_3-report-<team name>-<student lastname>.pdf” (e.g., Test_3-report-TeamX-McGrenere.pdf).