This is the first part of the Test stage of Design Thinking. The goal of this milestone is to craft an experiment design and to complete your medium-fidelity prototype.
Note that there is a relatively high degree of interaction between parts 1 and 2 during this stage of the project. While they are presented sequentially in this document, some of the steps will need to be done in parallel. You will need to brainstorm possible experiment goals and designs in order to help you scope your med-fi prototype(s); on the flip side you will not be able to commit to an experiment design until you know how far you will get with your prototype. Note that you need to submit your proposed goals for the experiment as an interim stage, and then incorporate the instructor feedback before proceeding.
1. Experiment Design
By now, you have one design approach (or possibly two), which you’ve subjected to a qualitative walkthrough analysis. It’s time to determine using a quantitative controlled manner whether this approach is effective.
Proposed Goal(s) of Experiment:
Assess the key usability challenges or uncertainties of your design approach. What needs to be answered to help establish whether this is a good design approach? Use the insights gained during the cognitive walkthrough to inform your reflection, as well as any other input you have obtained (e.g. by discussing with other teams).
If you have developed two prototypes, or there is a competitor system for certain tasks that your system supports, the main goal is likely to determine which of the two systems is most effective overall (or for given tasks). This represents a classic evaluation of Design A vs. Design B.
Another possible goal is whether users will be able to do a certain task within a threshold amount of time, or within a threshold number of errors. The goal in this case would be to test the task against a reasonable usability requirement target. This approach is well covered in Section 10.8 of the Experiments in Support of Design chapter — which the course staff can provide to you as a supplemental reading. Note that if you take this approach, you need to clearly justify the usability requirement target.
Interim reporting of the proposed goal(s) of experiment: Make a comprehensive list, then rank items by (a) importance and (b) your ability to test them within the scope of this course. Then clearly identify which of the goals you will address in your evaluation. As a guideline, you should have 1 substantial goal or up to 3 smaller goals.
Length: approximately half a page
Where: Please submit your document to Canvas with a title of as “Fifth project milestone: Proposed goals of the experiment ”
Final Goal(s) of Experiment:
1.1 Report the (revised) final goal(s) of experiment: Incorporating the feedback you received from your interim post, indicate the final goals that you will address in your experiment.
Length: approximately a third of a page
Design the Experiment:
Now it is time to actually design the experiment that will evaluate your system. This will require a lot of effort, much more than the informal evaluations you have conducted to date.
Determine your hypotheses. These are very specific instantiations of some of your evaluation goals. They need to be testable. You will likely have 2 to 4 hypotheses.
Consider your intended participants, and how you will recruit them. One possibility may be that you believe demographics will impact the usability of your system and so you may want subjects in two different demographic groups (i.e., this will be an independent variable in your experiment). One example of this is novice and expert users.
Determine the method you will use to test your hypotheses. This forms the bulk of the experiment design and includes your independent and dependent variables, the formal experimental design (e.g., a 2 x 3 mixed factorial (split-plot) design, more specifically a 2 levels of expertise (between subjects) x 3 interfaces (within subjects) design), the experimental tasks that your subjects will carry out, specific details on how you will measure the dependent variables, and the statistical analysis you expect to carry out.
Create all the study instruments you will need. These will likely include questionnaires, interview questions, observation sheets, and the script you will follow so that all subjects get identical instructions.
Every experiment has limitations. You need to carefully think through the limitations of your chosen design, in particular with respect to different kinds of validity that were discussed in class.
1.2 Report the experiment method: This includes a detailed description of the following components, that should appear in the order given below unless you have good reason to adjust:
In an appendix, include: each of your study instruments, , your script, and a revised consent form. If you need more space to clarify the tasks than available in the main body of the report, such as showing screen captures of your system, or a competitor’s system against which you will be evaluating your system, include those in the appendix as well.
IMPORTANT: In terms of the level of detail required for the experiment deliverables, you should be able to give a classmate from another team your experiment description and materials and they should be able to carry out your experiment.
Length: up to approximately 3 pages (+ appendix for all supplemental experiment materials)
2. Medium Fidelity Prototype
Complete your implementation of your medium fidelity prototype.
2.1 Prototype demonstration: Document your final interface prototype implementation, using screenshots/screen capture, photos, or the equivalent for your project. Include explanatory/descriptive captions for figures, or narration for video, as appropriate. Where appropriate, justify design elements with respect to what you have learned about human abilities and limitations.
Length: There is no specific word limit for this part. However, please avoid excessive length and concentrate on conveying key aspects. If in doubt, consult course staff.
Presentation and report:
As with the previous milestones, your team will briefly present your work on this milestone in class: 5 minutes per team. Your presentation should focus mainly on the experiment design (~4 min), but in order to ground that, you should first very briefly present the status medium-fi prototype (~1 min). You will have the opportunity in the next design review class to present the prototype more fully.
For your report, include an appendix that identifies each team member’s contributions. You many optionally include additional appendices for content that does not fit into the main sections of the report.
Teams need to submit two pdfs — a pdf of the presentation, and a pdf of their report — in one single post to Canvas. Please give your pdf files name “Test_1-report-<team name>.pdf” and “Test_1-presentation-<team name>.pdf”.