Sixth project milestone: Test – Part 2

This is the second part of the Test stage of Designing Thinking. The goal here is to pilot test your experiment, adjust the experimental design as necessary, and finally run the experiment, i.e., collect all the data that will be analyzed in the next milestone (Test – Part 3).

Step 1: Pilot test your experiment

Run your experiment with 2 participants to make sure there are no major glitches in the experiment protocol. In particular, pay attention to overall length of session, clarity of instructions given to the participants, feasibility and appropriateness of the tasks, usefulness and utility of your other study instruments (questionnaires, interview scripts, coding sheets), your ability to operate the video equipment (if you are using video for data collection), and the integrity of data collection overall (e.g., if you are using software logs, are they outputting the data correctly?).

Note: you cannot use the data from these two pilot subjects in the analysis of the full experiment, nor re-run them as full participants in the next step.

1.1 Report Pilot Test and Changes to the Experiment: Briefly summarize what you learned from your pilot test, and any adjustments you made to the experiment protocol as a result of the pilot test. For each of the evaluation design components, clearly highlight any differences from the more comprehensive evaluation plan provided in the Test – Part 1 milestone. It should be very easy for the course staff to determine what has changed. In an Appendix, include any revised evaluation instruments or consent forms from what you have submitted previously. This should be a blank copy of whatever the pilot participants saw – e.g. the actual questionnaire, rather than just a list of the questions that were asked.

Length: up to one page (+ Appendices)

Step 2: Conduct the full experiment

Next you will run the full experiment (adjusted from the pilot study as necessary). Based on your Test – Part I deliverable and feedback from the course staff, you should have a clear idea how many participants is reasonable for your particular experiment design. Generally, the minimum is 10, but this number will vary depending on the project.

2.1 Describe your Participants: Briefly describe the people that you actually used as participants in such a way that the reader can assess their representativeness for the interactive system you have designed. (As with earlier milestones, you should not name the actual participants.)

Length: a medium paragraph (possibly plus a table)

2.2 Report a Summary of the Data Collected: Briefly describe the full set of raw data you have collected and the format that it is in (e.g., “completed questionnaires, data transferred to a spreadsheet” or “observational notes taken, text file”).  In an appendix, include all of the raw data that is reasonable to include, such as the completed questionnaires, the observational notes. Do not include, for example, pages and pages of system log data.

Length: up to half a page (+ Appendices)

2.3 Reflect on the Experience of Running the Experiment: Briefly reflect on your experience — What was easier than you expected it to be? What was more difficult? What would you do differently if you were to run it again? etc.

Length: a medium paragraph

Step 3: Be prepared for a hands-on demonstration of your final prototype with the course staff during the working class given on the schedule page.

Course staff will meet with each team, one by one, to interact with their final prototype. If there are materials (e.g., script or set of tasks) that the instructors should follow while they step through the prototype, please have those available.

Step 4: Start data analysis

Although there is nothing to hand in for this step for the current milestone, we highly recommend that you start on Step 1 of the next milestone (Analyze the data-done by the group) in parallel with running the experiment. You are free to use any software that you want. At a minimum you should start to set up your data file(s) and figure out how to run the appropriate statistical tests.

Some tool options: Excel, however last we looked it could only do up to 2-factor ANOVA with one between-subjects variable, but that is likely sufficient for many experiment designs in this course; R is currently the tool taught to the CS444 students — it is much more powerful than Excel but has a higher learning curve; there are lots of online ANOVA calculators but we do not have any personal experience with them.

Report AND hardcopy deliverables:

For your report, include an appendix that identifies each team member’s contributions.

Teams need to submit a pdf of their report — in one single post to Canvas. Please give your pdf files name “Test_2-report-.pdf.

Hardcopy deliverables for the final class, please submit:

• Any hardcopy raw data (e.g., completed questionnaires) and completed consent forms from your project neatly assembled together (e.g., in an envelope or in a binder). [You should not have any hardcopy participant data/information left after this submission, for compliance with the ethics protocol.]
• Adherence to Ethics Protocol Form: all team members must sign and submit this hardcopy. (find it here: https://www.ugrad.cs.ubc.ca/~cs444/resources.html#ethics)

Rubrics: