Paper Prototypes & User Testing

Posted: Friday, October 10
Deadline: Tuesday, October 28 (before classtime)
Point Total: This assignment is worth approximately 11% of your project grade (4.5% overall).

Assignment Overview
Your assignment is to convert a subset of your more promising sketches from the last assignment into paper prototypes and to test them with four users. You will create and test two different versions of your prototypes. As I've noted before in class, comparing two designs is far easier/better than evaluating one single design in isolation. For example, it's easier for a user to express preference between one design over another rather than articulating, exactly, why they don't like a single design.

Following from the IRB protocols we discussed in class, your participants must sign an informed consent document before taking part in your test. With the participant's permission, you should video record all sessions. This can be done with any type of recording device from a DSLR to a mobile phone camera.

Your report should include figures of your paper prototypes, a description of your study method, and a breakdown of your primary findings from the user testing. The video recordings will be used later in the semester for TA10 (so do not lose them).

What to Do

  1. To start, watch the Nielsen Norman Paper Prototyping: A How-To Video. This will help prepare you for this project assignment and the next one as well. In addition, prototyping--and, specifically, rapid prototyping--is perhaps one of the most valuable techniques in HCI/design. So, it's worth spending some time on the readings and, of course, applying the prototyping concepts in your projects.

  2. Iterate and refine the three primary tasks that users should be able to accomplish with your application (based on learnings/reflections from the last assignment).

  3. Then, transition to the first core part of the assignment: riff, iterate, and create two different paper prototypes for the three primary tasks. That is, you must create "Paper Prototype #1" that allows your users to accomplish the three tasks one way and "Paper Prototype #2" that allows your users to accomplish the three primary tasks another way. The paper prototypes should be functionally different so that your users can compare and contrast their experiences with both of them. So, for example, the two prototypes should represent the tasks in fundamentally different ways. Remember, the focus here is not on aesthetics/beauty but rather on understandability, usability, approachability, and, to some degree, layout, widget type, etc.

  4. Once, you've created the two paper prototype designs, beta test them with members of your team and make requisite changes. This "eating your own dog food" is a good way to catch errors before investing time in testing with actual users.

  5. Now you're ready for real user testing (the second core part of the assignment). Recruit four independent users to test out your paper prototypes. These users cannot be members of the class; however, they can be members of Dr. Vibha Sazawal's section. Each user testing session must be done in isolation (that is, you cannot have more than one user testing at a time) with at least two experimenters present. For the testing session, you should follow this protocol:
    1. First, download and modify this IRB "informed consent" template to fit your project [link]. At the beginning of the user testing session, read the "Purpose of this Study" section of the consent form out loud to your participants. This should be done consistently for each participant. Then, give your participants a chance to read the entire consent form, ask questions, and, if they agree to participate, have them sign the form. If they do not agree to participate, simply wish them a nice day and recruit another participant (it can be slightly awkward but this happens!). If they do agree to participate, provide a copy of the form and take the signed copy for yourself (please scan in the signed consent form and include it in your report appendix).

    2. After the informed consent process, you can begin user testing the prototypes. Follow the method described in the Nielsen Norman "Paper Prototyping: A How-To Video." Because you have two different prototype designs for your tasks, you should fully test one prototype for each of the three tasks before testing the other prototype. Because all subjects are exposed to all conditions, this is a "within subjects" study design. Consequently, you need to counterbalance your prototype conditions. In this case, two users should test Prototype #1 then Prototype #2 and the other two users should test Prototype #2 then Prototype #1. The condition order for all four participants should be set before you test your first participant.

    3. As the user is interacting with your paper prototypes, ask them to "think-aloud." One experimenter should record notes about how the participant was using the prototype, the problems/successes encountered, and comments made during the testing sessions. Scan in and include these raw notes in your appendix for each user testing session. I would like each individual student to serve at least once as a "observer/recorder" and at least once as the experimenter running the session.

    4. After both prototypes have been evaluated, provide your participant with a short paper survey with a few questions asking about which prototype the user liked better and why as well as questions seeking participant ideas for improvement. Scan in and include these responses in your appendix for each user testing session.

  6. After user testing, analyze your collected data (e.g., observational notes, post-study survey data). As you have only four users, it does not make sense to perform statistical analyses but you can report the raw data (e.g., in tables or graphs), which should indicate trends (or the lack thereof).

  7. Finally, you need to write up your report. See below.

The Report (126 pts)

Note: we will grade the report out of 126 points; however, the grade used in Canvas will be out of 46 (the % breakdown will be the same).

Please use the following full section headers (style Headings 1) in your report; it makes it easier for us to grade. So, for example, you should literally have headings with Section 1: Abstract and Section 2: Task Descriptions in your report.

Top of Report (5 pts)
At the top of the report, please include a title (centered and bold), a tag line, and a listing of each team member. For each person listed, include 1-2 sentences on their primary role/accomplishments on this assignment. (5 pts).

At the top of the report, please include a title (centered and bold) followed by the names of each team member.

Section 1: Abstract (10 pts)
The abstract should provide a 5-7 sentence overview of your report including: (i) a description of your application, (ii) a description of your three primary tasks, (iii) a high-level description of the two paper-prototypes and how they differ, (iv) a description of your evaluation method, and (v) a summary of your primary findings.

For grading, you will receive up to:
  • 2 pts for each of the above points (2 x 5 = 10 pts total)
  • -------
  • 10 pts total

Section 2: Task Descriptions (20 pts)

Describe your three primary tasks (iterate from previous incarnations). This should be a four paragraph section. The intro paragraph summarizes the three tasks. Each subsequent paragraph should start with a bolded name of the task and a more lengthy description. If the tasks changed from the previous assignment, please describe these changes and why they were made (e.g., was it based on your own design reflections, was it based on the experiences of storyboarding, was it based on feedback from in-class critiques, etc.).

For grading, you will receive up to:
  • 5 pts for your introductory content
  • 5 pts per task description (15 points total)
  • ---------
  • 20 pts total

Section 3: The Paper Prototypes (41 pts)

The introductory paragraph to this section should introduce the two paper-prototype designs and, at a high-level, the primary similarities and differences between each. Why are they different? Then, there should be two sub-sections (one sub-section for each paper-prototype design). Name the sub-sections Paper Prototype Design #1: <name> and Paper Prototype Design #2: <name>. In each of these sub-sections, describe the paper prototypes for the three primary tasks. Again, make it easy for your readers to understand the most important points and, in particular, how you made different design choices with each design. Obviously, you will want to include as many inline figures as is appropriate to helping describe your paper prototypes. You will include scans/pictures of all paper prototypes for the two designs in the Appendix.

For grading, you will receive up to:
  • 5 pts for introductory content
  • 6 pts for each implemented task in the two designs (3 tasks x 2 designs = 36 pts total)
  • -------
  • 41 pts total

Section 4: Testing the Paper Prototypes (20 pts)

This section should have four sub-sections focused on:
  • Pilot testing the paper-prototypes among your group and the resulting changes that were made either to the study protocol or to the paper prototypes themselves
  • Participant recruitment and demographics, which includes a description of how you recruited participants and demographics relevant to your project. Justify why these participants were appropriate for your project.
  • Study method, which details how you performed your study including the study protocol, the location, the length of the study, and a description of the data collection instruments (e.g., the video recording setup, the post-study survey).
  • Analysis, which describes how you analyzed the data you collected.

For grading, you will receive up to:
  • 5 pts per sub-section
  • ------
  • 20 pts total

Section 5: Results from User Testing (10 pts)

This section should detail the primary results from user testing.

Section 6: Learnings from this Project Assignment (10 pts)

This section should offer reflections on what you learned from this project assignment and what you would do differently if you had to do the assignment again.

Section 7: Appendix (10 pts)

The appendix should include:
  • A scan/pictures of all paper prototypes created for the two designs that cover the three primary tasks
  • A scan of the four signed informed consent forms
  • Raw notes from four user testing sessions (notes should be clearly marked with a timestamp and session number)
  • A scan of the post-study paper survey responses (these should also have session numbers)