UI Experiment & Analysis

Posted: Saturday, November 15
Due: Tuesday, December 2, 7AM
Point Total: This assignment is worth approximately 5% of your overall grade
This assignment is to be completed with one CMSC434 partner of your choice.

Assignment Overview

In this assignment, you will take on the role of HCI experimentalist and run a small UI experiment and analysis. Along with a team of graduate and undergraduate students, Professor Findlater and I have built a web-based platform for evaluating the accessibility of computers and input devices. In this assignment, we will use this test platform as a way for you to study the efficiency and error rate of various interaction techniques (e.g., point-and-click, crossing, dragging,) and input devices (e.g., mouse vs. touchpad). Ordinarily, you would run such an experiment with a large, diverse population; however, as a learning exercise and to keep the workload low, you only need to test yourself and your partner.

The overarching goals of this assignment are to give you tangible experience running a quantitative experiment of user interaction/performance, analyzing the experimental data, modeling Fitts' law, drawing insights from your analysis, and establishing/testing hypotheses.

Assignment Parts

1. Select two input devices. First, you and your partner will select two input devices to test. For example, you could test a Macbook Air touchpad vs. a touchpad on an inexpensive laptop, an external mouse vs. an IBM Thinkpad red nub pointing stick, an external mouse vs. touch input on an iPad, a list of optional input devices follows in a sub-section below--feel free to brainstorm and test others). Note: the testbed may not work with smaller screens like the iPhone 4s/5s (though it might work with the iPhone 6 and 6+).

2. Create hypotheses. Once you've decided on the input devices, you and your partner should come up with and write down two lucid, testable hypotheses about the expected outcomes of your experiment. For example, H1: The crossing condition will perform faster than the dragging condition. As a second example, if you decided to examine an external mouse and a Mac touchpad as your input devices, you might state H2: The external mouse will have fewer overall errors across interaction techniques than the Mac touchpad. It's important that you state these hypotheses before you conduct the experiment.

3. Run the experiment. Now it's time to actually run the experiment. You and your partner need to use the exact same configuration for both of your chosen input devices. You should limit distractions and perform the experimental tasks quickly but accurately. To run the UI testbed, visit this website: http://access.umiacs.umd.edu/cmsc434/. The experiment should take roughly 20 minutes per person for each device. Counterbalance which input device you each use first, so one of you completes the experiment with device 1 then device 2, while the other one uses device 2 first then device 1.

A few important notes before you begin:
  • You need to enable popups for the experiment website (access.umiacs.umd.edu). Why? Because we've implemented a small hack to provide you with the experimental log data so you can perform your analysis. Obviously, the actual website that we use with real participants does not offer this feature.

  • After you complete each task, a new tab (or window) will open (depending on your web browser) as long as you have properly enabled pop-ups. You need to copy and paste each task's data into a separate plaintext file. Save the files with the following structure: e.g., <participant_device_taskname.csv>. For example, 1234CMSC434_MacTouchpad_Crossing.csv, 1234CMSC434_MacTouchpad_Dragging.csv

  • Update (11/30): Some of you are reporting NaNs in your data (this is an infrequent error but frequent enough that I wanted to write this update). We've never seen this error before and believe it's coming from the hack we quickly wrote to give you access to your data via the pop-up. If this happens to you, just continue the experiment as normal and copy the data into a CSV as you otherwise would. If the NaNs occurred during the Pointing task, then you should calculate the Fitts' Fit on the Dragging task data. You must note that this error occurred in your report. Alternatively, if you are not satisfied with having NaNs, you could redo the task that the NaN occurred for (however, this is not required). Update 2: it appears that the NaN occurs if you double click on a target during the Pointing task. Once you do this, there is some faulty data that gets into the dataset and causes an entire array of calculations to come back NaN. So, to avoid the problem, just do not double click during the Pointing task (and you shouldn't have to anyway).

  • Your username must be of the format [last 4 digits of UID] + CMSC434. For example: "1234CMSC434". Use the same username for both input devices.

  • The tutorial videos don't perfectly match the tasks (we need to redo them), so you can skip the short videos

  • You do not need to sign up for the Amazon gift card drawing at the end of the experiment. We are not actually using your data in our research (for this, we would need IRB permission)--we will only be using the data to validate that you actually did the assignment.

  • Update (11/27): As I mentioned in lecture on Thursday, we will be giving out bonus points for: (i) the fastest overall speed regardless of condition and input device (the error rates must be less than 4%); (ii) the fastest overall speed using a touchscreen (again, error rates must be less than 4%); (iii) the use of the most zany input device; (iv) the first submission of the assignment; (v) the best Fitts' r-squared fit; and (vi) the most devices tested.

4. Analyze the data. After you and your partner complete the tasks, you should have a total of 16 files (4 files per task x 2 people x 2 input devices). Now, you must analyze this data and see if there is support for your hypotheses. Open each CSV file in Microsoft Excel and do the following:
  • Calculate Descriptive Stats: add five new rows called Mean, Median, Stdev, Min, and Max and calculate these stats for the Mean Elapsed Time and Num of Error Trials. You can calculate these automatically using Excel formulas.

  • For the Pointing Task Only, Calculate Fitts' Fit: insert a new column called "Index of Difficulty" for the four pointing task files (2 people x 2 input devices). Calculate the Index of Difficulty (log base2 of A/W) for each condition (a condition here is an Amplitude/Width combination). Graph a scatterplot of Index of Difficulty vs. Mean Elapsed Time--the resulting plot should look like a line. Right-click on a datapoint and select "Add trendline..." In the pop-up dialog, select "Display equation on chart" and "Display R-squared value on chart." The R-squared value is the Fitts' fit; higher is better.

  • Save the file as with the format .xls or .xslx. You will upload these files in Canvas.

5. Add data to shared spreadsheet. Open this Google Spreadsheet, which will store all the data for class. For each of the xls files, copy the condition rows into Tab 1 "PerCondition" and the average data along with the trendline equation and R-squared fit value into Tab 2 "Aggregate."

6. Write a brief report. Finally, you need to write a brief report on your experience, the input devices you selected, and whether there is support for your hypotheses. You need to upload this report to canvas. The report should be 1-2 pages maximum. Your report must include:
  • A detailed description of your two pointing devices. This should be sufficient to allow the reader to reproduce your test exactly. For example, if using a mouse, what brand and model number? What mouse settings did you use on the computer? What browser did you use to run the test? If using a tablet, what exact screen resolution and screen size (diagonal)? (You can often find these by searching Google for "screen resolution" -- e.g., Apple says the "Display" on the iPhone 5s is: "4-inch (diagonal) Retina display / 1136-by-640 resolution / 326 ppi"). Please also include photos of your actual devices in your report.

  • A detailed description of the environment in which you did the test. In particular, were the participants (you) seated at a desk, standing, at a table? If a phone or tablet, was it on the table or in their hands? (You could even test both if you want and see if there is a difference.) Also include the order of the devices used for each user. Please include a picture (or pictures if more than one) of the environment in which you did the test.

  • Results. Your results table should include the following columns: participant name, computer device, input device, task, total conditions, total trials, average elapsed time across conditions, and average error across conditions. For the pointing tasks, also include the trendline Fitts' equation and R-squared value. I then want you to describe the results in prose and answer questions like: Did one participant perform better than the other? If so, describe. Did one device perform better than the other? Again, describe. Did one condition seem to perform better than the other? Did you find support for your hypotheses? Why or why not? Please insert two Fitts' graphs: one for each participant using the same pointing device for the pointing condition. As an assignment, I am not asking you to perform or report on all analyses you would typically find in a UI experiment, including:
    • A participant table that includes demographic information like age, gender, motor ability, etc.

    • In order to truly test your hypotheses, you would need to perform inferential statistics not just descriptive statistics. In addition, you would need more participants. For the inferential statistics, for example, you could run a paired t-test to compare the mean elapsed time between the two input devices for each task.

Input Device Ideas

You are free to use any input device that you can get to work with our software and that also allows you to copy the results out of the webpage. Some ideas are listed below:
  • Mouse
    • Optical mouse vs. "ball" mouse
    • Various "gain ratios" on mouse settings
    • Laptop touchpad from various laptops:
      • Macintosh
      • Windows laptop
    • IBM Trackpoint on Thinkpad laptops (red nub)
    • Trackball
  • Touch with finger:
    • iPad (of various sizes) - touch with finger
    • Android tablet (of various sizes) - touch with finger
    • Windows Surface Tablet
    • Note: currently we do not officially support mobile phones so try at your own risk.
  • Touch with stylus:
    • Smartphone / tablet - touch with stylus, like the Samsung Galaxy Note
  • Game controller connected to a PC to control the cursor
  • etc. -- what other pointing devices can you get access to?