conduct a user test using the Ethnographic Observation design process

1. You will conduct a user test using the Ethnographic Observation design process.
2. The results of the user test can lead to valuable evaluation and recommendations for
improving the quality of the product under evaluation.
3. This exercise will also give the student first-hand experience in assessing user interface
design through systematic and direct observation.
4. Conduct and report a user test using 5 participants
5. You will prepare a discussion of the user test process from conceptualization to reporting
results and making modifications to address issues users encountered.
6. The user test can be based on previous investigations of products or interfaces evaluated
in former experiments.
7. You will respond to the user interface evaluation by providing recommended
modifications to the evaluated user interface.
8. Consider the difficulties users experienced during the user interface evaluation as pseudo-
requirements to modify the interface.
9. Identify the requirements of the interface to improve its usability.
10. Use a high-fidelity prototype methodology to develop the modified user interface (i.e.
Photoshop, Gimp, Paint(?))
11. Consider yourself as a consultant in a situation where you are proposing to the user
interface owning company how to improve their user interface to maximize
marketability.
General Method
1. You will need 5 human subjects (participants) for this exercise.
2. Choose a hardware platform and software product that you can arrange for your
participants (one at a time, if necessary) to work through the major features of the
software product or user interface you have chosen to evaluate.
3. You will prepare a list of at least 5 and no more than 8 tasks that each subject will
perform with a brief description of each task to be performed on the system.
4. You may (if appropriate) prepare a flowchart showing the order tasks are to be performed
and other important events and sequences.
5. Your job, as the researcher, is to give each participant written and/or verbal directions on
how to complete the task; observe the subject working through the task; and record (on
paper) the sequence of events.
6. Upon collecting data from all invited participants, prepare a report describing the
evaluation process and the results of the usability evaluation.
7. A reminder to student researchers in HCI: It is the product that is being tested, not the
users!
Planning Stage
1. Identify the test goals, what are you trying to prove?
2. Describe what test method(s) you will use to reach test goals, the answer is up above but
please discuss Ethnographic Observation.

Page 2 of 5

3. Pre-test survey This process should give attention to:
a. User experience level/skills/capabilities
b. Education
c. Attitudes/willingness
d. Demographics (age, sex, language, etc.). Do not ask any intrusive information
about the individual.
e. User’s knowledge of the product
4. Create a task list of workable tasks that help the user evaluate the usability of the
interface.
a. Provide a general description of each task to be performed.
b. Describe what steps are in each task.
c. Distinguish interaction with other tasks.
d. Order and prioritize the tasks.
5. Create the scenario (test lab) needed to conduct the evaluation (The "test lab" can be your
office, home, or wherever you can set up the hardware and software.)
6. Task Observation form to determine which performance and subjective measurements to
take
Conducting the Test and Collecting Data
1. There are different strategies for structuring the evaluation for your participants.
a. have all participants work together at the same time (if you have access to
multiple workstations)
b. co-pair the participants at one time (if you have at least two workstations)
c. observe one participant at a time.
2. Whichever option is workable for you, your main role will be to initially describe the
evaluation procedures to the participant and RECORD their actions, do NOT video or
audio tape the participant.
3. It is recommended that you do not offer assistance to participants during the evaluation.
4. Ask them to tell you what they are doing and thinking (Speak out) as they work through
the task list, write down what they tell you, his may be important data, as later you will
try to analyze their reactions (and therefore the usability of the interface).
5. Explain and describe the procedures to the participant, including reasonable if simple
rationale, if these seem needed to justify (perhaps unfamiliar) procedures.
6. Record the participant’s actions during the usability evaluation. Recording possibilities
could include, but are not limited to:
a. Elapsed time spent on single tasks and through the overall evaluation.
b. Time of day and the date participant completed the evaluation.
c. The number of errors the participant encountered. (Remember the focus on
“errors” is on the interface design, not the user.)
d. The number of successes the participant gained.
e. How the user was able to recover from errors.
f. How often the user could not recover from errors, including processes various
users attempted in their recover attempts.
g. How often the user sought help through online help or written documentation.
h. The number of times the participant sought assistance from the researcher.
i. The nature of the usability problem encountered.

Page 3 of 5

j. The number of usability problems located during the evaluation period.
7. Be sure to follow good practices when working with your participants.
a. Treat them with respect; explain you are evaluating the system, not them.
b. Explain that they can stop the evaluation at any time, if they are uncomfortable,
without penalty.
c. Explain that their results will be reported, but without identifying information
about them and without publishing the results (this is an educational exercise
only)
d. Express your appreciation and thank them for their participation.
8. Post-test survey
Reporting the Data
1. Prepare a document that describes what you did and what you found. Include the entire
process (planning stage through collecting data) and the results of the usability
evaluation. A template will be provided as guidance.
2. You should be able to highlight unique events that occurred on the basis of the
participant’s performance and your systematic observation.
3. Identify the major variables associated with the usability of the product you have chosen
4. Discuss your results in relation to HTI concepts presented in the required texts and in
class discussions.
5. You must include recommended improvements to the user interface evaluated based on
the results of their usability evaluation.
6. Use high-fidelity prototype methodologies as appropriate to develop the modified user
interface.