Usability Evaluation Methods and Procedures ======================================================= Usability Evaluation Define usability; define purpose of the evaluation A description of the interface Users and their characteristics important to this evaluation Establish test goals and test plan What is going to be measured? Establish test tasks Establish test environment Data collection procedure inspection method -- heuristic evaluation, other? Stages of the test Results Observing, analyzing usability problems severity ratings subjective satisfaction ratings Recommendations for improvement *integrate literature into ALL aspects of the report ================================ Test Plan AND Report Format: "Typical sections" to include: INTRODUCTION Purpose Problem Statement/Test Objectives User profile (see section on subjects) About the product being tested METHOD (see section on Method) Method (test design) Description of test procedure Task list Test environment/equipment Test monitor role Evaluation measures (data to be collected) RESULTS Report contents and presentation DISCUSSION What does the data mean? RECOMMENDATIONS List the changes that should be made REFERENCE LIST APA format -- author by alpha APPENDIX OR APPENDICES Materials given to subjects Questionnaires Statistical output Other ========================== PURPOSE: What is the rationale for the test? What is the general goal? A narrative description with background information as to how this idea came about. The purpose of the usability evaluation is to .............. The purpose of this project was to begin investigation into the usability of multimedia software packages for high school students studying the French Revolution. The purpose of this usability evaluation is to test a revised graphical user interface to a computer-automation system which controls a chemical reactor. ================= PROBLEM STATEMENT/TEST OBJECTIVES Problem statement/test objectives The more specific the better. EXAMPLES: The product may contain major usability flaws that prevent completion of common and special tasks. Novice and intermittent users may have problems adapting to advanced features of the software. Is there an appropriate balance of ease of use and ease of learning? ======================= PLANNING STAGE Is the test formative or summative? If summative, then a measurement test is a typical method. Test Plans Before the test, the following should be determined: -- The goal of the test: What do you want to achieve? -- Where and when will the test take place? -- How long is each test session expected to take? -- What computer support will be needed for the test? -- What software needs to be ready for the test? -- What should the state of the system be at the start of the test? -- What should the system/network load and response times be? -- Who will serve as experimenters for the test? -- What test tasks will the users be asked to perform? -- What criteria will be used to determine when the users have finished each of the test tasks correctly? -- What user aids (manuals, online help, etc.) will be made available to the test users? -- To what extent will the experimenter be allowed to help the users during the test? -- What data is going to be collected, and how will it be analyzed once it has been collected? -- What will the criterion be for pronouncing the interface a success? ====================== TEST BUDGET Typical cost elements of user test budget are: -- usability specialists to plan, run, and analyze the test: out-of-pocket expense if consultants are used -- administrative assistants to schedule test users, enter data, etc. -- software developers to modify the code to include data collection or other desired test customization -- the test users' time: out-of-pocket expense if outside people are hired for the test -- computers used during testing and during analysis -- the usability lab or other room used for the test -- video tapes and other consumables: out-of-pocket expenses =========================== ROLES IN TESTING Test Subjects: - attempt to perform task - share surprises and frustrations Experimenter/Facilitator: - orchestrates test - manages communication with subjects Observers: - observe - watch for problems and opportunities Chinese proverb: "Problems are opportunities in work clothes." ================================ WHERE'S THE LAB? - observation is the main method of testing - the most benefit comes from developers watching users attempt to use the product - video is cheap for archival purposes ============== FINDING SUBJECTS: - subjects should match experience and background of users - involve average users, avoid outliers for small samples - user groups, community services, and temp. services are good sources - classified advertising - corporate training - reward subjects for their time =========================== METHOD (TEST DESIGN) A detailed description of how you are going to carry out the research with the participants, and how the test session will unfold Design Issues: Between-Subjects versus Within-Subjects Testing between-subjects: each test user participates in one single test session Sending Messages Editing Messages User 1 User 2 User 3 User 4 within-subjects -- each test user participates in test session of system one and system two Sending Messages Editing Messages User 1 User 1 User 2 User 2 User 3 User 3 User 4 User 4 testing multiple product versions Version A Version B team#1 n=4 n=4 team#2 n=4 n=4 randomized order of module completion -- counterbalancing can limit the effects of learning transfer Subject Module Sequence User 1 A,B,C User 2 B, A, C User 3 C, B, A User 4 A, C, B Other things regarding experiment design random assignment -- reduces experimenter bias employ an adequate number of subjects conduct a pilot test make the testing environment as realistic as possible ==================== DESCRIPTION OF THE TEST PROCEDURE Provide a FULL description. Why? -- the description enables others to understand and visualize what will happen so that they can comment and make suggestions accordingly. -- as an experimenter, it allows one to focus on what has to be done -- it allows multiple test monitors to conduct the test in the same manner to insure reliability of the test Things to do: Describe the evaluation method -- Is it formative or summative? Is it heuristic evaluation? What heuristics were applied? Is it think aloud evaluation? Describe the process of how this type evaluation would be carried out. Describe how subjects are greeted, the orientation process. Provide example computer screens -- what does the prototype or parts of it look like? Describe the performance test itself. What do the subjects have to do under what conditions (time frame, sequence of tasks, etc.) Describe the process of collecting data -- how did you capture error rates, time stamps (via stopwatch?), opinions, etc. Describe the process of subject debriefing ================ TASK LIST When choosing a sample of tasks, exercise as many of the most important aspects of the product as possible and address all test objectives. Common methods for prioritizing: prioritize by sequence prioritize by frequency prioritize by criticality prioritze by vulnerability In regard to tasks, be sure to: provide a general description of each task to be performed describe what steps are in each task distinguish interaction with other tasks if necessary identify if the task is an individual or group task ========================= EXAMPLE TASK LIST 1. Start the Windows Write program from the Windows Program Manager 2. Type in two paragraphs from a sample page 3. Type in two more paragraphs from a second sample page 4. Cut and paste a block of text a. select a specified block of text b. cut the selected block of text from the document c. paste the clock at a different location 5. Center and boldface the heading a. select the heading text block b. center the heading c. boldface the heading 6. Change the first paragraph to italics a. select the first paragraph as a block of text b. change the font to italics 7. Save the file to drive a: 8. Print the file 9. Exit the word processor. These tasks were selected in order to gather the following information: -- the number of tasks completed successfully -- the number of errors during completion of task -- the time required to complete each task -- the number of requests for help -- the thoughts of each subject during the test, as evidenced by comments made during the test ============ USER PROFILE IDENTIFYING TEST SUBJECTS: Important Information Usually Generated Through A Questionnaire user experience level/skills/capabilities education attitude/willingness to participate demographics (age, sex, language, etc) user satisfaction of the product (outcomes) Other requirements as appropriate: Determine: speed required of user skill required of user physical capability of user responsibility required of user ease of use considered for the user the user's potential for misuse or error Get written consent for participation ============================ PRESENTING RESULTS Present data in narrative format -- describe the completion of test by each subject if sample is small. Highlight important observations. Summation of Test Results for User #1 Summation of Test Results for User #2 Summation of Test Results for User #3 Summation of Test Results for Users #1, #2, and #3 Present data in tabular format -- show error rates, time completion rates, means and sd's of subjective preference ratings Table 1 -- Task Times Task User#1 User#2 User#3 1 2 3 Table 2 -- Number of Errors Made Task User#1 User#2 User#3 1 2 3 ======================= DISCUSSION Narrative discussion of the results and their connection to the usability test goals and objectives. What are the basic conclusions of the test? RECOMMENDATIONS Narrative format, plus list of actual changes to be made =======================