Current Issue
Usability Testing in Chinese Industries - 1 April 2005
by Xiaowei Yuan and Xiaolan Fu
Read this article in Chinese (translated by Sean Liu)
Published with Beijing ISAR User Interface Design Co. ’s permission
Abstract
Nowadays, usability testing has become one of the most important and popular methods to test the usability and acceptability of products. Usability testing came from the West, and spread to China at the end of the 20th century. Since then usability testing has made effect on Chinese industries. This paper will mainly introduce the research and application of usability testing in Chinese industries
Key words: usability testing, Chinese industries, ISAR User Interface Design, cell phone
1. INTRODUCTION
Usability testing can be best described as putting a product on a trial-market. In the test, participants represent the target group of product users and are confronted with carefully designed scenarios that occur in the actual use of a product or program. For example, project engineers configure parameters for chemical plants using a new version of a programming tool, or doctors make clinical diagnoses with the help of a prototype of a computerized tomography scanner. Because these scenarios are very similar to the users’ real-life situations, the test offers the opportunity to discover the problems of products and to further improve product at the early deployment stages. Therefore, usability testing enables planners to optimize the product specifications.
A usability test delivers detailed recommendations for improvement and further creative development of the products. In fact, by employing sophisticated psychological observation and communication methods in a professionally organized test environment, few users (e.g., 5 or 6 users) are sufficient to reveal 75% of the improvement potential. The sophisticated methodological evaluation of the critical acceptance factors also enables people to strategically enhance the market attractiveness of their products. From the reports of usability testing, managers, marketing consultants, and development specialists obtain the experiences of the customers’ reactions to products, therefore, they can better develop product strategies and improvement ideas.
2. Application of Usability Testing in China
Usability testing as a research method was introduced to China from the West at the end of the 20th century. It has now been used in Chinese industries and has achieved great success.
2.1 Application in Foreign Enterprises
In Chinese industries, usability testing has mainly been introduced by the foreign enterprises. In the Chinese branches of large-scale international enterprises, such as Siemens and Microsoft, laboratories of usability testing have been established. They conduct usability testing research and carry out actual tests by using their research achievement.
2.2 Application in Chinese Domestic Enterprises
Being influenced by the western modernized enterprises, Chinese domestic large-scale enterprises, including Haier and Legend, has begun to pay attention to usability testing and its roles. They tried to use the method of usability testing in the industrial practice. For example, usability testing was applied in the compilation of the user’s manual of “Tianjiao and Tianrui double-mode computer” produced by Legend Group. By usability testing, problems were found in the compilation of the user’s manual, and a better manual for the product was finally produced.
2.3 The Birth of a Professional Company
When the Chinese industries were increasing their understanding of usability testing, with professional technology and experience, Prof. Dr. Yuan, who was the leader of User Interface Design of Siemens China, established the first Chinese professional usability testing company in Beijing, ISAR User Interface Design Co. Ltd, in 2002. The birth of the professional company means that the operation research of Chinese industries has stepped into a new phase. ISAR User Interface Design is an industry oriented support center for usability engineering in China, including user analysis, user interface design, and usability inspection. It devotes to increasing the awareness of usability engineering, conducting application oriented research on human-computer interaction, and providing services for its applications in Chinese industry. ISAR User Interface Design has successfully conducted several HCI projects derived from Chinese industry. It has also helped Chinese domestic big enterprises, such as Legend and Haier, establish their own Usability Centers. These centers play important roles in evaluating the usability of products.
3. Usability Testing of Mobile Phone
At present, in the design of information technology product, it is a new trend to shift from technique design to usability design. In the competitive cell phone market in China, the usability of cell phone products is becoming the most important criteria of customer selection. Therefore, usability is a critical factor to develop cell phone’s application technologies and increase cell phone’s marketing competition. It is highly required to do a systematic study on cell phone’s usability and its potential deign development.
3.1 Purposes of Testing
For benchmarking, the usability testing has been done on Xiaxin A8, Motorola V60, and Samsung A308 for the following two purposes: 1) To do comparative usability analysis on three type cell phone products, and find their weak points in terms of usability and give them improvement expertise; 2) To standardize basic usability testing routine.
3.2 Designing the Test Scenarios
The design of the scenarios is the script of the usability testing. In order to make the testers perform well and to reflect what consumers are really thinking of when using the product, this design of script should also contain the process of the task.
According to the manual of the three brands of cell phones as well as the engineering design of all parts and the function keys of the machine, we divided the whole design of the scenarios into four parts: 1) the installation of battery, 2) the settings of the mobile phone, 3) the settings of the communication, and 4) the settings of the short message service. In the designed scenarios, participants were asked to complete all the tasks in a relaxing atmosphere. A two-part questionnaire was designed too. The first part is about personal background, such as sex, age, educational level, profession, the experience of former use of mobile phone, and the brand used. The second part is about the satisfaction degree of each operation.
3.3 Participant Selection and Test
In the aspects such as the knowledge about the tested cell phones and the usage skills, the participants were consistent with the consumer characteristic. In other words, they were embody of the ordinary consumers. In this testing, seven end users were recruited as participants. After completing the planning, there was the rehearsal of the test. That is ‘the test of the test’, which is a part of the whole usage test. It would help us to evaluate whether the scenarios design is reasonable and whether the time is sufficient.
The method of Thinking Aloud was used in all tests.
3.4 Evaluation and Analysis
3.4.1 Description of Errors
After the test, we made statistical analysis with the problems occurred in the test, and finally classified the errors into 5 types:
- Project faults (PF): User’s target is always erroneous when completing the tasks.
- General faults (GF): Design violates users’ general knowledge and experiences.
- Operation faults (OF): User has the right intention, but always makes operational error.
- Sense faults (SF): Misconception due to incorrect translation/interoperation of the information or symbols (e.g., color, form, size, position etc.).
- Interpretative faults (IF): Incorrect interpretation of information.
The error degree was calculated according to the number of the faults founded in the six formal tests. The error degrees were different from A to D: the error degree was A if it occurred in 4 or more tests, B if in 3 tests, C if in 2 tests, and D if in 1 test.
Besides, we made some detailed analysis mainly with Error A and Error B in five aspects of the applicable principle: easy-learning, efficiency, consistency, fault toleration, and satisfaction. The examples of Error A were showed in Table 1.
Table 1. The Examples of Error A
3.4.2 Satisfaction of End Users
A 10-point scale was used in the questionnaire to test the satisfaction of end users to the cell phone products at the following 16 aspects:
- Battery Installation
- Rings Setting
- Calling Forward
- Time Setting
- Alarm Setting
- Random Key and Open Cover Lid to Receive
- Calling Waiting and Shifting
- Save Phone Number
- Directory Lookup
- Input Method Shifting
- Punctuation mark and Symbol Shifting
- Text Input
- Short Message Viewing
- Number Extracting from Short Message
- Key Operations
- Hand Satisfaction degree
Figure 1 showed the users’ satisfaction to three cell phone products, Xiaxin A8, Motorola V60, and Samsung A308. Obviously, Samung A308 got the highest satisfaction.
Finally, based on the analysis, we also provided our suggestions on the structure and interactive design.
4. Summary
Usability testing has become one of the most important and popular methods for testing the usability and acceptability of products in China. ISAR User Interface Design is the first industry oriented company for usability engineering in China. ISAR has successfully conducted several HCI projects derived from Chinese industry. In a case study, 3 brands of cellular phone, Samsung, Motorola, and Xiaxin, were selected. Seven participants were required to use these three cellular phones to perform several tasks. The results of the test showed the shortcomings of the three cellular phones on usability as well as the user?ˉs satisfaction degree of every function. The improvement suggestion was provided finally.
REFERENCES
Mayhew, D. J. (1999). The usability engineering lifecycle. Morgen Kaufmann Publisher, Inc.
Mandel, T. (1997). The elements of user interface design. John Wiley & Song, Inc.
Nielsen , J. (1993). Usability engineering. Morgen Kaufmann Publisher, Inc.
Rubin, J. (1994). Handbook of usability testing. John Wiley & Song, Inc.
Siemens CT IC 7. (1999). Leitfaden zur organisation und durchf1hrung von usabilitytests. Munich.
Wiklund, M. E. (1994). Usability in practice. Academic Press, Inc.
Comments made
Possible Related Articles:
Latest Comments