iPhone Evaluation Report - 24 April 2008

Read this article in Chinese (translated by Maggie Shang, proof read by Christina Li)

This article is edited by uiGarden from the three user testing reports that published by User Centric) 

User Centric, a privately held usability consulting firm based in Chicago, has evaluated the long-term usability and user experience of the iPhone in 2007.

General usage

Their first study, conducted six days after the release of the iPhone, ran from July 5 to July 6, 2007 and was used to assess the usability of conventional and novel user tasks, such as:

• ease of web browsing
• use of the soft keyboard
• visual voicemail
• different types of text messaging
• placing and receiving calls
• using dynamic map interfaces

Bearing in mind that the time of the test meant that iPhone owners had only owned their phone for a short time, the data still revealed some interesting points.

For example, the ability to receive a call while listening to music proved to be an intuitively designed experience with 100% of the participants completing this successfully on their first attempt. Receiving a call, whilst typing an SMS message was successfully completed by 90% of the participant on their first try.  Additionally, taking a picture and emailing it was also a feature that garnered a high success rate on first attempt (90%), even though 40% of owners had yet to take a picture with their iPhone and 80% of owners had never emailed a picture from their iPhone.

Saving a contact, using visual voicemail and switching between portrait and landscape views proved to be easy and engaging features as well.

Interestingly, the perception from users about entering text on the iPhone was that the soft keyboard (the GUI that appears with a QWERTY type keyboard since the iPhone does not have physical buttons for text entry) was much easier than entering text on their previous phone. The owners of the iPhone were able to type 24 words per minute on their iPhone, yet averaged 20 words per minute on their previous phones.

However, weaknesses in the iPhone user experience were also uncovered. Participants found the forced use of a vertical keyboard and the lack of visibility for editing the middle of a word or sentence to be difficult and, at times, frustrating. At the same time, they lamented that landscape view for text entry was only available in the Safari web browser. “The keyboard is bigger (in horizontal view) and you don’t hit the wrong button.”

The pinch, a new use interface paradigm that the iPhone’s multi-touch interface has introduced to users, proved difficult to master with Google Maps. Additionally, participants were often frustrated with their web browsing experience due mainly to the speed of the Edge network and the lack of Flash and Java capabilities.

In conclusion, participants were asked to rate their first time usage of the iPhone for key tasks and the average rating, for one such task (changing personal settings) was 4.7 out of 5, with 1 being “very hard” and 5 being “very easy”.

This marked the end of User Centric’s first stage of testing. In the following paragraphs we will look at the findings from their next study, which was focused on the text entry.

Text Entry

Previously, test participants found the iPhone’s touch keyboard overly sensitive despite the iPhone’s overall high usability. From July 24 to July 26, 2007, User Centric compared texting experiences of iPhone owners and non-owners across devices.

User Centric collected data from 60 participants who entered specific text messages and completed mobile device tasks.  Twenty iPhone owners (had iPhone for at least one month), 20 hard-key QWERTY phone (aka QWERTY) owners, and 20 numeric phone owners (multi-tap texters) all entered six fixed-length text messages on their own phones.  Non-iPhone owners also entered six messages on a test iPhone and a phone of another type.  Here, numeric users used a Blackberry while QWERTYs used a Samsung E300. 
 
Participants with a QWERTY phone keyboard took significantly longer to type text messages using the iPhone than their own phones. Participants using the “multi-tap” keyboard method did not experience a difference in the time required to type messages on their own phone and the iPhone. However, it should be remembered that participants had a lot more experience on their own phones than the iPhone.

Texting Errors More Frequent on iPhones than Hard-Key QWERTY and Numeric Phones

iPhone owners entered text as rapidly as QWERTY owners on their own phones.  However, iPhone owners made significantly more texting errors on their own phone (5.6 errors/message) than both QWERTY owners (2.1 errors/message) and numeric phone owners (2.4 errors/message) on their own phones, p < .01.

The error rate for QWERTY and NUMERIC keyboard phone users was higher on the iPhone (average of 11 per text message) compared to an average of 3 errors per text message on their own phones.

Interestingly, comparing texting performance between iPhone owners and non-owners on the iPhone found no significant difference in error rates.

“While the iPhone’s corrective text feature helps, this data suggests that iPhone users who have owned the device for a month still make about the same number of errors as when they purchased it,” said Gavin Lew, Managing Director.

Performance on the iPhone did improve over time as users took less time to type messages as they became more familiar with the phone. On their own phones, the improvement was 10 seconds on average, whereas the iPhone resulted is a 1 minute improvement on average.

Aside from the empirical data, the qualitative feedback indicated that the iPhone keypad exhibits some ergonomic and sensitivity issues. The crux of the matter being that keys were felt to be placed too close together. Participants experienced difficulty with the Q&W and O&P keys as well as the space bar, return and backspace keys.  Some users actually asked if the iPhone came with a stylus as they felt they could be more accurate with a narrower pointing device than their fingers. The lack of tactile feedback was also commented upon and many participants said they could not see themselves attempting text entry on the iPhone in distracting conditions.
               
On feature of the popup keyboard on the iPhone is the drag and lift feature which is said to reduce errors. Unfortunately not one user discovered this feature.

It is worth noting, however, that performance on the iPhone did improve over time. Users took less time to type messages as they became more familiar with the iPhone. On their own phones, the improvement was 10 seconds on average, whereas the iPhone resulted in a 1 minute improvement on average. 

Unfamiliar Users Performed Best on Hard-Key QWERTY Phone

User Centric also compared users’ performance on unfamiliar phones.  Numeric phone owners had faster text entry on a hard-key QWERTY phone than on the iPhone and made significantly fewer errors on the hard-key QWERTY.

“Participants indicated a preference for hard-key QWERTY phones when texting,” said Jen Allen, User Experience Specialist.

Lew added that the “iPhone is a great switch from a numeric phone. But if you’re switching from a hard-key QWERTY phone, try the iPhone in the store first.”

User Centric is a privately-held consulting firm, focused on improving user experience since 1999.  Services include user research, eye tracking, expert evaluations, design consulting, and global studies.  http://www.usercentric.com

Possible Related Articles:

• Predicting What Mobile Phones Will Do for us Next
• Transforming Taiwan Aboriginal Cultural Features into Modern Product Design: A Case Study of a Cross-cultural Product Design Model
• Driver Segmentation, New Technology and Safety
• Immersion in videogames
• The Role of Ethnography in the Organizational Implementation of IT
• Involving Local Undergraduates in Fieldwork
• Design for people with disabilities in Japan
• So… Does the iPhone live up to its hype?
• Culture, Internationalisation and Usability - III
• Culture, Internationalisation and Usability - II