Journal of Usability Studies archive · 2012 · 193 citations · 16 references
Customer SatisfactionCross-sectional MethodEngineeringConsumer ResearchProduct ExperienceManagementUser PerceptionLongitudinal StudiesUsability EngineeringDesignUser ExperienceUser EvaluationMarketingUser ResearchInteractive MarketingHuman-computer InteractionTechnologyUser-centric EvaluationSurvey MethodologyWeb Site
Longitudinal studies consider prior user experience, but are difficult to conduct and the literature on how experience affects System Usability Scale ratings is sparse, despite recent evidence that prior experience can influence ratings. The study aimed to determine whether domain specialists using geoscience products exhibit the same experience effect on SUS ratings as users of off‑the‑shelf office products or personal web applications. A cross‑sectional analysis of 262 end users testing two related oilfield product releases—one web‑based and one desktop‑based—was performed to assess SUS scores relative to users’ prior experience. Users with extensive prior experience scored 15–16 % higher on the SUS than those with limited or no experience, regardless of product type, and these results informed practical guidelines for product analysts.
Longitudinal studies have to do with testing over time and thus take into consideration previous user experience with a product or product versions. However, it is difficult to conduct these types of studies. Therefore the literature is sparse on examples of the explicit effect of user experience on user satisfaction metrics in industry-standard survey instruments. During a development experience in 2009, we used a cross-sectional method to look at the effects of user profiles on ratings for commercial products that use one such instrument, the System Usability Scale or SUS. Recent research has reported finding that differences in user ratings could be based on the extent of a user's prior experience with the computer system, a Web site being visited or a desktop application like Microsoft's Office suite being used. Compared to off-the-shelf office products or personal Web applications, we were curious if we would find the same experience effect for domain specialists using geosciences products in the course of their daily professional job roles. In fact, from data collected with 262 end users across different geographic locations testing two related oilfield product releases, one Web-based and one desktop-based, we found results that were quite close to early assessment studies: Users having a more extensive experience with a product tended to provide higher, more favorable, SUS scores over users with either no or limited experience with a product---and by as much as 15-16%, regardless of the domain product type. This and other observations found during our product testing have led us to offer some practical how-to's to our internal product analysts responsible for managing product test cycles, administering instruments like the SUS to users, and reporting results to development teams.
16