Assessing the quality of ideas from prolific, early-stage product ideation

Abstract

The goal of many blue-sky idea generation techniques is to generate a large quantity of ideas with the hope of obtaining a few outstanding, creative ideas that are worth pursuing. As such, a rapid means of screening the resulting sketches to select a manageable set of promising ideas is needed. This study explores a metric for evaluating large quantities of early-stage product sketches and tests the metric through an online service called Mechanical Turk. Reviewers’ subjective ratings of idea creativity had a strong correlation with ratings of idea novelty (r 2=0.80), but negligible correlation with idea usefulness (r 2=0.16). The clarity of the sketch positively influenced ratings of idea creativity. Additionally, the quantity of ideas generated by an individual participant had a strong correlation with that participant's overall creativity scores (r 2=0.82). The authors suggest a metric of three attributes to be used as a first pass in narrowing a large pool of product ideas to the most innovative: novel, useful (or valuable), and feasible (as determined by experts).