2011 · 48 citations · 15 references
ComputingComputer ScienceHuman-computer InteractionInteraction DesignVisual RepresentationAccessibility In VisualizationInterface DesignGraphical User InterfacesMetadataPixel-based Image AnalysisPixel-based VisualizationPlatform Independence
Pixel-based methods are emerging as a new and promising way to develop new interaction techniques on top of existing user interfaces. However, in order to maintain platform independence, other available low-level information about GUI widgets, such as accessibility metadata, was neglected intentionally. In this paper, we present a hybrid framework, PAX, which associates the visual representation of user interfaces (i.e. the pixels) and their internal hierarchical metadata (i.e. the content, role, and value). We identify challenges to building such a framework. We also develop and evaluate two new algorithms for detecting text at arbitrary places on the screen, and for segmenting a text image into individual word blobs. Finally, we validate our framework in implementations of three applications. We enhance an existing pixel-based system, Sikuli Script, and preserve the readability of its script code at the same time. Further, we create two novel applications, Screen Search and Screen Copy, to demonstrate how PAX can be applied to development of desktop-level interactive systems.
15
Tom Yeh, Tsung-Hsiang Chang, Robert C. Miller · 2009 · 341 citations
GUI testing using computer vision
Tsung-Hsiang Chang, Tom Yeh, Robert C. Miller · 2010 · 205 citations · Full text
Desney Tan, Brian Meyers, Mary Czerwinski · 2004 · 171 citations
Wolfgang Stuerzlinger, Olivier Chapuis, Dusty Phillips et al. · 2006 · 143 citations · Full text
Arts, Design, Facade Design +16