Deep Learning Based Inference of Private Information Using Embedded Sensors in Smart Devices

TLDR

Smart mobile devices and apps have proliferated, turning them into general‑purpose platforms whose sensory data—often deemed innocuous—are widely available without user permissions. This article demonstrates that such innocuous sensory data can pose serious privacy risks. We show that users’ tap positions can be identified from sensory data using deep learning, that tap‑stream profiles reveal app‑usage habits, and that experiments collected sensory data and app usage from 102 volunteers. The experiments show that tap‑position inference achieves at least 90 % accuracy with convolutional neural networks, and that inferred tap positions can enable high‑accuracy inference of users’ app‑usage habits and passwords.

Abstract

Smart mobile devices and mobile apps have been rolling out at swift speeds over the last decade, turning these devices into convenient and general-purpose computing platforms. Sensory data from smart devices are important resources to nourish mobile services, and they are regarded as innocuous information that can be obtained without user permissions. In this article, we show that this seemingly innocuous information could cause serious privacy issues. First, we demonstrate that users' tap positions on the screens of smart devices can be identified based on sensory data by employing some deep learning techniques. Second, it is shown that tap stream profiles for each type of apps can be collected, so that a user's app usage habit can be accurately inferred. In our experiments, the sensory data and mobile app usage information of 102 volunteers are collected. The experiment results demonstrate that the prediction accuracy of tap position inference can be at least 90 percent by utilizing convolutional neural networks. Furthermore, based on the inferred tap position information, users' app usage habits and passwords may be inferred with high accuracy.