French student and software engineer, Tony Beltramelli, has published his master thesis called Deep-Spying: Spying using Smartwatch and Deep Learning, in which he presents a new attack method that allows attackers to extract sensitive information like credit card or phone access PIN codes from motion sensors in wearable devices.
Mr. Beltramelli's research, while at the University of Copenhagen, Denmark, expanded on previous work done by Romit Roy Choudhury, Associate Professor at ECE Illinois, who showed how wearable devices (a Samsung Gear Live smartwatch) can be used to log keystrokes on a keyboard.
In Mr. Beltramelli's research, he narrowed down the attack surface to 12-key keypads, usually found on ATMs and the touch display of your smartphone, when using a PIN lock. Using an RNN-LSTM (Recurrent Neural Network - Long Short-Term Memory) deep learning algorithm, he trained an artificial neural network capable of interpreting data from a smartwatch's motion sensor and later making an analogy to each PIN pad's keys.
To prove his theories in practice, Mr. Beltramelli created a smartwatch application for a Sony SmartWatch 3, which he used to record accelerometer and gyroscope sensor data. Because of the watch's technical limitations, he wasn't able to send the data directly to a server, but to a nearby Android device (LG Nexus 4) (via Bluetooth), which then relayed it to a server for further analysis.
Using an algorithm that combined Java, Python, and Lua code, he was able to sift through all the data, eliminate noise movements, and detect patterns for various events, like when the user moves and taps his finger on a phone's touchscreen to unlock a PIN-protected phone, or when the user enters a PIN code on an ATM's keypad.
The algorithm is capable of both keylogging and touchlogging
"This architecture can achieve touchlogging and keylogging with a maximum accuracy of 73% and 59%, respectively," Mr. Beltramelli explained. "Moreover, the system is still able to infer keystrokes with an accuracy of 19% when trained and evaluated with datasets recorded from different keypads," he also added. "This result suggests that an attacker could log keys from a wide range of devices even if its classifier is trained with measurements from a different compromised device."
For now, everything is theoretical, but to advance his work, he also made the app and server-side code available on GitHub. While PIN-logging attacks via smartwatches may be a theoretical attack at this point, it may be the time to start wearing your smartwatch on the hand you don't use to enter PINs. Or, you could just be more careful what apps you install on your smartwatch, and avoid letting attackers have a foothold on your device in the first place.
