Information security specialists from University of Texas and University of Oklahoma developed a sophisticated method to determine what users of video call platforms are writing on their keyboards entirely based on their motion. This method also applies for streaming on YouTube, Twitch and other popular platforms.
The method of determining which keys the user presses involves a three-step process; first, users have to preprocess the recorded video. At this stage, the background is removed and the video is grayed out. Next, the segments of the left and right hand are segmented relative to the person’s face, detected using face detector tools such as FaceBoxes.
During the second stage of this process, the frames are determined where the user presses the keys. Segmented hand frames are extracted to detect the structural similarity index (SSIM) to quantify body movements between successive frames in each of the left and right video segments and to identify potential frames in which keystrokes occurred.
Finally, the third stage is the definition of the words typed on the keyboard by the user. At this stage, segments of the frame with keystrokes are analyzed in order to find signs of movement before and after each detected keystroke. Then, using a dictionary-based prediction algorithm, users got a very specific word selection.
The development team tested this framework with 20 volunteers in a highly scenario. During the tests, these people used both clumsy typing with two fingers and blind typing and wore different clothes with several types of sleeves. Experts also used different backgrounds, webcam and keyboard models, and different video calling software, including Zoom, Hangouts and Skype, getting similar results.
This is a sample of the great diversification capacity that a group of threat actors can reach, so users must be aware of the words they type on their keyboards when using Zoom, Google Meet, Microsoft Teams and other similar tools. To learn more about information security risks, malware variants, vulnerabilities and information technologies, please feel free to visit the International Institute of Cyber Security website (IICS).
He is a cyber security and malware researcher. He studied Computer Science and started working as a cyber security analyst in 2006. He is actively working as an cyber security investigator. He also worked for different security companies. His everyday job includes researching about new cyber security incidents. Also he has deep level of knowledge in enterprise security implementation.