FingerTrak bracelet with a thermal imaging camera for hand tracking in VR

FingerTrak bracelet with a thermal imaging camera for hand tracking in VR

Experts SciFi Lab at Cornell University in conjunction with the University of Wisconsin-Madison have developed a unique device in the form of bracelet, which uses four small (9,30 mm) synchronized thermal imaging cameras for more accurate tracking VR and AR.


The device is called FingerTrak, and it captures the hand movements and then transmits that data to bound the Raspberry Pi device connected to the computer. Then the computer starts a deep learning model that can predict all 20 of the regulations of the finger joints.



Each thermal imaging camera tracks the movement of the top, bottom, left and right hand pieces, and then combines them into one stitched thermal image in real-time. Because FingerTrak uses thermal image, the system is able to more accurately track the movement of the hands compared to devices such as Leap Motion or Oculus Quest.


To announce their achievements in the technology of hand tracking, SciFi Lab released a whitepaper along with a video detailing their work on the tracking device for a thermal imaging camera. This video shows the hand of the user, because this man performs everyday actions such as flipping through the phone book and the entry in the journal, the presentation of a Cup of coffee and removal of the mobile device.



Even subtle hand movements can be tracked. Thermal imaging camera with low resolution (32 x 24) is able to assess the situation of bare hands, tracking the contour of the silhouettes.


The video also shows how FingerTrak allows you to control a robotic arm in real time. When the user moves fingers clenches fist, closes and opens his brushes, robotic hand reflects every movement in real time. This technology can be incredibly useful in a variety of dangerous situations such as bomb disposal, the study of dangerous conditions or even remote provision of care at home.



However, according to Sci-Fi Lab, have a lot of work before their product is ready for mass production. Like most other forms of tracking technology at the hands of the consumer, FingerTrak could not provide any tactile feedback, making it difficult to perform tasks that depend on the ability to “feel” the world around us. Another problem is that FingerTrak has a small angular error, which ranges from 6.46 to 8.06 degrees. Although it may not seem like much a drawback, it may be of great value during operations requiring precision such as, for example, cutting certain wires on the bomb.


SciFi Lab will present their work at International joint conference ACM 2020 to common and ubiquitous computing in September. You can also get access to their documentation here.



Go to our cases Get a free quote