The system is the first of its kind to estimate 3D hand poses using a camera...
The system is the first of its kind to estimate 3D hand poses using a camera focusing on the back of the hand.
Source: Tokyo Institute of Technology

3D hand pose estimation using a wrist-worn camera

Researchers at Tokyo Institute of Technology (Tokyo Tech) working in collaboration with colleagues at Carnegie Mellon University, the University of St Andrews and the University of New South Wales have developed a wrist-worn device for 3D hand pose estimation. The system consists of a camera that captures images of the back of the hand, and is supported by a neural network called DorsalNet which can accurately recognize dynamic gestures.

Being able to track hand gestures is of crucial importance in advancing augmented reality (AR) and virtual reality (VR) devices that are already beginning to be much in demand in the medical, sports and entertainment sectors. To date, these devices have involved using bulky data gloves which tend to hinder natural movement or controllers with a limited range of sensing.

Now, a research team led by Hideki Koike at Tokyo Tech has devised a camera-based wrist-worn 3D hand pose recognition system which could in future be on par with a smartwatch. Their system can importantly allow capture of hand motions in mobile settings. "This work is the first vision-based real-time 3D hand pose estimator using visual features from the dorsal hand region," the researchers say. The system consists of a camera supported by a neural network named DorsalNet which can accurately estimate 3D hand poses by detecting changes in the back of the hand.

The researchers confirmed that their system outperforms previous work with an average of 20% higher accuracy in recognizing dynamic gestures, and achieves a 75% accuracy of detecting eleven different grasp types.

The work could advance the development of controllers that support bare-hand interaction. In preliminary tests, the researchers demonstrated that it would be possible to use their system for smart devices control, for example, changing the time on a smartwatch simply by changing finger angle. They also showed it would be possible to use the system as a virtual mouse or keyboard, for example by rotating the wrist to control the position of the pointer and using a simple 8-key keyboard for typing input.

They point out that further improvements to the system such as using a camera with a higher frame rate to capture fast wrist movement and being able to deal with more diverse lighting conditions will be needed for real world use.

Subscribe to our newsletter

Related articles

Using AI to generate 3D holograms in real-time

Using AI to generate 3D holograms in real-time

A new method called tensor holography could enable the creation of holograms for virtual reality, 3D printing, medical imaging, and more — and it can run on a smartphone.

AR makes chemistry and biology accessible everywhere

AR makes chemistry and biology accessible everywhere

A new website allows teachers and students to explore concepts from chemistry and biology by manipulating virtual molecules in augmented reality.

Augmented Reality in the OR: matching man and machine

Augmented Reality in the OR: matching man and machine

One of the crucial future technologies in surgery is Augmented Reality. Most experts agree that AR will increase safety and efficiency, improve surgical training and decrease costs.

3D hand-sensing wristband

3D hand-sensing wristband

Researchers have designed a wrist-mounted device and developed software that allows continuous tracking of the entire human hand in 3D.

Microsoft HoloLens cleared for medical use

Microsoft HoloLens cleared for medical use

The OpenSight® Augmented Reality System is the first AR medical solution for Microsoft HoloLens cleared by the FDA .

Olfactory VR show promise for mental health practices

Olfactory VR show promise for mental health practices

New research on Olfactory Virtual Reality (OVR) paints a clearer picture for clinical psychiatrists about how it could be used to safely and effectively help mental health and mood disorders.

Neural networks speed up holographic imaging

Neural networks speed up holographic imaging

Researchers have developed a holographic technique that can rapidly reconstruct microscopic images of samples with up to 50-fold acceleration compared to existing methods.

Augmented reality helps to see, feel and understand

Augmented reality helps to see, feel and understand

X-ray vision, context-sensitive guidance, coordinator, training assistant and more: augmented reality (AR) has hit the OR.

Nanophotonic optical element slim VR glasses

Nanophotonic optical element slim VR glasses

New optical elements that could revolutionize VR/AR glasses. At its heart is a nanophotonic optical element, which the developers call a metasurface.

Popular articles

Subscribe to Newsletter