TapID allows for extremely precise free-hand input in virtual spaces.
"TapID" allows for extremely precise free-hand input in virtual spaces.
Source: siplab / ETH Zurich

Virtual reality at your fingertips

When a person taps with their fingers, each finger generates a different vibration profile propagating to the wrist through bones. ETH Zurich researchers have now leveraged this discovery in the development of a dual-​sensor wristband that brings intuitive free-​hand interaction to virtual productivity spaces.

Virtual reality technology is advancing into new and different areas, ranging from medical applications, to pilot training in flight simulators to spatial visualisations, e.g., in architecture and increasingly life-like video games. The possibilities afforded by simulating environments in combination with technology such as VR glasses are practically endless.

However, VR systems are still rarely used in everyday applications. “Today, VR is used mainly to consume content. In the case of productivity applications such as in office scenarios, VR still has much potential for development to replace current desktop computers,” says Christian Holz, a professor at ETH Zurich’s Institute for Intelligent Interactive Systems. There is enormous potential indeed: if content were to be no longer limited to a screen, users would be able to leverage the nature of three-dimensional environments, interacting with great flexibility and intuitively with their hands.

Each finger causes different vibration profiles

What’s preventing this from becoming a reality? Holz thinks the main problem lies in the interaction between humans and technology. For example, most of today’s VR applications are either operated with controllers that are held in the user’s hand or with hands in the air, so that the position can be captured by a camera. 

The user is also typically standing during interaction. “If you have to hold your arms up all the time, it quickly becomes tiring,” says Holz. “This currently prevents normal work processes from becoming possible, as they require interaction with applications for multiple hours.” Typing on a virtual keyboard, for example, presents another problem: the fingers move only slightly and cameras cannot capture the movement as precisely as current mechanical keyboards do. With in-air typing, the usual haptic feedback is also lacking.

For this reason, it's clear to Holz’s research team that passive interfaces will remain important for the viable and productive adoption of VR technology. That could be a classic tabletop, a wall or a person’s own body. For optimal use, the researchers developed a sensory technology called “TapID”. The prototype embeds several acceleration sensors in a normal rubber wristband.

These sensors detect when the hand touches a surface and which finger the person has used. The researchers found that their novel sensor design can detect tiny differences in the vibration profile on the wrist in order to differentiate between each characteristic finger movement. A custom machine learning pipeline the researchers developed processes the collected data in real time. In combination with the camera system built into a set of VR glasses, which captures the position of the hands, TapID generates extremely precise input. The researchers have demonstrated this in several applications that they programmed for their development, including a virtual keyboard and a piano (see video).

Virtual piano using the smartwatch

The virtual piano does a particularly good job of demonstrating the advantages of TapID, explains Holz: “Here, both spatial accuracy and timing are essential. The moment at which the keys are touched must be captured with maximum precision. The wrist sensors can do this more reliably than a camera.” The relatively simple technology used by our system offers several advantages; for example, producing this type of wristband should cost only a few francs.

The research team also compared their system with existing technology: in a technical evaluation with 18 participants, they managed to show that TapID not only works reliably with the specially developed electronics in the wristband, but the method could also transfer to existing fitness wristbands and everyday smartwatches, because they are all equipped with inertia sensors. Looking ahead, the researchers plan to continue to improve the technology with more test subjects and develop more applications to integrate TapID into productivity scenarios and to support offices of the futures.

Holz thinks “mobile virtual reality” is another exciting possibility: “Our sensor solution is portable and it has the potential to make VR systems suitable for productivity work on the go. TapID allows users to operate applications with their hand or thighs – anywhere and any time.”

As a professor of computer science, Holz sees the future of virtual reality in being able to work together from any physical location – not limited by hardware but as if users were all in the same room. “TapID could be a big enabler in moving into that direction,” he adds. He and his team with Manuel Meier, Paul Streli and Andreas Fender will continue their research in this area.

Subscribe to our newsletter

Related articles

Sensor for smart textiles survives hammers

Sensor for smart textiles survives hammers

An ultra-sensitive, resilient strain sensor that can be embedded in textiles and soft robotic systems survived being tested by a washing machine and a car.

A wearable soft sensor

A wearable soft sensor

Biocompatible sensor could be used in diagnostics, therapeutics, human-computer interfaces, and virtual reality.

Sensor warns of impending COVID-19 cytokine storm

Sensor warns of impending COVID-19 cytokine storm

Scientists report preliminary results on a sweat sensor that acts as an early warning system for an impending cytokine storm, which could help doctors more effectively treat patients.

Wearable sensors to track Parkinson's symptoms

Wearable sensors to track Parkinson's symptoms

Scientists have developed algorithms that, combined with wearable sensors, could help clinicians to monitor the progression of Parkinson’s disease.

Wearable antennae stretches boundaries of medical tech

Wearable antennae stretches boundaries of medical tech

Researchers from Penn State led two international collaborations to prototype a wireless, wearable transmitter while also improving the transmitter design process.

Optimal use of activity trackers fails due to outdated computer skills

Optimal use of activity trackers fails due to outdated computer skills

Activity trackers are rising in popularity. Yet a new study demonstrates that many struggle to optimally use these devices. The cause? Outdated digital literacy skills.

Sticker detects cystic fibrosis in newborn's sweat

Sticker detects cystic fibrosis in newborn's sweat

Researchers have developed a novel skin-mounted sticker that absorbs sweat and then changes color to provide an accurate, easy-to-read diagnosis of cystic fibrosis within minutes.

A new medical device for monitoring vital signs

A new medical device for monitoring vital signs

A new device consisting of a 3D-printed wristband can remotely monitor patients' vital signs, such as body temperature, oxygen saturation, pulse, and respiratory rate.

Harvesting energy from radio waves to power wearables

Harvesting energy from radio waves to power wearables

Researchers have developed a way to harvest energy from radio waves to power wearable devices.

Popular articles

Subscribe to Newsletter