Professor Markus Meister and his laboratory combine augmented reality...
Professor Markus Meister and his laboratory combine augmented reality technology with computer vision algorithms to create goggles that can help blind people navigate unfamiliar spaces.
Source: Caltech

Augmented reality to guide the blind

Augmented reality technology gives objects "voices," enabling a vision-impaired person to hear what is in their surroundings.

Picture yourself going to an unfamiliar supermarket for the first time. If you are a person who can see, you can simply look around to guide yourself and identify objects and obstacles. However, blind people must use other senses to find their way through a new space.

Soon, the blind might have some navigational help, thanks to Caltech researchers who have combined augmented reality hardware and computer vision algorithms to develop software that enables objects to “talk.” Worn as a portable headset, the technology translates the optical world into plain English audio. The device could one day be made available in banks, grocery stores, museums, and other locations, to help blind people make their way through unfamiliar spaces. “Imagine you are in a world where all the objects around you have voices and can speak to you,” says Markus Meister, Professor of Biological Sciences and executive officer for neurobiology at CalTech. “Wherever you point your gaze, the different objects you focus on are activated and speak their names to you. Could you imagine getting around in such a world, performing some of the many tasks that we normally use our visual system for? That is what we have done here—given voices to objects.”

Led by graduate student Yang Liu, the team of scientists developed a system they call CARA, or the Cognitive Augmented Reality Assistant. CARA was developed for a wearable headset computer called a HoloLens, developed by Microsoft, that can scan a user’s environment and identify individual objects such as a laptop or a table. With CARA, each object in the environment is given a voice and will “say” its name upon the user’s command. The technology utilizes so-called spatialized sound, which causes objects to sound different depending on their location within a room. If the object is, for example, far to the left of the user, its voice will sound like it is coming from the left. Additionally, the closer the object, the higher the pitch of its “voice.”

The Meister lab had great success with initial testing and hopes the technology...
The Meister lab had great success with initial testing and hopes the technology can one day be offered by places like banks, grocery stores, museums, and more.
Source: Caltech

Various modes

To avoid a cacophony of objects speaking at once, Liu and his team programmed CARA with several different modes. In the first, called spotlight mode, an object only says its name when the user is facing it directly. As the user turns their head to face various objects, the objects each say their name, and the pitch of the object’s voice provides an auditory cue about its relative distance from the user. In this way, a vision-impaired user can “look around” to explore their environment. In a second mode, called scan mode, the environment is scanned from left to right with objects saying their names accordingly. The third mode is target mode, where the user can select one of the objects to talk exclusively and use that as a guide to navigate.

To test CARA’s practical applications, the researchers designed a test route for blind volunteers through the Beckman Behavioral Biology building at Caltech. To prepare the task, the scientists first developed a route through the building and followed it while wearing the HoloLens, which scanned the environment and saved it to memory. Then, the blind volunteers were asked to navigate the route using CARA as a guide. As each volunteer began, a voice, seemingly emanating from a location ahead on the route, called out “Follow me,” while also telling the user about stairs, hand rails, corners to turn, and the like. Led by CARA, all seven of the volunteers completed the task successfully on the first try.

CARA is still in its early stages, but will benefit from the rapid development of algorithms for computer vision. The researchers are already implementing new schemes for real-time identification of objects and pedestrians. Eventually, they hope that places like banks, hotels, and shopping malls will offer CARA devices for use by their blind customers.




Subscribe to our newsletter

Related articles

AR allows researchers to see patients’ real-time pain

AR allows researchers to see patients’ real-time pain

Researchers have developed a technology to help clinicians "see" and map patient pain in real-time, through special augmented reality glasses.

Rats in AR show how the brain determines location

Rats in AR show how the brain determines location

A study found that the brain continually updates its orientation in the world from physical cues and landmarks.

Staying fit with Pokémon GO

Staying fit with Pokémon GO

Pokémon GO is a very popular augmented reality game and researchers revealed for the first time how the game positively impacted the physical activity in players over 40.

Using VR to study the neuroscience of hearing

Using VR to study the neuroscience of hearing

Cool tools for scientists: Researchers view cells for hearing in 3D using virtual reality

Augmented reality app may aid patients with Parkinson’s

Augmented reality app may aid patients with Parkinson’s

Bioengineering students program smartphone to guide patients who ‘freeze’ while walking.

AR lets doctors see under patients’ skin

AR lets doctors see under patients’ skin

Augmented reality system, called ProjectDR, lets clinicians see patients’ internal anatomy displayed right on the body.

Using CRISPR to knock down gene messages

Using CRISPR to knock down gene messages

Researchers have harnessed CRISPR technology to target gene messages (messenger RNA) involved in early vertebrate

Network medicine makes drug repurposing effective

Network medicine makes drug repurposing effective

Artificial intelligence can increase the effectiveness of drug repositioning or repurposing research.

A new way to deliver drugs through the skin

A new way to deliver drugs through the skin

Scientists have showed that applying "temporal pressure" to the skin of mice can create a new way to deliver drugs.

Popular articles