Proposed MR system: Integration of semantic segmentation into MR
Proposed MR system: Integration of semantic segmentation into MR
Source: Osaka University

Mixed reality gets machine learning upgrade

Scientists from the Division of Sustainable Energy and Environmental Engineering at Osaka University employed deep learning to improve mobile mixed reality generation. They found that occluding objects recognized by the algorithm could be dynamically removed using a video game engine. This work may lead to a revolution in green architecture and city revitalization.

Mixed reality (MR) is a type of visual augmentation in which real-time images of existing objects or landscapes can be digitally altered. As anyone who has played Pokémon Go! or similar games knows, looking at a smartphone screen can feel almost like magic when characters appear alongside real landmarks.

This approach can be applied for more serious undertakings as well, such as visualizing what a new building will look like once the existing structure is removed and trees added. However, this kind of digital erasure was thought to be too computationally intensive to generate in real time on a mobile device.

Now, researchers at Osaka University have demonstrated a new system that can construct a MR landscape visualization faster with the help of deep learning. The key is to train the algorithm with thousands of labeled images so that it can more quickly identify occlusions, like walls and fences. This allows for the automatic "semantic segmentation" of the view into elements to be kept and others to be masked.

MR-based landscape visualization with dynamic occlusion handling in field...
MR-based landscape visualization with dynamic occlusion handling in field validation
Source: Osaka University

The program also quantitatively measured the Green View Index (GVI), which is the fraction of greenery areas including plants and trees in a person's visual field, in either the current or proposed layout. "We were able to implement both dynamic occlusion and Green View Index estimation in our mixed reality viewer," corresponding author Tomohiro Fukuda says.

Planting simulation with MR and greenery estimation
Planting simulation with MR and greenery estimation
Source: Osaka University

Live video is sent to a semantic segmentation server, and the result is used to render the final view with a game engine on the mobile device. Proposed structures and greenery can be shown even when the viewing angle is changed. "Internet speed and latency were evaluated to ensure real-time MR rendering," first author Daiki Kido explains. The team hopes this research will help stakeholders understand the importance of GVI on urban planning.

The research was published in Advanced Engineering Informatics.

Subscribe to our newsletter

Related articles

'Origami' testing app tackles spread of malaria

'Origami' testing app tackles spread of malaria

A new approach to tackling the spread of malaria in sub-Saharan Africa, which combines affordable, easy-to-administer blood tests with machine learning and unbreakable encryption, has generated encouraging early results in Uganda.

'Deepfaking the mind' to improve brain-computer interfaces

'Deepfaking the mind' to improve brain-computer interfaces

Researchers are using generative adversarial networks to improve brain-computer interfaces for people with disabilities.

AI picks up mutations in colorectal cancers

AI picks up mutations in colorectal cancers

A deep learning algorithm picks up molecular pathways and the development of key mutations more accurately than existing methods.

Researchers psychoanalyse artificial intelligence

Researchers psychoanalyse artificial intelligence

We can run tests and experiments, but we cannot always predict and understand why AI does what it does.

The progress and risks of artificial intelligence

The progress and risks of artificial intelligence

Artificial intelligence has reached a critical turning point in its evolution, according to an international panel of experts.

Deep learning helps visualize X-ray data in 3D

Deep learning helps visualize X-ray data in 3D

Scientists have leveraged artificial intelligence to train computers to keep up with the massive amounts of X-ray data taken at the Advanced Photon Source.

Artificial intelligence shortcuts introduce bias in cancer treatment

Artificial intelligence shortcuts introduce bias in cancer treatment

AI tools models are a powerful tool in cancer treatment. However, unless these algorithms are properly calibrated, they can sometimes make inaccurate or biased predictions.

The gait lab for the pocket – app-based fall prevention

The gait lab for the pocket – app-based fall prevention

Every day, elderly people fall – be it at home or in care facilities. Lindera aims to reduce the risk of falling with the help of artificial intelligence.

Using AI to predict 3D printing processes

Using AI to predict 3D printing processes

Engineers use Frontera supercomputer to develop physics-informed neural networks for additive manufacturing.

Popular articles

Subscribe to Newsletter