Proposed MR system: Integration of semantic segmentation into MR
Proposed MR system: Integration of semantic segmentation into MR
Source: Osaka University

Mixed reality gets machine learning upgrade

Scientists from the Division of Sustainable Energy and Environmental Engineering at Osaka University employed deep learning to improve mobile mixed reality generation. They found that occluding objects recognized by the algorithm could be dynamically removed using a video game engine. This work may lead to a revolution in green architecture and city revitalization.

Mixed reality (MR) is a type of visual augmentation in which real-time images of existing objects or landscapes can be digitally altered. As anyone who has played Pokémon Go! or similar games knows, looking at a smartphone screen can feel almost like magic when characters appear alongside real landmarks.

This approach can be applied for more serious undertakings as well, such as visualizing what a new building will look like once the existing structure is removed and trees added. However, this kind of digital erasure was thought to be too computationally intensive to generate in real time on a mobile device.

Now, researchers at Osaka University have demonstrated a new system that can construct a MR landscape visualization faster with the help of deep learning. The key is to train the algorithm with thousands of labeled images so that it can more quickly identify occlusions, like walls and fences. This allows for the automatic "semantic segmentation" of the view into elements to be kept and others to be masked.

MR-based landscape visualization with dynamic occlusion handling in field...
MR-based landscape visualization with dynamic occlusion handling in field validation
Source: Osaka University

The program also quantitatively measured the Green View Index (GVI), which is the fraction of greenery areas including plants and trees in a person's visual field, in either the current or proposed layout. "We were able to implement both dynamic occlusion and Green View Index estimation in our mixed reality viewer," corresponding author Tomohiro Fukuda says.

Planting simulation with MR and greenery estimation
Planting simulation with MR and greenery estimation
Source: Osaka University

Live video is sent to a semantic segmentation server, and the result is used to render the final view with a game engine on the mobile device. Proposed structures and greenery can be shown even when the viewing angle is changed. "Internet speed and latency were evaluated to ensure real-time MR rendering," first author Daiki Kido explains. The team hopes this research will help stakeholders understand the importance of GVI on urban planning.

The research was published in Advanced Engineering Informatics.

Subscribe to our newsletter

Related articles

Artificial intelligence shortcuts introduce bias in cancer treatment

Artificial intelligence shortcuts introduce bias in cancer treatment

AI tools models are a powerful tool in cancer treatment. However, unless these algorithms are properly calibrated, they can sometimes make inaccurate or biased predictions.

The gait lab for the pocket – app-based fall prevention

The gait lab for the pocket – app-based fall prevention

Every day, elderly people fall – be it at home or in care facilities. Lindera aims to reduce the risk of falling with the help of artificial intelligence.

Using AI to predict 3D printing processes

Using AI to predict 3D printing processes

Engineers use Frontera supercomputer to develop physics-informed neural networks for additive manufacturing.

Deep learning predicts viral infections

Deep learning predicts viral infections

Using fluoresence images from live cells, researchers have trained an artificial neural network to reliably recognize cells that are infected by adenoviruses or herpes viruses.

AI app could help diagnose HIV more accurately

AI app could help diagnose HIV more accurately

New technology could transform the ability to accurately interpret HIV test results, particularly in low- and middle-income countries.

A new framework for machine speech translation

A new framework for machine speech translation

Researchers propose a deep learning-based model for mimicking and continuously modifying speaker voice identity during speech translation.

Detecting carpal tunnel syndrome with AI and a game

Detecting carpal tunnel syndrome with AI and a game

Researchers combined motion analysis that uses smartphone application and machine learning that uses an anomaly detection method, thereby developing a technique to easily screen for carpal tunnel syndrome.

Using AI to generate 3D holograms in real-time

Using AI to generate 3D holograms in real-time

A new method called tensor holography could enable the creation of holograms for virtual reality, 3D printing, medical imaging, and more — and it can run on a smartphone.

AI identifies 'ugly ducklings' to catch skin cancer

AI identifies 'ugly ducklings' to catch skin cancer

Deep learning-based system enables dermatologist-level identification of suspicious skin lesions from smartphone photos, allowing better screening.

Popular articles

Subscribe to Newsletter