Proposed MR system: Integration of semantic segmentation into MR
Proposed MR system: Integration of semantic segmentation into MR
Source: Osaka University

Mixed reality gets machine learning upgrade

Scientists from the Division of Sustainable Energy and Environmental Engineering at Osaka University employed deep learning to improve mobile mixed reality generation. They found that occluding objects recognized by the algorithm could be dynamically removed using a video game engine. This work may lead to a revolution in green architecture and city revitalization.

Mixed reality (MR) is a type of visual augmentation in which real-time images of existing objects or landscapes can be digitally altered. As anyone who has played Pokémon Go! or similar games knows, looking at a smartphone screen can feel almost like magic when characters appear alongside real landmarks.

This approach can be applied for more serious undertakings as well, such as visualizing what a new building will look like once the existing structure is removed and trees added. However, this kind of digital erasure was thought to be too computationally intensive to generate in real time on a mobile device.

Now, researchers at Osaka University have demonstrated a new system that can construct a MR landscape visualization faster with the help of deep learning. The key is to train the algorithm with thousands of labeled images so that it can more quickly identify occlusions, like walls and fences. This allows for the automatic "semantic segmentation" of the view into elements to be kept and others to be masked.

MR-based landscape visualization with dynamic occlusion handling in field...
MR-based landscape visualization with dynamic occlusion handling in field validation
Source: Osaka University

The program also quantitatively measured the Green View Index (GVI), which is the fraction of greenery areas including plants and trees in a person's visual field, in either the current or proposed layout. "We were able to implement both dynamic occlusion and Green View Index estimation in our mixed reality viewer," corresponding author Tomohiro Fukuda says.

Planting simulation with MR and greenery estimation
Planting simulation with MR and greenery estimation
Source: Osaka University

Live video is sent to a semantic segmentation server, and the result is used to render the final view with a game engine on the mobile device. Proposed structures and greenery can be shown even when the viewing angle is changed. "Internet speed and latency were evaluated to ensure real-time MR rendering," first author Daiki Kido explains. The team hopes this research will help stakeholders understand the importance of GVI on urban planning.

The research was published in Advanced Engineering Informatics.

Subscribe to our newsletter

Related articles

Detecting carpal tunnel syndrome with AI and a game

Detecting carpal tunnel syndrome with AI and a game

Researchers combined motion analysis that uses smartphone application and machine learning that uses an anomaly detection method, thereby developing a technique to easily screen for carpal tunnel syndrome.

Using AI to generate 3D holograms in real-time

Using AI to generate 3D holograms in real-time

A new method called tensor holography could enable the creation of holograms for virtual reality, 3D printing, medical imaging, and more — and it can run on a smartphone.

AI identifies 'ugly ducklings' to catch skin cancer

AI identifies 'ugly ducklings' to catch skin cancer

Deep learning-based system enables dermatologist-level identification of suspicious skin lesions from smartphone photos, allowing better screening.

Smart apps help people with hearing loss

Smart apps help people with hearing loss

Researchers have developed smartphone-based apps that solve the biggest problems for people with hearing loss: filtering out background noise and improving speech perception.

Biomedical research: deep learning outperforms machine learning

Biomedical research: deep learning outperforms machine learning

Deep-learning methods have the potential to offer substantially better results, generating superior representations for characterizing the human brain.

Using AI to find new uses for existing medications

Using AI to find new uses for existing medications

Scientists have developed a machine learning method that crunches massive amounts of data to help determine which existing medications could improve outcomes in diseases for which they are not prescribed.

Deep learning-based image segmentation

Deep learning-based image segmentation

Scientists have presented a new method for configuring self-learning algorithms for a large number of different imaging datasets – without the need for specialist knowledge or very significant computing power.

How AI can improve medical imaging

How AI can improve medical imaging

AI offers not only the possibility of better detection of a tumor, a skin lesion or some other indication but also can improve accuracy and efficiency for radiologists.

Machine learning hunts for COVID-19 therapies

Machine learning hunts for COVID-19 therapies

Researchers have created a deep learning model for drug developers targeting the SARS-CoV-2 main protease.

Popular articles

Subscribe to Newsletter