Using VR headsets, healthcare professionals are able to conduct interviews with...
Using VR headsets, healthcare professionals are able to conduct interviews with the avatars.
Source: Roberto Schirdewahn

Mentally ill avatars

To teach medical students, scientists have created avatars that feature typical symptoms of different mental disorders as well as psychiatric and psychotherapeutic treatment situations.

The diagnosis of mental disorders depends not only on the answers patients provide in the interview with a healthcare professional. The way they behave is likewise relevant: how does their voice sound? Do they look the other person in the eye? What’s their posture like? Are they pacing restlessly? Is the face stony or animated? “An experienced healthcare professional monitors and assesses all these aspects while talking to a patient, noticing changes to the facial expression within milliseconds,” outlines Associate Professor Paraskevi Mavrogiorgou from the RUB clinic for psychiatry, psychotherapy and preventive medicine.

Medical students have to learn how to do that, which isn’t always easy. “At the university clinic, we mainly see patients who are seriously ill,” says the physician. “It is difficult to motivate them to take part in training sessions with the students. Plus, they are not the kind of patients whom the future doctors will most likely be seeing in the course of their career.”

In order to enable students to practise with patients presenting the full range of symptoms and degrees of severity and to broaden the panorama, the clinic has launched a project focusing on creating avatars that feature typical symptoms of different mental disorders as well as psychiatric and psychotherapeutic treatment situations. Students wearing a virtual reality headset (VR headset) will be able to meet those avatars in a three-dimensional space – or, as experts call it, “exploration”.

The clinic has already established rooms and prepared them structurally. Here, small cubicles will be set up that students will enter with their VR headsets in order to talk with their virtual patients. “As far as we know, this has never been done anywhere yet; we are pioneering the field,” says Professor Dr. Georg Juckel, Medical Director at the clinic.

The interaction with the avatars will be taking place in small cubicles, in...
The interaction with the avatars will be taking place in small cubicles, in order to make students feel like they’re in a real interview setting.
Source: Roberto Schirdewahn

Difficulty: designing a dialogue flow

The preparations have taken two to three years; now, the virtual characters are being fleshed out. “They need a biography, a case history,” explains Paraskevi Mavrogiorgou. “And they must be able to answer specific key questions in accordance with their respective mental disorder.” If the attending doctor asks an avatar suffering from schizophrenia whether he hears voices, the avatar should say he does. If the attending doctor asks a question that doesn’t help find the correct diagnosis, the avatar mustn’t simply remain silent; rather, he has to give a plausible, evasive answer.

“At the moment, this is the greatest difficulty: designing a dialogue flow,” says Georg Juckel. “We have compiled hundreds of pages with instructions and dialogues,” points out Paraskevi Mavrogiorgou. For the dialogue to work, high-performance speech recognition software is necessary to ensure that the computer in the background understands the questions and can provide relevant answers.

In order to create avatars with plausible physical features, the medical professionals intend to collaborate with a commercial enterprise specialised in visualisation for medical applications – after Arthur Berns, young computer engineer from Göttingen, has designed the logical framework. The avatars’ body language is to be adapted to the type of mental disorders they represent. They might be sitting, standing, lying or pacing during the interview. Their facial expressions, too, are supposed to tell the trained observer something about the underlying disorder. “The programmer has to design each of these facial expressions individually,” as Georg Juckel elaborates the immense complexity of the task. “It’s the same as with Walt Disney’s animated movies, or with computer games today.”

The avatars are to be built up segmentally, “or else the whole thing would soon become quite boring for the students,” says Juckel. Thanks to the modular system, the difficulty level of the exploration can be regulated. The selected avatar represents a specific psychiatric diagnosis or treatment situation. However, the way students get there, the questions they ask, is not predefined: this is what they will be learning under guidance and supervision.

The project team expects that the avatars will be taking their first steps in spring 2020. In summer semester 2020, students might begin conducting the first interviews with them. “We’re hoping that this approach will catch on with the students – especially in view of today’s leisure activities,” concludes Paraskevi Mavrogiorgou. They are definitely curious to see what the virtual patients are all about.

Subscribe to our newsletter

Related articles

World-leading pioneers speak at Medical XR congress

World-leading pioneers speak at Medical XR congress

At Shift Medical, more than 60 leading medical XR experts will present and discuss the latest developments on the use of digital technologies such as virtual reality, augmented reality and mixed reality in medicine.

Exposure therapy in VR to prevent disorders

Exposure therapy in VR to prevent disorders

VR can have significant impact on the validity of remote health appointments for those with eating disorders, through called Virtual Reality Exposure Therapy (VRET).

Virtual reality: Avatars against obesity

Virtual reality: Avatars against obesity

A collaborative project develops virtual reality methods to positively affect the body perception of obese patients.

Virtual reality improves mental health of cancer patients

Virtual reality improves mental health of cancer patients

Using VR, the patients felt improvements to their mental health and reported feeling less nervous, anxious, or on edge after the trial.

Like yin and yang: mindful breathing for pain control

Like yin and yang: mindful breathing for pain control

Researchers compared traditional mindful breathing and virtual reality, 3D-guided mindful breathing to reduce pain.

E-mental health: A solution for a rising challenge

E-mental health: A solution for a rising challenge

E-mental health services could provide a response to these challenges and offer effective ways for prevention, diagnosis, treatment, and aftercare.

Different affections of VR

Different affections of VR

While very little is known on the effects of immersive VR on adults, there is next to no knowledge on the impact of such systems on the sensorimotor abilities of young children.

The application of XR in mental health care

The application of XR in mental health care

The past two decades have witnessed the emergence of extended reality as a tool for investigation, assessment, and management in mental healthcare.

Augmented reality helps tackle arachnophobia

Augmented reality helps tackle arachnophobia

Researchers have developed an augmented reality app for smartphones in order to help people reduce their fear of spiders.

Popular articles

Subscribe to Newsletter