A participant tries out the intelligent tutoring system for microsurgical...
A participant tries out the intelligent tutoring system for microsurgical operations on the eye. The glasses show him a model at a high level of magnification.
Source: ETH Zurich / Sandro Ropelato

Eye surgery: Gaining insights through AR

Eye surgery is a delicate and precise process. A new simulation platform based on augmented reality allows surgeons to practice surgical procedures on a virtual model in three dimensions.

The first thing you notice when you visit Marino Menozzi’s office in Zurich is an eye the size of a medicine ball. The polystyrene model sits on a shelf behind Menozzi’s desk, but its purpose only becomes clear when you enter the lab next door. Positioned in front of a computer screen is a wooden board with two small cameras mounted at right angles to each other. Next to this is a handle with a tiny pair of forceps and bulky glasses that look like something out of Robocop. “What we have here is a simulation platform for eye surgery,” says Menozzi, a Privatdozent at the Chair of Consumer Behaviour.

Aspiring ophthalmologists have been visiting the laboratory for months, putting their skills to the test in surgery for epiretinal membrane, also known as macular pucker. This kind of operation becomes necessary when the connective tissue membrane around the vitreous of the eye becomes cloudy and creates tension on the retina. This can lead to tears which can cause severe loss of vision.

The first step in the surgical procedure is to remove the transparent gel in the posterior chamber of the eye known as the vitreous body. Next, the surgeon inserts very fine forceps through a cannula with a diameter of less than one millimetre and peels away the wafer-thin membrane from the retina. If all goes well, the patient’s vision returns to normal a few weeks later. This kind of operation requires great delicacy and experience, which is why budding surgeons often spend years playing an assisting role in the procedure while gradually improving their skills by practising on plastic models and live animals. This latter approach is expensive, raises ethical questions, and does not enable surgeons to practice difficult movements repeatedly.

This is where the Robocop glasses in Menozzi’s laboratory come into play, offering trainee surgeons the opportunity to practice in a virtual environment supported by augmented reality. Put on the glasses and a virtual eye appears in front of you, hovering in the air and magnified eight times – exactly the same level of magnification that surgeons normally use when performing surgery under the microscope. Unlike virtual reality (VR), where the user can no longer see their real-life environment at all, augmented reality (AR) merely enriches reality by adding virtual elements to what is already there.

The additional virtual elements in this case are the magnified eye and instructions for the operation that appear in the field of vision, including features such as arrows, lines and circles that show the trainee surgeon which path they should ideally follow with the forceps. At the same time, they can still see the real world around the magnified eye, allowing them to see their hands and the position of the surgical instrument.

The tiny cameras on the board record every hand movement and transmit it to the glasses. The computer can use this information to precisely calculate the degree of accuracy with which the participant has performed the operation. Budding surgeons can also work through a virtual “intelligent tutoring” process based on an algorithm that calculates which stages of the operation they performed successfully and which they struggled with. This feedback is used to create a sequence of exercises that guarantees the most effective learning experience. “We hope this will improve the training effect, reducing the time it takes to get surgeons into the operating room and minimising the number of mistakes they make,” says Menozzi.

Lack of haptics and a delay

Three Master’s theses are currently exploring the benefits and acceptance of the simulation with the help of 23 aspiring surgeons. Gian-Luca Köchli is one of the participants. He is in the final year of his Master’s degree programme and has already tried the AR simulator six times. “The simulation is remarkably sensitive and precise,” he says. “I was amazed at how realistic it already is.” One thing that bothered him, however, was the slight delay between his actual movements and the images shown in the glasses. He would also like to see realistic haptics incorporated in the simulator. In general, though, practising surgery with the simulator before performing it on patients is something he sees as a promising approach.

Menozzi currently sees three key challenges. The first is the 20 to 30 millisecond delay spotted by Köchli. This occurs because the virtual images that show the hand movements must first be calculated by a computer and then sent to the glasses via a wireless LAN, and Menozzi is aware that this latency is too high. The second challenge is that some users respond to certain simulations by becoming dizzy or even vomiting. The third challenge is achieving the feeling of actually being present during the simulation. “We see this as an important indication of how well the results of the simulation can be transferred to reality,” Menozzi says. At present, for example, users still have a severely restricted field of vision. Healthy human eyes have a natural field of view of around 200 degrees, while the mixed reality glasses only offer a visual field of 35 degrees. Achieving a realistic sensation would require a 120 degree field of view. The sense of reality could also be enhanced by sounds, smells and haptics.

Sandro Ropelato, who developed the simulator as part of his doctoral thesis, believes that the technology’s potential extends far beyond eye surgery. Other complex and risky surgical procedures could also be modelled for AR training simulations in the future, enabling surgeons to practice all their hand movements on a virtual model before operating on a patient. He also envisages applications in the realm of electronics, such as preparing silicon wafers for microchips, for example.

Ropelato’s longer-term vision includes a kind of app store that would offer a variety of simulations for his hardware set-up of the HoloLens, micro-cameras and computer. The 3D simulation is based on Unity, a leading game development platform that has also become the number one choice for a whole host of AR and VR applications. That means thousands of software components are available over the Internet, which simply need to be adapted to each particular application. In the medium term, Ropelato and Menozzi are planning to create a spin-off to market the simulation platform. “But we need to learn a lot more from our users before we do that,” Menozzi says. “Their acceptance of this product is the key to our success.”

Subscribe to our newsletter

Related articles

Ocutrx unveiled 8K AR surgery system

Ocutrx unveiled 8K AR surgery system

Ocutrx Vision Technologies has released a new system that provides the most modern options for surgery visualization designed to make it easier for surgeons to perform procedures.

VR and lightweight robot simulate hip replacement

VR and lightweight robot simulate hip replacement

Researchers have developed a virtual reality simulator for a particularly critical step in hip joint implantation.

Endoscopic prostate surgery performed on human-like phantom

Endoscopic prostate surgery performed on human-like phantom

Scientists have made a decisive contribution to improving complex surgical training by developing a very realistic prostate phantom.

Augmented reality and 3D printing for surgery

Augmented reality and 3D printing for surgery

Researchers are combining virtual reality, augmented reality and 3D printing in order to improve the planning and realization of surgeries.

Augmented reality makes for better surgeons

Augmented reality makes for better surgeons

A PhD candidate in computer science has developed an augmented reality headset to help brain surgeons train for high-risk operations.

Fundamental Surgery integrates HaptX Glove into platform

Fundamental Surgery integrates HaptX Glove into platform

With the integration of HaptX Gloves, FundamentalVR demonstrates the ability to integrate its Surgical Haptic Intelligence Engine with third-party devices.

Promising applications of mixed realities in medicine

Promising applications of mixed realities in medicine

At ECR 2019, researchers talked about the practical applications of mixed realities in medical education and training as well as preprocedural planning and visualization during a surgery.

New AR cellular with eye-tracking glasses unveiled

New AR cellular with eye-tracking glasses unveiled

Ocutrx Vision Technologies, LLC, a manufacturer of augmented reality (AR) glasses, announced a new, state-of-the-art design for the company’s flagship Oculenz AR Wear glasses.

SimVis allows patients to experience vision improvement before surgery

SimVis allows patients to experience vision improvement before surgery

Thanks to a new wearable visual simulator, patients will be able to experience how their vision will improve after cataract surgery, just before surgery.

Popular articles