Augmented reality and 3D printing for surgery

New technologies can help to provide surgeons in surgical theaters with important information and thus significantly increase the chances of success. A research group led by the Center for Computing Technologies (TZI) at the University of Bremen is currently developing test processes and applications. The use of virtual reality (VR), thus artificial realities created with the help of computers, augmented realities (AR), i.e. computer-supported expanded reality, and 3D printing are to improve the planning and realization of surgeries. The results are to be additionally employed for training purposes and for patient information.

Preoperative planning

The planning of an operation with 3D models has significant advantages over the current method with 2D pictures. Doctors have a notably more realistic and comprehensible overview of the area on which they wish to operate. Alongside the 3D computer models, the VIVATOP researchers are testing the use of lifelike physical models of the affected organs. The picture data is analyzed and prepared for this so that it can be imported to a 3D printer, in order for a lifelike organ, for example a patient’s liver, to be made.

The stated printed models make it possible for doctors to gain haptic experience with each organ, thus the use of their sense of touch, but can also be applied together with the virtual model. It is therefore possible for someone to hold the physical model in their hands and to explain with gestures, whilst other participants can see the gestures in connection with the highly detailed computer model through a VR headset. This method of use correlates more with our natural habits than the use of controllers in the way that they are used for the controlling of movement in games consoles.

The 3D prints, which can be either transparent or multicolored, do not only simplify the planning but also the consultation with patients. They allow the team of doctors to explain the procedures in a clearer manner.

Support during surgery

During surgery, picture data and models of the organ help doctors to orient themselves and to make decisions. Additionally, the VIVATOP team is expanding an available augmented reality application, which allows for the relevant information to be shown in the field of view of a pair of glasses. Surgeons are able to control the unit with voice controls and without using their hands. At the same time there is also the possibility of presenting surgeries live in virtual reality so that experts outside of the surgical theater can be consulted (remote, collaborative, immersive surgery consultation). The hand movements of doctor teams and the procedures on the operated organ can be recorded using depth cameras, sensors and further technologies. These can then be shown in virtual reality. The planning details can also be displayed in virtual reality at the same time. Using telepresence, the system supports worldwide cooperation of specialists during a surgery.

Furthermore, vocational training, professional development and further education of young doctors can be designed in a more efficient manner by integrating AR, VR and 3D printing. Specific cases are able to be discussed using the 3D print data and can be precisely comprehended in virtual reality. The printed 3D models allow young doctors to gain valuable first experience. They can, for example, learn to examine tumors at varying stages by palpation (haptic feedback) and to subsequently judge the location in virtual reality. Individual surgery steps can also be trained in virtual reality.