Mixed reality surgical training with the VirtaMed LaparoS simulator

Bex Cross

 

Recognized as cutting-edge medical technology, the VirtaMed LaparoS uses Unity’s real-time 3D digital twin technology and integrated haptic feedback to provide hyperrealistic surgical training.

Note: This blog contains medical images that may be uncomfortable for some.

VirtaMed, a Swiss-based developer of mixed reality simulators for medical education, has created a solution that reduces the risks associated with traditional surgical training. Leveraging Unity’s digital twin technology to recreate the complex surgical training scenarios, VirtaMed’s innovative and award-winning LaparoS is revolutionizing how doctors learn to perform surgery.

Dr. Ivan Puente, MD, FACS Program Director, Broward Health, said, "From the moment you walk up to the simulator you can set it up just like we do in the operating room (OR). We are able to mimic exactly what we do on the real patient. I like that we can add complications such as bleeding and bile leakage to better prepare residents for the OR. Clearly, training on the simulation before the operation enhances skills and improves patient safety."

In this article we take a closer look at VirtaMed’s simulator training program built on real-time 3D digital twin technology, and the benefits of simulated surgical training.

Building a better training program

Keeping live patients safe from the learning curve

Traditional surgical training is based on the “see one, do one, teach one” model. However, this model doesn’t scale in a world of advanced medical technology and techniques. VirtaMed believes that the first time a surgeon performs a procedure should never be on a live patient. The LaparoS mixed reality simulator allows surgeons to train on laparoscopic general surgery (where a small camera is inserted through an incision in the abdominal wall) and gynecological procedures on “digital humans” in a hyperrealistic and risk-free environment.

Combining ultrarealistic real-time 3D visualization with an innovative simulation platform that provides haptic feedback, VirtaMed has created a laparoscopic training simulator that breaks the mold of traditional medical training. It takes the burden of the medical training learning curve off of the patients and places it on the simulator.

Hyperrealistic visual assets are essential for a successful training experience

Improve training performance

Scientific research has demonstrated that simulations dramatically improve training outcomes over those of traditional methods. A quality simulated experience allows trainees to suspend their disbelief and immerse themselves in a training scenario. For example, if a virtual patient bleeds during training, it should look and feel sufficiently real for the trainee to become genuinely invested in seeking a quick resolution.

The LaparoS simulator also records trainee behaviors with millimeter precision, providing valuable feedback not only on the trainee’s performance during the procedure, but also on the potential outcomes for the simulated patient.

Simulated training using a digital twin of the abdomen and other body parts not only accelerates the learning curve, getting new surgeons into the operating room more quickly, but also reduces overall training costs by allowing for team training scenarios.

Accurate measurement of instrument placement in simulated surgical processes enables essential constructive feedback, which is critical for training success.

Universalizing surgical skills training

Traditionally, expert surgeons share their skills using an apprenticeship model. Training by this method relies on the skills of the teacher. VirtaMed’s goal is to codify the techniques of globally recognized surgical experts and translate those skills into the simulator program, hence making them available to any trainee using the simulator.

Using machine learning, VirtaMed’s simulation platform uses data generated by expert surgeons to quantify best practice surgical techniques. Their expertise is translated to the simulation platform and used as a standard against which to measure trainee performance. Trainees also receive context-dependent feedback based upon these expert-derived standards. In this way, the growing availability of digital twin-based simulation training democratizes specialized surgical skills by removing many of the barriers to access.

“We use disruptive technology to improve surgical training in the service of patients. Using Unity allows us to leverage the substantial investments and advancements made in gaming where we can have the biggest impact – now and in the future.”

– Stefan Tuchschmid, Co-CEO and Founder, VirtaMed

Immersive real-time 3D visual assets combined with haptic feedback provide hyperrealistic training simulation.

From theory to reality

The building bricks for LaparoS

VirtaMed needed a real-time 3D engine that enabled its existing technology to scale, to support state-of-the-art graphics capable of replicating the mechanics and intricacies of a real operating room procedure.

Basil Fierz, a senior software engineer at VirtaMed, identifies why the company chose Unity as its technology partner:

  • Unity is committed to investing in and improving technologies, and VirtaMed understood the potential capabilities of Unity’s High Definition Render Pipeline (HDRP).
  • The Unity engine’s innate extensibility enabled VirtaMed to extend its existing physics simulation tool into Unity, eliminating the need to start again with another physics engine.
  • Unity’s Integrated Success guaranteed a support structure for the VirtaMed team to push development boundaries with confidence.

“The Unity Developer Relations Manager is our bridge to Unity’s entire organization, helping us connect efficiently with everyone from engineering to marketing. Our regular meetings go beyond day-to-day support topics and are immensely helpful, whether it’s pointing us to relevant content or advising new features to improve our workflows.”

– Basil Fierz, PhD, Senior Software Engineer, VirtaMed

Integrating VirtaMed’s physics engine into Unity enables simulation of realistic surgical behavior such as bleeding.

Developing the ultrarealistic simulator experience

By adopting Unity, VirtaMed was able to combine realistic visual assets and haptic feedback for a superior training experience, unique to the LaparoS. Through integration of its existing physics engine as a module directly into Unity, VirtaMed is able to accurately represent both the physics behavior of internal organs and the interaction of surgical tools with those organs.

The initial development phase focused on ensuring that the simulated organs physically behaved just like real organs. VirtaMed achieved this through collating information sources such as actual video footage of medical procedures, an understanding of the surgical procedures, observation of operations, and direct physician input. The physics simulation reacts to the interactions with the tools, which compute feedback forces and translate those to the simulator trocar tools to provide the haptic feedback.

VirtaMed’s Unity-based platform also enables flexibility for local training teams to adapt to regional differences in surgical practices, such as on which side of a patient the surgeon positions themself. Features like these are easily enabled or disabled according to the trainer’s preference.

During the development phase, the VirtaMed team used Unity’s Furioos streaming solution for internal testing. The Furioos cloud-based streaming technology makes it easy to access complex, interactive 3D applications in a web browser to support collaborative design review.

Simultaneously, VirtaMed developed the physical simulator machinery. Being able to conduct design review for the simulations using Unity’s Furioos solution avoided unnecessary delays in the simulator hardware production.

Combining realistic visual assets with accurate physics behavior of internal organs using the Unity Editor.

Planning for the future with Unity

Based on the success of the LaparoS training simulator, VirtaMed plans to migrate existing products to Unity in the next 12 months and use Unity for future products.

Working with the Unity Editor allows VirtaMed to create content faster and extend the benefits to a wider range of products, including building smaller custom projects for other industry customers. Future plans include building the training program skills repository to include rare case scenarios. By building the simulation platform with Unity’s digital twin technologies, VirtaMed is ensuring its readiness to expand and develop its simulation offerings.

“Our vision for a better way of training is to provide surgeons with the right tools to practice in a safe environment, without any risk to patients. As the training pathway becomes digital, it also becomes measurable. Combine that with self-guided training, gamification, adaptive learning and you have all the ingredients for a true transformation of surgical training.”

– Stefan Tuchschmid, Co-CEO and Founder, VirtaMed

 


 

For more information about the many use cases for digital twins built in Unity, including case studies, demos and on-demand webinars, check out our resources. If you’d like to speak to our expert team, contact us today.

 

No Previous Articles

Next Video
How the pandemic catalyzed enterprises to embrace immersive tech
How the pandemic catalyzed enterprises to embrace immersive tech

As many as 75% of Fortune 500 CEOs are accelerating technology transformation in response to the ongoing CO...

Try Unity Forma for free for 30 days

Learn more