Mixed reality techniques in cardiovascular medicine: a new approach towards image-based diagnosis and procedural support

Defining realities

Virtual reality (VR) is the application of computer technology to generate a three-dimensional simulated environment in which the user not only experiences but also interacts with the projected artificial environment, colloquially referred to as an immersive experience. This virtual reality is commonly rendered with a head-mounted display device. Augmented reality (AR) enhances real-world viewing with digitally generated components which provides the user with a more faithful representation of observations. Whereas VR engages a user in an artificial environment and AR superimposes digital objects on the real-world environment, mixed reality (MR) merges the fundaments of both VR and AR, in other words, a user can interact and manipulate both real and virtual items.

The three-dimensional digital models presented with display devices are in essence a form of holography, which is a diffraction technique that produces three-dimensional imagery effectuating amongst others the principles of depth perception such as parallax and perspective. Currently, the most popular head-mounted display devices are the Microsoft HoloLens (Redmond, Washington, USA) and Magic Leap (Plantation, Florida, USA). Notwithstanding, an inexpensive experience can be also attained by using a mobile phone with dedicated applications but this approach is limited in terms of user immersion and interactivity compared to the dedicated head-mounted display devices. 

Mixed reality techniques within the cardiology field

Instinctively mixed reality technologies lends itself well to educational domains. A number of MR derived training applications have been previously reported. The HoloAnatomy programme at Case Western Reserve University (Cleveland, Ohio, USA) resulted in the first MR healthcare application¹. The Lucile Packard Children’s Hospital Stanford (Palo Alto, California, USA) developed novel interactive visualisations as part of their Stanford Virtual Heart Project so that paediatric cardiologists can explain complex congenital heart defects to both trainees and patients alike.²

Additionally mixed reality technologies are gradually being introduced into pre-procedural planning protocols^3,4. Cardiac DICOM image visualization in three-dimensional simulated environment provides the user an enhanced appreciation of depth perception entailing accurate volume measurements and higher quality data management. It is principally correlated to computed tomography or magnetic resonance studies performed prior to specific cardiac procedures⁵. When comparing three-dimensional displays generated by AR or VR techniques to standard flat screens monitor displays data interpretation time is reduced with similar results in terms accuracy.

As a consequence, AR is being deployed for peri-procedural data visualisation with real-time three-dimensional rotational angiography or echocardiography data streaming from devices inside the surgical theatre ⁶ (Fig.2). Image / data manipulation using voice commands and hand gestures helps to establish a flexible and convenient workflow for physicians even throughout the procedure. A number of companies are developing AR technology for data streaming and holographic visualisation quality improvement  such as RealView Medical Imaging (Yokneam, Israel) ⁷,CarnaLife Holo - MedApp (Krakow, Poland) ⁸. Within the field of rehabilitation, MindMaze (Lausanne, Switzerland ) has applied VR technology to improve patient limb mobility in a post CVA setting ⁹.

Challenges and further works

It is clear that the promise presented by MR techniques in the medical field are almost infinite. Nevertheless, numerous iteration cycles will be fundamental before a comprehensive adoption can be considered. Among the current limitations are the concerns with precise stereoscopy views in VR and small view fields in AR devices. Conventional developments factors such as cost, size, weight and computer power will be detrimental to success.

Conflict of Interest

Zlahoda-Huzior and Maciej Stanuch are scientific developers for MedApp SA, Krakow, Poland. Dariusz Dudek is a member of the Scientific Board, MedApp SA, Poland.