Augmented reality, superhuman abilities and the future of medicine
Missed a session of GamesBeat Summit 2022? All sessions are now available to stream. Watch now.
Earlier this month, I participated as a panelist at the Digital Orthopedics Conference in San Francisco (DOCSF 2022) where an important theme was to envision the medical profession in the year 2037. In preparation for the event, a small group of us reviewed the latest research on the clinical use of virtual and augmented reality and critically assessed the current state of play.
I have to admit that I was deeply impressed by how far augmented reality (AR) has progressed for use in medicine over the past year and a half. So much so that I don’t expect to have to wait until 2037 for AR to have a major impact on the field. In fact, I predict that by the end of this decade, augmented reality will become a common tool for surgeons, radiologists, and many other medical professionals. And by the early 2030s, many of us will go to the GP and be examined by a doctor wearing AR glasses.
The reason is simple:
Augmented reality will give doctors superpowers.
I’m talking about superhuman capabilities for visualizing medical images, patient data, and other clinical content. The costs associated with these new capabilities are already reasonable and will fall rapidly as augmented reality hardware is produced in higher volumes in the coming years.
The first superpower is X-ray vision.
Visualization and augmented reality concept.
Augmented reality gives doctors the ability to look directly into a patient and see evidence of trauma or disease at the exact location in their body where it is. Of course, the ability to see under the skin already exists with tools such as CT and MRI scanning, but currently doctors view these images on flat screens and have to imagine how the images relate to the patient on the table. This kind of mental transformation is an impressive skill, but it takes time and cognitive effort, and isn’t nearly as informative as it would be if doctors could just stare into the human body.
With AR headsets and new techniques for recording 3D medical images on a patient’s real body, the superpower of X-ray vision is now a reality. In an impressive study from the Teikyo University School of Medicine in Japan, an experimental emergency room was tested with the ability to capture full-body CT scans of trauma patients and immediately enable the medical team, all with AR headsets, into the patient on the examination table and see the trauma in the exact location where it is located. This allowed the team to discuss injuries and plan treatment without having to refer back and forth to flat screens, saving time, reducing distractions and eliminating the need for mental transformations.
In other words, AR technology takes medical images off the screen and places them in 3D space at the exact location where they are most useful to doctors – perfectly aligned with the patient’s body. Such an ability is so natural and intuitive that I predict it will soon be applied in medical applications. In fact, I expect that in the early 2030s doctors will look back at the old way, looking back and forth at flat screens, as clumsy and primitive.
Augmented reality technology goes beyond X-ray vision to provide physicians with supportive content placed on (and in) the patient’s body to aid them in clinical tasks. For example, surgeons performing a delicate procedure are given navigational cues that are projected onto the patient in real time, showing the exact location where the interventions need to be performed with precision. The goal is to increase accuracy, reduce mental effort and speed up the procedure. The potential value for surgery is extreme, from minimally invasive procedures such as laparoscopy and endoscopy to freehand surgical efforts such as placing orthopedic implants.
The concept of enhanced surgery has been a pursuit of AR researchers since the core technologies were first invented. In fact, it goes back to the first AR system (the Virtual Fixtures platform) developed at the Air Force Research Laboratory (AFRL) in the early 1990s. The aim of that project was to demonstrate that AR can increase human dexterity in precision tasks such as surgery. As someone involved in that early work, I have to say that the progress the field has made over the past few decades is remarkable.
Think about this: When testing that first AR system with human subjects in 1992, we required users to move metal pins between holes 60 cm apart to quantify whether virtual overlays can improve manual performance. Now, thirty years later, a team from Johns Hopkins, Thomas Jefferson University Hospital and Washington University performed delicate spinal surgery on 28 patients using AR to help place metal screws with a precision of less than 2 mm. . as published in a recent studythe screw delivery system achieved such accurate registration between the real patient and the virtual overlays, surgeons scored 98% on standard performance stats.
Looking ahead, we can expect augmented reality to affect all aspects of medicine as its precision has reached clinically achievable levels. In addition, major breakthroughs are in the works that will make it faster and easier to use AR in medical environments. As described above, the biggest challenge for any precision augmented reality application is the accurate recording of the real world and the virtual world. In medicine, this currently means applying physical markers to the patient, which takes time and effort. In a recent study from Imperial College London and University of Pisa, researchers tested a “markerless” AR system for surgeons that uses cameras and AI to accurately align the real and virtual worlds. Their method was faster and cheaper, but not as accurate. But this is still early – in the coming years, this technology will make AR-assisted surgery viable without the need for expensive markers.
In addition, camera-based recording techniques will take AR out of highly controlled environments such as operating theaters and bring it to a wider range of medical applications. In fact, I predict that by 2030, GPs will often see patients with the benefit of: AR headsets†
This brings me to another superpower I expect from doctors in the near future: the ability to look back in time. That’s because doctors can capture 3D images of their patients with AR headsets and later view those images aligned with their patient’s body. For example, a doctor can quickly assess the healing progress of a skin lesion by examining the patient through AR glasses, interactively peering back and forth in time to compare the current view to what the lesion looked like during previous visits.
Overall, the progress made by researchers in the medical use of virtual and augmented reality is impressive and exciting, with significant implications for both medical education and practice. to dr. Stefano Bini from to quote UCSF Department of Orthopedic Surgery“the beneficial role of AR and VR in the upskilling of healthcare personnel cannot be overstated.”
I agree with dr. Bini and would go even further as I see augmented reality extending the workforce way beyond healthcare. After all, the superpowers of X-ray vision, navigation cues, agility support, and the ability to look back in time will be useful for everything from construction and auto repair to engineering, manufacturing, agriculture, and of course, education. And with AR glasses developed by some of the largest companies in the world, from Microsoft and Appleuntil meta† google† Magic Leap† HTC and snapthese superpowers will almost certainly come to mainstream consumers within the next five to 10 years, improving all aspects of our daily lives.
Louis Rosenberg, PhD is CEO and chief scientist of Unanimous AI and has been awarded more than 300 patents for his work in VR, AR and AI.
Welcome to the VentureBeat Community!
DataDecisionMakers is where experts, including the technical people who do data work, can share data-related insights and innovation.
If you want to read about the very latest ideas and up-to-date information, best practices and the future of data and data technology, join us at DataDecisionMakers.
You might even consider contributing an article yourself!
Read more from DataDecisionMakers