Context Aware Augmented Reality for Dental Applications – Interview with Darko Katic

Dental surgical procedures are quite interesting to be supported by Augmented Reality. Many steps in their workflows incorporate 3D data e.g. from CBCT or intra oral scanners for complex 3D tasks such as diagnostics, preoperative planning and intraoperative navigation. However, there are still only few Augmented Reality systems out there targeting this application domain.

Darko Katic of the Institute for Anthropomatics (IFA), Humanoids and Intelligence Systems Laboratories (HIS), Karlsruhe Institute of Technology (KIT), Germany addresses dental applications with his research projects. He developed an Augmented Reality system that incorporates a preoperative planning procedure as well as intraoperative navigation. In addition to that, he embedded the Augmented Reality visualization in a context aware framework, which allows him to automatically determine relevant phases of a surgical workflow in order to adapt the visualization to the requirements of this phase. Next to the dental applications, Mr. Katic applies context aware Augmented Reality to laparoscopic procedures. I’m happy to post an interview with Mr. Katic today that has been held on September 18, 2013.

Dear Mr. Katic, can you provide me with a little more information about the motivation of using Augmented Reality in dental surgery?
Dental surgery is of special interest for Augmented Reality for a number of reasons. For once, Augmented Reality allows the surgeon to keep an eye on the patient for the entire time without having to look away to get information from monitors. This helps reducing the duration of surgeries and improving safety, especially in the dental domain, where precision and a clear view on the patient is crucial. Secondly, a prominent issue in dental surgery is the presence of vital structures, such as the nervus alveolaris, which are situated beneath the surface. Augmented Reality allows surgeons to see these structure, which otherwise would be occluded beneath bone tissue. Coupled with the stereoscopic view offered by goggles, this can facilitate orientation and risk avoidance. Finally, we mainly deal with rigid structures in dental surgeries. In contrast to visceral surgery with soft tissues or neurological surgery with brain shifting, it is easier to deal with registration and object tracking in this case. We therefore can focus on the visualization and assistance function and more easily show the benefits of Augmented Reality. This allowed us to gain experience and raise confidence in Augmented Reality with clinicians, which is beneficial for our current work in laparoscopic surgery.

Who are your clinical partners?
We closely work together with our medical partners at the University Hospital in Heidelberg. They were the ones, who identified situations, which require special visualization. In the case of dental surgery, we work together with clinicians from the Department of Cranio-Maxillofacial Surgery at the University of Heidelberg. Our work on laparoscopic surgery is done in cooperation with the Department of General, Abdominal and Transplantation Surgery at the University of Heidelberg, within the setting of the SFB/Transregio 125 Cognition-Guided Surgery and the Research Training Group 1126 Intelligent Surgery.

What are the key situations that you filter to show Augmented Reality visualizations?
The main criterion for the inclusion of certain visualizations is that it is likely to help improve patient outcome as well as safety. For dental implant surgery, to name a few examples, we decided to assist during the drilling process. To avoid risky situations, we furthermore display the position of vital structures, concealed beneath tissue, in relation to the drill.

Can you describe what kind of information is superimposed on the patient and in which format?
We queried our medical partners about their information needs in each specific situation. In the case of dental surgery, we mainly display navigational information, whereas in laparoscopy the situations and information needs are more manifold. In case of dental implant surgery, we aim to improve the precision by visually relating the intended position of the dental implant and the current position of the dental drill we aim to improve precision. We also issue warnings about endangered vital structures. For each of the visualizations, we defined two different alternatives: one with Augmented Reality overlay of virtual images on the corresponding anatomical structure and one via the HUD paradigm. The idea is that while the Augmented Reality view is more intuitive, the HUD version allows for a clearer view on the patient since the area of interest is not occluded by virtual imagery. For each of these visualizations, the surgeon can decide, which version he likes best. For instance the relation between the intended position and the current position of the dental drill can be displayed by overlaying the virtual axis and marking the position with a virtual representation of the intended implant in the mandible. It can also be displayed in HUD fashion as an overhead view of the scene in a fixed position in the surgeon’s field of vision. Colors are also used to convey information, such as changing hue to emphasize the increasing degree of risk as the drill approaches some risky structure. This helps visual understanding and proved to be intuitive in our trails.

Where is the clinical benefit of using Augmented Reality in these key situations compared to workflows that do not use Augmented Reality?
For dental implant surgery, we see the benefits in improved ergonomics. The surgeon gets his information right in his field of view in an intuitive way. This is enhanced by our context-aware approach, where the visualizations automatically change to best accommodate the current needs of the surgeon. Combined, this allows the surgeon to make full use of all available information, which is relevant for the task at hand with minimal effort to collect this kind of information. This leaves more mental space to act and focus on the actual task, the performance of the surgery. Especially, in the case of dental interventions, the Augmented Reality approach allows for greater flexibility, since the pre-operative planning is readily available intraoperatively, yet the surgeon is free to deviate from it if new circumstances are encountered. In the case of laparoscopic surgery, we see even greater potential for Augmented Reality since the situation can get more complex as more instrument types and anatomical structures need to be dealt with. With more information being available it becomes more difficult to maintain situational awareness and pick up subtle visual cues in endoscopic images. Augmented Reality can help in such situations by highlighting important events in the image data. It can also be very beneficial by showing blood vessel underneath tissue which would be invisible otherwise.

What is your opinion regarding ergonomic advantages of Augmented Reality in this application scenario?
We did see ergonomic advantages to using Augmented Reality. Most of all, it allows the surgeon to see all information right in his field of view, without the need to turn his head towards various conventional displays. However, at least for the time being, our surgeons still prefer looking directly into the patient’s oral cavity. This is a safety precaution since it allows direct visual verification of what is currently happening at the region of interest. Looking through the facial skin is therefore used for orientation, yet not during the actual drilling. Future development, however, are likely to increase confidence in the system and allow for such changes in the workflow of the surgery.

What role does Augmented Reality supported navigation play for you?
Augmented Reality navigation, in my opinion, has great potential for future uses. There is more and more information available to surgeons to better treat their patients, yet the interfaces to access this information have not developed at the same pace. To really make use of all available data such as preoperative planning information and the status of devices in the OR, new man-machine interaction techniques are necessary. With its intuitive capability of displaying information and quickly making it available to the surgeon, Augmented Reality has great potential in this regard. However, it might not solve the entire problem. With more and more information becoming available, there is also an increased risk of informational overflow. Context-aware systems aim to recognize the current state of the surgery and automatically adapt the display. They act as intelligent information filters by selectively displaying just those pieces of information which are currently relevant for the surgeons. Augmented Reality systems combined with context-awareness are a promising way to aid in surgical navigation tasks as they provide targeted and relevant information quickly and efficiently to the surgeon.

What about using Augmented Reality for pre-interventional planning?
We did some work on pre-interventional planning of dental implants in the mandible yet ran into problems with tracking accuracy since it is vital to plan the positions with sub millimeter accuracy. Augmented Reality can also be used prior to the intervention to better understand the patient specific anatomy and prepare for the intervention. Still, our work is focused on intraoperative applications since we believe that it is there where the quick and intuitive visual understanding provided by Augmented Reality shows its greatest benefits.

Thank you for your time!

  • D. Katić, G. Sudra, S. Speidel, G. Castrillon-Oberndorfer, G. Eggers, and R. Dillmann. 2010. Knowledge-based situation interpretation for context-aware augmented reality in dental implant surgery.
    In Proceedings of the 5th international conference on Medical imaging and augmented reality (MIAR’10), Hongen Liao, P. J. Edwards, Xiaochuan Pan, Yong Fan, and Guang-Zhong Yang (Eds.). Springer-Verlag, Berlin, Heidelberg, 531-540.