Publications highlights - Applications in Interventional Healthcare

Laparoscopic surgery has evolved as a key technique for cancer diagnosis and therapy. While characterization of the tissue perfusion is crucial in various procedures, such as partial nephrectomy, doing so by means of visual inspection remains highly challenging. We developed a laparoscopic real-time multispectral imaging system featuring a compact and lightweight multispectral camera and the possibility to complement the conventional surgical view of the patient with functional information at a video rate of 25 Hz. To enable contrast agent–free ischemia monitoring during laparoscopic partial nephrectomy, we phrase the problem of ischemia detection as an out-of-distribution detection problem that does not rely on data from any other patient and uses an ensemble of invertible neural networks at its core. An in-human trial demonstrates the feasibility of our approach and highlights the potential of spectral imaging combined with advanced deep learning–based analysis tools for fast, efficient, reliable, and safe functional laparoscopic imaging.

Ayala, L., Adler, T. J., Seidlitz, S., Wirkert, S., Engels, C., Seitel, A., Sellner, J., Aksenov, A., Bodenbach, M., Bader, P., Baron, S., Vemuri, A., Wiesenfarth, M., Schreck, N., Mindroc, D., Tizabi, M., Pirmann, S., Everitt, B., Kopp-Schneider, A., ... Maier-Hein, L. (2023). Spectral imaging enables contrast agent–free real-time ischemia monitoring in laparoscopic surgery. Science Advances, 9(10). [pdf]

Purpose: Validation metrics are a key prerequisite for the reliable tracking of scientific progress and for deciding on the potential clinical translation of methods. While recent initiatives aim to develop comprehensive theoretical frameworks for understanding metric-related pitfalls in image analysis problems, there is a lack of experimental evidence on the concrete effects of common and rare pitfalls on specific applications. We address this gap in the literature in the context of colon cancer screening.

Methods: Our contribution is twofold. Firstly, we present the winning solution of the Endoscopy Computer Vision Challenge on colon cancer detection, conducted in conjunction with the IEEE International Symposium on Biomedical Imaging 2022. Secondly, we demonstrate the sensitivity of commonly used metrics to a range of hyperparameters as well as the consequences of poor metric choices.

Results: Based on comprehensive validation studies performed with patient data from six clinical centers, we found all commonly applied object detection metrics to be subject to high inter-center variability. Furthermore, our results clearly demonstrate that the adaptation of standard hyperparameters used in the computer vision community does not generally lead to the clinically most plausible results. Finally, we present localization criteria that correspond well to clinical relevance.

Conclusion: We conclude from our study that (1) performance results in polyp detection are highly sensitive to various design choices, (2) common metric configurations do not reflect the clinical need and rely on suboptimal hyperparameters and (3) comparison of performance across datasets can be largely misleading. Our work could be a first step towards reconsidering common validation strategies in deep learning-based colonoscopy and beyond.

Tran, T. N., Adler, T. J., Yamlahi, A., Christodoulou, E., Godau, P., Reinke, A., Tizabi, M. D., Sauer, P., Persicke, T., Albert, J. G., & Maier-Hein, L. (2023). Sources of performance variability in deep learning-based polyp detection. International Journal of Computer Assisted Radiology and Surgery, 18(7) [pdf].

Purpose: As human failure has been shown to be one primary cause for post-operative death, surgical training is of the utmost socioeconomic importance. In this context, the concept of surgical telestration has been introduced to enable experienced surgeons to efficiently and effectively mentor trainees in an intuitive way. While previous approaches to telestration have concentrated on overlaying drawings on surgical videos, we explore the augmented reality (AR) visualization of surgical hands to imitate the direct interaction with the situs.
Methods: We present a real-time hand tracking pipeline specifically designed for the application of surgical telestration. It comprises three modules, dedicated to (1) the coarse localization of the expert's hand and the subsequent (2) segmentation of the hand for AR visualization in the field of view of the trainee and (3) regression of keypoints making up the hand's skeleton. The semantic representation is obtained to offer the ability for structured reporting of the motions performed as part of the teaching.
Results: According to a comprehensive validation based on a large data set comprising more than 14,000 annotated images with varying application-relevant conditions, our algorithm enables real-time hand tracking and is sufficiently accurate for the task of surgical telestration. In a retrospective validation study, a mean detection accuracy of 98%, a mean keypoint regression accuracy of 10.0 px and a mean Dice Similarity Coefficient of 0.95 were achieved. In a prospective validation study, it showed uncompromised performance when the sensor, operator or gesture varied.
Conclusion: Due to its high accuracy and fast inference time, our neural network-based approach to hand tracking is well suited for an AR approach to surgical telestration. Future work should be directed to evaluating the clinical value of the approach.

Müller, L.-R., Petersen, J., Yamlahi, A., Wise, P., Adler, T. J., Seitel, A., Kowalewski, K.-F., Müller, B., Kenngott, H., Nickel, F., & Maier-Hein, L. (2022). Robust hand tracking for surgical telestration. International Journal of Computer Assisted Radiology and Surgery, 17(8), 1477–1486. [pdf]