Toward Detection and Localization of Instruments in Minimally Invasive Surgery

TLDR

Detecting and localizing surgical instruments in laparoscopic images is essential for advanced robotic and computer‑assisted interventions, yet robotic encoders lack accuracy from the surgeon’s perspective, making vision sensors a promising alternative for determining instrument position in the camera’s coordinate frame. This study proposes a vision algorithm that localizes the instrument’s 3‑D pose, leaving only rotation around the shaft axis ambiguous, and introduces a probabilistic supervised classification method to detect tool pixels in laparoscopic images. The algorithm initializes an energy‑minimization pose estimation of a prior 3‑D instrument model within a level‑set framework using the classifier output. The method proves robust to noise in simulated data, quantitatively matches ground‑truth optical tracker measurements, and successfully applies to in vivo minimally invasive surgery with both laparoscopic and robotic instruments.

Abstract

Methods for detecting and localizing surgical instruments in laparoscopic images are an important element of advanced robotic and computer-assisted interventions. Robotic joint encoders and sensors integrated or mounted on the instrument can provide information about the tool's position, but this often has inaccuracy when transferred to the surgeon's point of view. Vision sensors are currently a promising approach for determining the position of instruments in the coordinate frame of the surgical camera. In this study, we propose a vision algorithm for localizing the instrument's pose in 3-D leaving only rotation in the axis of the tool's shaft as an ambiguity. We propose a probabilistic supervised classification method to detect pixels in laparoscopic images that belong to surgical tools. We then use the classifier output to initialize an energy minimization algorithm for estimating the pose of a prior 3-D model of the instrument within a level set framework. We show that the proposed method is robust against noise using simulated data and we perform quantitative validation of the algorithm compared to ground truth obtained using an optical tracker. Finally, we demonstrate the practical application of the technique on in vivo data from minimally invasive surgery with traditional laparoscopic and robotic instruments.