Integrating Haptic and Visual Data for Low Latency transmission

Precisely determining the contact force during safe interaction in Minimally Invasive Surgery is still an open research challenge. Inspired by post-operative qualitative analysis from surgical videos, the use of cross-modality data driven deep neural network models has been one of the newest approach to predict sensorless force trends. The limited availability to vision-state-force datasets inspired the creation of a teleoperated laparoscopic forceps via robotic arm to recreate the forces of such tool when interacting with soft environments.

To solve this task, we use a laparoscopic forceps mounted on a robotic arm, that we teleoperate using a haptic device. We collect a dataset containing visual, robot state and force readings for different types of soft silicone environments. We make a deep analysis on how different artificial intelligence models predict forces, first on a randomized training scheme. As we have a big variability on our dataset, we test the adaptability of these models to different geometries, colors and stiffnesses, and how good the visual and state vectors help on this part. Addtionally, we determined the effectiveness of such models to adapt and learn from multiple dataset.

In the next part of my project, we decided to use generative deep learning techniques in order to model soft tissue dynamics from purely visual information. The approximation of such motion is done using modal coordinate analysis, really common in Finite Element Method simulations. We use the mathematical representation of the sinusoidal solutions on the frequency domain, using the Fast Fourier Transform, so we can further discretize the components and consider only a reasonable set of solutions.