Pingyu Xiang

Safe operation in visually obstructed anatomy relies on accurate in vivo localization of medical devices, yet current magnetic systems struggle to adapt across device types and clinical environments. Here we present a portable and flexible patch that functions as an intermediary in a dual-stage localization paradigm of a source-sensors-source configuration, enabling three-dimensional localization in vivo with mean positional errors under 300 micrometres and orientation errors under 0.3 degrees. The patch can be customized in shape and size to accommodate diverse medical scenarios requiring precise localization. We validate its flexibility and versatility through systematic characterization, and through in vitro and in vivo studies in vascular and gastrointestinal surgery, where the system maintained continuous magnetic capsule tracking for 6 hours. Our magnetic localization patch could potentially broaden the compatibility and adaptability of magnetic localization across various medical applications.

Magnetic localization, owing to its immunity to line-of-sight occlusion and noncontact nature, is considered a promising technology for medical applications. While it is intuitive that localization performance degrades as the target moves farther from the sensor array, uncertainty analysis has long been overlooked, which is essential for quantifying localization quality. In this work, we present a pose estimation and uncertainty analysis framework on SE(3) for magnetic source localization using sensor arrays, which enables concise formulation of the problem and quantitative assessment of the results. The volume of the uncertainty ellipsoid is used to characterize localization confidence, while the surface shape in Cartesian space is used for visualization. This also provides insight into the effective workspace of the magnetic localization system prior to deployment. Guided by this analysis, we designed a movable magnetic sensor array to expand the limited sensing volume and mitigate localization uncertainty, thereby enhancing overall localization performance. Simulations and pose tracking experiments validate the effectiveness of this framework. By dynamically moving the sensor array to minimize the volume of the uncertainty ellipsoid, localization errors are reduced compared with static and projection-based strategies by 67.04%, 16.51% in position and 43.87%, 20.73% in orientation, respectively. Furthermore, phantom experiments on distal locking screw alignment and magnetic capsule endoscope tracking demonstrate the system’s capability in improving localization accuracy (reducing alignment errors by 71.88%) and expanding the effective workspace by several folds.

Slender medical continuum robots with flexibility and highly redundant degrees of freedom are widely used in various minimally invasive surgery. However, when interacting with anatomical structures, the continuum robot adopts diverse shapes, posing challenges for operation and control. To achieve real-time intraoperative shape sensing and provide online guidance for manipulation, most existing methods rely on optical fibers embedded within the robot, which often require specialized robot designs and come with high costs. Here, we present a novel approach utilizing thin and flexible carbon nanotube piezoresistive fibers as a bandage, helically integrated on the surface of existing slender medical continuum robots for shape sensing. The spatial configuration of the robot is effectively inferred by downsampling the resistance changes along the robot’s body and applying a learning-based method. The results demonstrate that the proposed helically arranged carbon nanotube piezoresistive fibers, combined with a data-driven approach, are capable of reconstructing the robot’s spatial shape. In vitro and ex vivo experiments on animal tissues further highlight its promising potential for enhancing the shape-sensing ability of existing medical continuum robots.

The unequal distribution of medical resources and scarcity of experienced practitioners confine access to bronchoscopy primarily to well-equipped hospitals in developed regions, contributing to the unavailability of bronchoscopic services in underdeveloped areas. Here, we present an artificial intelligence (AI) co-pilot bronchoscope robot that empowers novice doctors to conduct lung examinations as safely and adeptly as experienced colleagues. The system features a user-friendly, plug-and-play catheter, devised for robot-assisted steering, facilitating access to bronchi beyond the fifth generation in average adult patients. Drawing upon historical bronchoscopic videos and expert imitation, our AI-human shared control algorithm enables novice doctors to achieve safe steering in the lung, mitigating misoperations. Both in vitro and in vivo results underscore that our system equips novice doctors with the skills to perform lung examinations as expertly as seasoned practitioners. This study offers innovative strategies to address the pressing issue of medical resource disparities through AI assistance.

Small-scale continuum robot-assisted minimally invasive surgery has received crucial attention due to its smaller incisions and high dexterity. In medical scenarios such as radiofrequency ablation and nasal/throat swab sampling, monitoring and controlling the forces applied to human tissue can help improve the safety and comfort level of the procedure. However, the tip-sensor-based force detection method can barely be deployed due to the miniature size of the continuum robot; meanwhile, the mechanical modeling-based high-precision force estimation cannot be realized on account of the continuum robots’ complex structure with high nonlinear properties. To address the high-precision force estimation challenge for further compliant control during minimally invasive interventions, a learning-based high-precision force estimation method via long short-term memory (LSTM) is proposed in this paper. On this basis, compliance control and high-precision force tracking can be further realized for small-scale continuum robot. The compliance control ensures a smooth and stable transition during the interaction between the robot and the environment, and force tracking can be utilized for maintaining or precisely controlling the force applied to the human tissue. Finally, the contact force sensing and control experiments are carried out on a small-scale continuum robot system prototype, and a demonstration using a human nasal cavity model is conducted. The results validate that the proposed LSTM neural network fits the mechanical model of the continuum robot well with the root mean square error of 3.44mN, and the control method can significantly compensate for the instantaneous impact during contact with an attenuation of 59.3% and rapidly respond to keep the force accurately at the expected value with the mean absolute error of 2.41mN. Note to Practitioners—This research is motivated by the increasing number of applications for small-scale continuum robot-assisted minimally invasive interventional surgery, such as radiofrequency ablation, biopsy, and endoscopic submucosal dissection. These surgeries require high-precision contact force sensing and control. However, due to the small size and complex structure of the continuum robot, traditional methods such as installing force sensors and mechanical modeling are not effective. Therefore, this paper utilizes a neural network to infer contact force through more accessible information about the robot, such as the tension of the actuators. This method allows for compliant force control during the dynamic contact between the small-scale continuum robot and human tissue. Experiments conducted on a human nasal cavity model demonstrate that the proposed method can improve the safety and reliability of small-scale continuum robot-assisted minimally invasive surgery.

Soft continuum robots, attributable to inherently compliant trunks and shape manipulability, have been widely deployed in complex scenarios requiring safe human-robot interaction. However, their nonlinear deformations and hyperredundant degrees of freedom pose substantial challenges for full-body shape sensing and closed-loop control of the end effector. A low-cost yet accurate feedback solution is thus highly desirable. To address this, we present a hydraulic driven reciprocating magnet strategy, integrated with magnetic localization, to enable both shape sensing and tip pose estimation of soft continuum robots, thereby facilitating precise closed-loop control. The proposed approach time-multiplexes a single magnet under different operational phases to fulfill two functions: full-body shape reconstruction and tip pose tracking. We validate the effectiveness of the reciprocating magnet system on a pneumatic manipulator prototype with two active degrees of freedom. Experimental results show that the magnet can travel through the guide channel at a maximum speed of 6.5 cm/s, achieving average errors of less than 2 mm in position (1.1% of the robot’s length), 3° in orientation for shape sensing and tip pose estimation. Using this sensing strategy, we demonstrate a simple closed-loop control on the soft continuum robot. Owing to its simplicity, low cost, and high precision, the proposed method holds promise as a practical alternative for state feedback in soft continuum robots.

Magnetic continuum robots are subject to external magnetic fields and deformed remotely, simplifying the robot’s transmission mechanism and providing it with significant potential for miniaturization and operational flexibility. However, modeling magnetic field distribution generated by permanent magnets is complex and requires time-consuming pre-calibrations. Moreover, it is highly susceptible to environments with ferromagnetic materials, posing significant challenges for the control of magnetic continuum robots. In response, we propose an approach that does not overly focus on the magnetic field distribution but instead directly learns the inverse kinematics of magnetic continuum robots end-to-end. Binding the robot’s configuration to the pose of external magnets, precise control of continuum robots is facilitated. Additionally, we leverage teleoperation techniques to broaden the applicability of this method. By mounting magnets on a robotic arm and directly utilizing the target pose of the external magnet predicted by a multi-layer perceptron (MLP), we achieve the operation and navigation of magnetic continuum robots in complex environments. Experiments demonstrate that the mean control accuracy along the robot using our learning-based inverse kinematics is about half of the robot’s diameter.