Researchers Build an AI That Matches Surgery X-Rays to a Patient's 3D Scan in Seconds
In many small-incision operations, doctors steer thin tools through the body while watching flat X-ray images, and working out exactly where a tool sits in three dimensions is hard. Researchers at MIT and partner hospitals describe an AI program that lines those X-rays up with the patient's earlier 3D scan in seconds, to within less than a millimeter. It has been tested on existing patient records, not yet in live operations.
Researchers at MIT, working with doctors at Massachusetts General Hospital, Harvard Medical School and other institutions, published a paper today in the journal Nature describing an AI program meant to help surgeons know exactly where their instruments are inside a patient. They call it xvr.
The problem it tackles is common. Many operations are now done through tiny cuts instead of large ones: a doctor threads a thin tube or a small camera into the body and steers it while watching live X-rays on a screen. That is easier on the patient, but an X-ray is a flat picture, and from a flat picture it is hard to tell exactly where a tool is and which way it is pointing. "It takes decades of training for a clinician to become skilled enough to see grainy, 2D images and understand how everything is oriented," said Vivek Gopalakrishnan, the MIT researcher who led the work. Before surgery, most patients have already had a detailed 3D scan, such as a CT scan or an MRI. Lining up the live X-ray with that 3D scan would show the tool's true position, but doing it by hand, by typing in numbers or clicking on landmarks, is slow, and earlier AI tools for the job did not work reliably across different patients.
The new program takes a different approach. It was first trained on whole-body 3D scans from more than 2,000 people. Then, for each new patient, it spends about five minutes studying that one person's own scan before surgery, generating around a thousand simulated X-ray views of their body every second to practice on. After that, it can match a live X-ray to the patient's 3D scan in a matter of seconds, to within less than a millimeter. Because the practice images are calculated from the patient's real scan using the physics of how X-rays pass through the body, Gopalakrishnan said, "there is no room for hallucinations," meaning the program is not inventing details that are not there.
The team tested it on the largest collection of such cases available, real X-rays and scans from five hospitals covering adults and children and dozens of bones and organs. It beat other AI methods at the same task by a wide margin, and ran fast enough, the researchers say, to be useful even in emergency surgery.
It is not in operating rooms yet. The testing used records that had already been collected, and the researchers say they still need to make it faster for use during a live operation, check that it holds up in more situations, and teach it to handle parts of the body that move. All of the comments so far come from the team that built it. They say they are now working with surgical robotics companies and clinical groups to turn the research into tools doctors can use.