Apple Highlights Stryker, an NJIT Alum's Medical Tech Group, for VisionPro App

Robert Cohen, a three-time New Jersey Institute of Technology alumnus and current chair of the university’s Board of Trustees, leads a digital, robotics and enabling technology division at medical technology firm Stryker Corp. that grabbed Apple’s attention this month for its work with the new VisionPro mixed-reality headset.

VisionPro headsets put screens and software directly in front of a physician’s eyes. Stryker’s Vision Pro application is called MyMako. It lets surgeons plan and simulate pre-planning for robot-assisted knee replacement operations before they happen, and view the plans in 3-D from anywhere there’s an Internet connection.

The potential for extending Vision Pro into real-world operating rooms can provide new information about a case, letting doctors keep their eyes on the patient instead of external monitors. Training surgical residents with Vision Pro in unprecedented realistic situations is now also possible, Cohen said.

“The MyMako app for Apple Vision Pro allows surgeons the ability to access intricate surgical plan details and insights at their fingertips in a 3D-native, intuitive and dynamic way. This level of insight — anytime, anywhere — was previously not possible,” Cohen stated in an Apple announcement.

“Super cool,” Cohen said, about Apple’s support. Stryker already has software on the market for Microsoft’s HoloLens augmented reality headset and VisionPro is a continuation of that development strategy, he explained.

“Each of these platforms are unique … HoloLens, you're actually looking through the lens and we're superimposing images. VisionPro, everything's a 4K-resolution camera,” he noted. “Lots of great uses we are considering. For example, with a physician in surgery during the actual case, by using Vision Pro and MyMako I could FaceTime in and the surgeon will see me. I'm looking at what the surgeon’s looking at.”

Cohen is in a good position to understand the new technology. At NJIT, he earned a B.S. and M.S. in mechanical engineering, and a second M.S. in engineering management. Drawing on his academic background and long career in medical technology, he’s excited about mixed-reality headsets.

“You get more information on a patient and it can be visualized digitally. Right now, it is a challenge to display to a surgeon so much information on two dimensional monitors, because we can’t provide it in a usable way. Just think about CT scans and three-dimensional virtual models of a patient’s actual bones. If I'm looking at your knee, let's say a total knee replacement, your knee is different than my knee, what's the back end look like? What’s the front of your knee look like, the sides? You almost get brain overload, right? So you try to present a visual interpretation and a data interpretation. Then when you do a visual interpretation, where's your joint line? You could get visual overload.”

But with mixed reality, “You get to look at the three-dimensional bone digital model of the knee from the sides, back and front. Now you could rotate this and say, ‘Okay, that's exactly this person's joint line. That's where I'm going to put my prostheses, and this person's legs are out of alignment or they’re walking bow legged, how am I going to change the alignment and still keep the joint line?’ These are all things that now, with six degrees of virtual freedom, you can move around the virtual models of the implants and visualize it in three dimensions so you can synthesize and act on information.”

Cohen said that he imagines artificial intelligence could eventually be added into the mixed reality equation. For example, he said, AI software could make recommendations to doctors based on patient data. By using a mixed-reality headset, he noted, “It could be a vehicle to display AI information. So if you want to do automatic planning one day, we have AI which can now take the phenotype of the patient, all the characteristics of the patient and look at all those characteristics and then say, ‘Here's a recommended placement location of the implant,’ giving the surgeon an advanced starting point. Based on that person's bone quality, comorbidities and body weight, can AI select the right implant and place the right implant? That's a great potential for innovation for the R&D teams.”

Whether it’s orthopedic implants or almost any other field — from architecture to computer science to physics — almost everyone can benefit from learning to use augmented and virtual reality tools and robotics, Cohen said. “Vision Pro is a fundamental tool. It's a technology block that students and faculty should be embracing. It provides us more opportunity to do more things with information than we ever had available to us or access before. And it's really exciting.”

“But the technology by itself, like VisionPRO, is nothing without smart people and smart students who figure out how to use it. It’s people that will bring innovative applications that actually benefit society in amazing ways. And right now, this is so cutting edge. This mixed reality is a core technology that students should learn on and know.”