NJIT Experts in Augmented Reality Help Train Caretakers for the Elderly

With the world's population of geriatric patients increasing faster than enough caretakers can be trained for the difficult job, NJIT Assistant Professor of Informatics Salam Daher thinks augmented reality technology may help close the gap.

Daher and her students are prototyping a digital model of an older person which is aware of its feelings and environment. Existing models only cover physical aspects, so it's opening new ground to have a patient simulator that teaches caretakers about the emotional and psychological aspects of their daily work.

This could attract more people to the field. People who are studying to become certified nurse assistants will interact with the patient through virtual reality or projections. The patient will be controlled initially by teachers and eventually through artificial intelligence.

For training purposes and patient outcomes, "One day, 30 or 40 years from now, I might be on the other end of it so I better do a good job now," Daher said. "We want to do training that improves communications, empathy and perceptions. We want to create a proof of concept for this type of training, and use it to investigate if it makes a difference."

The project will also be significant for pushing the limits of augmented reality. Daher said their software will cause the virtual patient to remember conversations or take actions such as turning off a television when a nursing student enters the room — "It's a new class of virtual agents … They may make comments that give the illusion that they're aware of you or the environment around them, as opposed to this is canned or unaware of what's going on."

She's the right person for the job, as an award-winning specialist in healthcare simulation who previously developed mock patients using digital assistants, interactive video and physical models. Her latest work is funded by a $110,000 grant from the National Science Foundation. Her co-principal investigator is Distinguished Professor Julie Ancis. Collaborators include the University of Delaware and University of Central Florida.

Behind the scenes, Daher's team is developing their software in Unity, which is popular for applications such as mobile phone games. Latency could be a technical challenge, because realism would suffer if patients respond too slowly. Inherent limitations of artificial intelligence could also be problematic, especially if caretakers go off-script. Considering that humans often have misunderstandings with each other, asking software to understand body language, humor or nuance is a tall order, she noted.

Kimia Naeiji, a senior information technology major from Westfield, is working on the 3D modeling. She grew up wanting to become a dentist before discovering her passion for software, and said she is happy to have found a way to combine her interests in healthcare and technology. She also came to understand the importance of elderly care when her grandmother had a stroke.

Naeiji uses software called Maya to build her model. "It's pretty interesting. I learned a lot about 3D characters. I learned a lot about bones, joints and how I can make a character look realistic," she said.

The character is female but does not yet have a name. "We definitely have to give it a name. When it goes to the VR and doctors and nurses talk to this person, they have to call it something. I am currently working on the texture. I will also improve the rigging and gestures."

After graduation, Naeiji wants to study for a master's degree in computer science and specialize in artificial intelligence.