Researchers at NJIT are exploring how a combination of physiological sensors, AI and extended reality technologies can help to increase student persistence and educational success.
In the concept, Minds and Machines: A Neuro-Adaptive AI Learning Companion, body distress signals are used to identify when students might fall behind their peers, or when they’re hyperengaged and ready for new challenges.
The process begins with students wearing sensors, such as a wrist device or even a headset. Sensors would check for physical indicators such as overloaded brain waves, via electroencephalogram, during lectures or other instructional settings.
When triggered by these indicators, AI decides alternate ways of addressing the topic, based on pre-programmed knowledge of educational best practices and student motivations. The output could be delivered in real-time or later, and through augmented or virtual reality, to help bring concepts to life.
Systems like this could also be used outside of classrooms in experiential learning settings, such as NJIT’s makerspace environment, explained neuroscientist Jessica Simon, who currently serves the university as pedagogical and instructional innovation specialist at the Center for Educational and Innovation and Excellence. She's joint principal investigator on the $50,000 university-funded grant, along with Zhifeng Kou, associate chair of biomedical engineering.
In educational research, “What's been shown is that when you do receive personalized teaching, you can increase your performance by at least two standard deviations,” Simon said. “We can use neural activity in the classroom. This is an emerging field in education. In the intersection of education and neuroscience, because both are relatively young, we really haven't been focused as much on the empirical, evidence-based sciences that could improve our students' learning outcomes.”
“The ideas come from incremental changes. I've been using AI to do biomedical research, and in particular biometric imaging analysis for years. So when these large language models came out, my first reaction was that we could build something to help the students to learn,” Kou added.
The team now faces a major challenge in making all of the parts of their system work together. They must master how exactly their software will communicate between brain signals and custom instruction. Individual signals are reliable, and language model integration is becoming routine, but it’s the juggling of physical sensors vs. background electrical noise, multiple signals, and real-time software customization — all happening in a live classroom, not just a laboratory — that will make for a tough task. They also must consider whether students are comfortable sharing body data, and they must ensure the data stays private.
When the technical issues are solved, then they’ll work on turning this research into a product. “In a lab is one thing. We want to deliver a real product, not just being used by NJIT. We could share the technology with other universities, so that many college students can really benefit from this," Kou noted. “We are not just scientists. We really want to build something truly translational and impactful.”
Co-principal investigators include NJIT’s Nicole Bosca, director, Center for Educational Innovation and Excellence; Elisa Kallioniemi, assistant professor, biomedical engineering; Bryan Pfister, chair and professor, biomedical engineering; Mengjia Xu, assistant professor, data science; and Lin Zhao, assistant professor, biomedical engineering.
