Researchers Find Ethical Risks Where Mixed-Reality Gaming Meets AI

Researchers at NJIT and Universidad Carlos III of Madrid wanted to know if mixed-reality games could be improved by handing control to artificial intelligence software in the form of conversational language applications — it turns out they can, but the results may put players in danger.

The question stems from an annual Halloween tradition of serious-yet-silly experiments by Assistant Professor Niccolo Pescetelli, of NJIT’s College of Science and Liberal Arts humanities department, researcher Manuel Cebrian of Carlos III de Madrid’s statistics department and Santander Big Data Institute, along with student assistants at both schools. Their favorite example is from 2018 when they gave control of Pescetelli’s movements and actions to random Internet users, so that he might better solve puzzles.

This year’s project resulted in a working paper, Proof of Concept for AI-Driven Alternate and Augmented Reality Games, published on both professors’ websites. With the combined term AARG, they define alternate reality as narrative-driven experiences using the real world as a platform, like Global Thermonuclear War in Wargames, and augmented reality as overlaying digital information onto the real world, as in Pokémon Go.

The team performed seven tests of prompting their language model to develop a game narrative for real-world situations, such as being stranded in a parking lot in a large city.

“The conversational assistant demonstrated considerable skill at analyzing textual and visual inputs to dynamically construct customized AARG games tied to specific users and locations. The assistant proved adept at crafting cohesive narratives, puzzles and challenges that responded in real-time to participant actions. This points to an exciting new frontier for interactive, social storytelling across virtual and physical spaces,” they wrote.

They said one of the primary limitations is that the software sometimes lost its train of thought. “The model struggled to keep the original narrative in focus without being swayed by more recent prompts. The ability to maintain top-level goals in working memory while adapting low-level action plans to reach sub-goals is a key attribute of human cognitive control,” they added.

However, when the researchers instructed their language model that “this is not a game,” the model sometimes took that guidance too seriously. “A key issue was the potential for the AI assistant to prioritize crafting an intense experience while sacrificing user consent and agency. For example, the engrossing augmented reality game concepts could lead players to misinterpret reality and lean on conspiratorial thinking, without their consent. Similarly, the Blair Witch Project-inspired psychological horror experience promoted fear, based on user actions without extensive concern for their comfort. … More research is imperative to develop models that align with human values.”

“There were also indications that the assistant’s world model may lack sufficient understanding of ethics, legality and social norms to guide responsible AARG game development alone. This necessitates proper oversight and fail-safes by accountable humans with contextual judgment. It also underscores the need for AI creators to prioritize ethics and human well-being in the development process.”

They continued, “If the future development of AI-driven AARGs will be able to formulate robust ethical frameworks to protect user privacy, obtain informed consent, and promote psychological well-being, incredible creative potential will be unlocked. This requires collaborative dialogues between AI researchers, game developers, policymakers, and the public to address AI’s societal and regulatory implications in AARGs. Conducting user studies is crucial for understanding player preferences and the psychological impacts of AARGs.”

Pescetelli said his career began in psychology and morphed from neuroscience into behavioral science — he’s fascinated by the study of how people and software alike make decisions. Here at NJIT, he leads the Collective Intelligence Laboratory, which investigates the interaction between technology and human collective decision-making. In addition, he noted that real-world applications of planning via artificial intelligence go beyond games. Planning skills could be used for life-saving robots in natural disasters, efficiency improvements for assembly lines or better sorting algorithms in computer science.

Others at NJIT agree that research through gaming is a way to make it fun and accessible. Assistant Professor Amy Hoover, in Ying Wu College of Computing, is teaching a course this semester on how to develop large language models. Her students flock to gaming as a key application. Faculty in Hillier College of Architecture & Design, such as Assistant Professor Mathew Schwartz, are teaching digital design courses with AI tools.

“Now with large language models, especially the new generation that can do image processing and sound processing, it’s not just language. It adds senses,” Pescetelli said. “A lot of the debate that I'm reading now is, can the LLMs plan? Can they really go and create tasks and subtasks to reach a goal? Or are they just parroting what people have said online? I’m more inclined to think that they can plan, but I’m skeptical.”