In Augmented Reality Project, a Rodent of Unusual Size Educates Kids
New research shows that AI-enhanced augmented reality, when used for supporting computational thinking skills in K-12 education, can be more effective than simply letting children passively consume content built on the same technologies.
Lei Zhang, assistant informatics professor in NJIT’s Ying Wu College of Computing, and colleagues at Princeton University presented their results in Empowering Children to Create AI-Enabled Augmented Reality Experiences at the 38th Annual ACM Symposium on User Interface Software and Technology, in Busan, Korea during the fall 2025 semester.
Zhang’s team calls their experimental software Capybara because it’s a trendy animal among today’s children, and it’s in the spirit of turtle characters from Logo, which was the original children’s programming language 60 years ago. Capybara, now available as an Apple iOS app, has three major functions in AR environments: to animate characters through the user’s physical movements, through text-to-3D generative AI and through the Scratch block-based programming language, itself a successor to Logo.
“The main motivation is to promote the idea of AI literacy and computational thinking among the younger generation, by letting them learn by building stuff,” Zhang said. "Capybara is inspired by Scratch, but it extends those ideas into AI and augmented reality, where children can work directly in the 3D space and see how their programs interact with the physical world."
“As they're creating novel experiences with AI and AR, they get to know more about the technical boundaries of these models. We observed that they began to reason about the limitations of AI models, for example, by adjusting their programs or camera positions when object recognition models failed, and by forming hypotheses about why the AI made mistakes sometimes."
The team tested their software with 20 students aged 7-16 in the U.S. and Argentina. Most of the students already had some coding experience, and Zhang said they reported wanting to learn more after using Capybara. “It facilitates them to explore more, and they want more expressiveness when they're creating these experiences. For example, some of the more advanced techniques that they want are to customize their physical environments even more. Right now, you can only customize the characters themselves, but what if you can turn your table into something that's more relatable in a magical world? So as they are building, they want more expressiveness that needs more advanced tools — so if you want to build things like particle effects, or a more sophisticated 3D virtual world, you may,” he noted. “We've seen those sort of natural transitions from, say, Scratch, to more advanced languages like Python, and then C++ and so on.”
“We would like to have a more longitudinal study, which means that we deploy at a larger scale in a longer time span, where we study the learning effects and learning outcome from people,” Zhang said. That’s why they put their software into the Apple App Store. Having already measured engagement, “What we're trying to measure next is their learning outcome. That is not just related to computational thinking — we do not only want to measure their knowledge about writing FOR loops, IF events — but also their 3D computational thinking, whether they understand for example how these virtual objects are coordinated in the physical space, and also their understanding of AI models. We aim to measure how much they learn when it comes to vision models or generative AI.”
Another possibility is to collaborate with NJIT’s Center for Pre-College Programs, Zhang said. “We wanted to do that mainly because the existing participants are more privileged in terms of their prior programming skills. One of the kids was an advanced Python user already, at the age of 7. We want to test with kids who have fewer educational resources or less exposure to AI literacy and computational thinking. So that is definitely something that we want to do in the long term.”
And finally, “We explored the idea of programming by demonstration, in which children teach the system new behaviors by demonstrating with their own bodies. Together, these approaches point toward a future where children learn to work creatively and critically with AI — not just as consumers, but as producers capable of creating intelligent and interactive experiences.”