Computing Grad Student Adapts Leading-Edge Tech for Drone and Robot Vision

If you can envision a future where robots need eyeglasses to accurately deliver packages and safely perform dangerous missions, then Craig Iaboni would be your local android optician.

Until then, Iaboni is pursuing an M.S. in computer science at NJIT by coding new kinds of neural networks and using cutting-edge architectures to help electronic beings better see the world around them.

Iaboni originally studied political science at Hofstra University but decided he prefered programming, so he transferred to a community college and then to NJIT, where he now shares his coding skills in a student team led by Assistant Professor Pramod Abichandani from Newark College of Engineering's School of Applied Engineering and Technology.

"As the computer science guy, we are trying to solve drone swarming and coordination. So what my research focus has been is computer vision, which is how do we actually get these robots to sense, using visual sensors to perceive their environment," he explained.

The team hit on an idea where drones and robots can make their own decisions by filtering the local landscape through a deep learning neural network, as seen through a high-tech camera. They use a neuromorphic sensor camera, which means it looks for pixel-by-pixel changes rather than taking repeated snapshots through a shutter. They tested it on ground-going robots a year ago and drones six months ago.

Initial tests used cluster detection, where the logic stated that many pixels bunched together are probably the drone or robot. But that fails in the real world where the bunch could also be a bird or rock, Iaboni noted. Tests also highlighted another problem — clustering methods and neuromorphic cameras aren't the most efficient combination.

To fix it, they're switching to what is called a spiking neural network, where the term spiking means it operates on individual changes rather than whole scenes, just like the camera. That's also how the human brain works when, for example, we respond to touch. Iaboni and his colleagues intend to publish a paper about spiking networks in the next year, and then they'll implement it in their tracking system.

Powering that network also requires faster hardware, so they're testing an experimental neuromorphic processor made by Intel, called Loihi. That falls in the realm of cognitive computing, which has logic based on the design of brain synapses, rather than conventional computer architecture that has roots with Charles Babbage.

"The research path and everything that we've talked about ties into this one ultimate goal of having real-time object detection," Iaboni said. "We are really trying to explore the efficiencies of how fast can we get these cameras to operate, how little power can they draw and how can these neural networks operate fast enough to keep up."

"And the answer is event-based cameras and spiking neural networks," he added. "When we don't have to sample repeated parts of the scene that aren't moving, we save a lot in terms of power and data storage. It has this cascading effect of reducing so much load on the network. It's really attractive to computer vision."