Shilpa Mukhopadhyay is a first-year Ph.D. student in the Electrical and Computer Engineering Department at NJIT, where she is pursuing her doctoral studies under the guidance of Dr. Arnob Ghosh. Her research interests focus on developing safe and robust reinforcement learning (RL) agents, with an emphasis on combining theoretical guarantees with empirical experimentation. Her current research focuses on developing safe reinforcement learning agents that consistently prioritize safety across different time frames while adhering to specified constraints. At the same time, these agents are designed to achieve the best possible task performance without compromising safety requirements. Another key aspect of her work is improving the robustness of safe RL agents, enabling them to adapt to changes in system transitions within specified limits. This robustness is particularly important for supporting the successful deployment of trained agents from simulation to real-world environments. Through her research, Shilpa aims to advance reinforcement learning methods that are not only effective but also safe, reliable, and robust for real-world applications.
What would you say that could be the next big thing in your area of research?
One of the key challenges in reinforcement learning is ensuring that agents remain safe and reliable when transitioning from simulated environments to real-world applications. Even robust agents can face uncertainty when real-world conditions differ from the environments in which they were trained.
I believe the next major advancement will be combining safe and robust reinforcement learning with uncertainty-aware world models. These models would allow agents to plan using learned representations of their environments while also recognizing when their predictions may be unreliable. By accounting for uncertainty, agents can avoid potentially unsafe actions when faced with unfamiliar conditions or sim-to-real mismatches.
Reinforcement learning (RL) has become a cornerstone in the realm of AI. Typical RL solutions require extensive tests in simulation which makes the so-called "sim-to-real gap" a hot research topic. Please share your understanding of what sim-to-real gap is and the current mainstream method to address it.
Sim-to-real refers to the gap an intelligent agent (trained with an RL algorithm) faces when it is trained in the simulator and then deployed in the real world. The gap exists because of the mismatch on sensor inputs and environment dynamics.
In order to address the sim-to-real gaps that arise due to mismatch in sensor information, we generally adapt Domain randomization. In domain randomization, while training the agents in a simulator, we also keep randomizing the sensor input. To explain better I will take the example of a camera sensor. It is a common issue, if the lighting conditions change slightly than the trained environment, the agent is extremely sensitive and fails to take the optimal action. The agent’s understanding of the sensor stays limited without domain randomization. For a camera sensor, an agent is trained under different variations of lighting conditions using domain randomization. When deployed in the real world, domain randomization makes the learned policy robust/invariant to variations seen during training and performs its task correctly.
The sim-to-real gap due to mismatch in environment dynamics is generally handled using Robustness. In simple terms, we train the agent to be robust against changing environment dynamics (more appropriately transition dynamics). For model-free RL, the agent usually learns a value function or policy directly from interaction, implicitly learning the transition model. Therefore, if the transition dynamics change, the policy of the agent fails to take the correct action. We train the agent to be robust against this failure, so that if the transition dynamics change to a certain limit than the trained dynamics, the agent can still adapt to it. This direction of work is generally handled in Robust RL problems.
Ensuring safety of AI in real world applications has become an important topic. Please share some specific examples that you have come across in your research on RL.
Safety of AI in real world applications is indeed an important topic today. My research is concerned about 2 kinds of safety currently:
(a) Bounding Expected cumulative cost below a safety threshold
(b) Maintaining persistent safety (at all time steps)
For applications which require considering additive cost over the entire horizon, we maintain cumulative safety. Examples include maintaining the total money/power incurred for completing a particular task below the availability. For applications which are risk sensitive, we want to maintain persistent safety. An example includes a collision scenario. A vehicle maintains safety for every time frame in a long horizon task, but hits a pedestrian at only one time step. This can be catastrophic.
I have worked on Safety in Robotics. Some important real world applications I have come across are in the domains of:
(a) Humanoids (or any robot involving multi joint dynamics with contact) : restricting joint movements to maintain posture stability
(b) Autonomous driving: maintaining collision free driving
(c) Quadrotors: maintaining the flight position within a particular altitude.
Overall, I find working on specific real-world applications to ensure safety a very interesting and challenging problem from different directions.
