Two Research Papers on Mitigating Potential Security Flaws in LLMs Presented at ACM CCS 2024

Ph.D. candidates Nhi N Nguyen, Mahmoud Nazzal and Khiem Ton recently presented two papers at the 31st ACM Conference on Computer and Communications Security (CCS 2024) that could aid future breakthroughs in improving the trustworthiness of Large Language Models (LLMs) when generating code. The research has the potential to be a game changer in the use of generative AI for many sectors of industry and academia, especially software development.

ACM CCS is a competitive, top-tier venue in the field of security and privacy with a 16.9% acceptance rate for papers. The conference brings together information security researchers, practitioners, developers and users from all over the world to explore cutting-edge ideas and results.

Associate Professor Hai Phan in the Ying Wu College of Computing’s Department of Data Science, a pioneer in the field of secure and responsible AI, is the Ph.D. advisor to Nguyen and Ton. He is also a co-author, along with Nazzal, Professor Abdallah Khreishah in NJIT’s Newark College of Engineering and Issa Khalil, principal scientist in the Qatar Computing Research Institute at Hamad Bin Khalifa University (HBKU), on the paper titled ‘PromSec: Prompt Optimization for Secure Generation of Functional Source Code with Large Language Models.’ The project is part of a $740,000 grant through the Qatar Research, Development and Innovation Council and backed by a provisional patent supported jointly by NJIT and HBKU.

According to Nazzal, LLMs offer promising potential in efficiently generating high-quality and well-functioning source codes like Python, C and Java codes. However, since LLMs are trained on vast open-source datasets, they tend to generate code containing hidden security vulnerabilities exploitable by malicious actors.

PromSec guides LLMs to generate code that maintains both functionality and security and is based on an iterative interaction between LLM and a novel graph generative adversarial network (gGAN).

“We propose a novel gGAN loss function that promises to meet the critical standards of security and functionality. Through comprehensive experiments, we have shown that PromSec achieves its goals in a variety of operation scenarios,” said Nazzal.

As a future outlook, the team is working on improving performance and exploring more applications of LLMs in software development, including hardware design automation.

The prevalence of security vulnerabilities in AI-powered code generation is further investigated by Phan, Nguyen and Ton in ‘Demo: SGCode: A Flexible Prompt-Optimizing System for Secure Generation of Code.’ The paper is a milestone for Ton, who had it accepted to CCS 2024 only one week into his Ph.D. program.

SGCode serves as a framework for running and integrating approaches like PromSec and other prompt-optimization methods in a unified system, along with security analysis tools. Users can switch between different methods for code security optimization and obtain detailed security analysis reports and performance insights.

The system architecture comprises back-end services integrating security analysis tools with commercial LLMs and a user-friendly web-based front end. SGCode’s lightweight AWS server and minimal cost provides a solution that is widely deployable and highly scalable.

Phan states that both projects “align with NJIT’s AI strategic development vision.”

Several researchers at the conference from the U.S., China and Korea have expressed interest in future collaborations with the team.