Publication
Benchmarking LLM Robustness Against Prompt-based Adversarial Attacks
| Summary: | Over the past few years, Large Language Models (LLMs) have revolutionized various fields by leveraging their extensive capabilities. Their ability to generate meaningful and contextually coherent text from minimal inputs, has led to significant advancements in areas such as Natural Language Processing (NLP) and Generative Artificial Intelligence (GenAI). These models have also shown potential in assisting or even automating complex tasks, including code generation. However, as LLMs become increasingly integrated into real-world applications and are made more accessible to the general public, concerns about the security and safety of the content they output have grown significantly. Recent research has shown that these models are vulnerable to adversarial techniques, which can manipulate them into generating harmful or biased outputs. These risks highlight the need for robust evaluation methods to assess the resilience of LLMs against these threats. While some efforts have been done to benchmark LLMs’ robustness against adversarial prompt-based attacks, these approaches are often highly context-specific and lack the comprehensive components required for a robust and systematic evaluation.In this dissertation, we have explored the concepts of LLMs, security, safety, and benchmarking with the goal of establishing a systematic benchmarking methodology for evaluating and comparing the robustness of LLMs built-in security measures when faced with adversarial prompt-based attacks. The proposed methodology is composed of the key components that make up a reliable benchmark, including scenarios, workloads, metrics and attack loads and is designed to ensure its representativeness, usefulness and adaptability to different contexts.To demonstrate the usefulness of the proposed methodology, we have also conducted benchmarking campaign with a varied set of LLMs in the context of vulnerable and also malicious code generation. From this instantiation, we uncovered significant insights into the models’ safety alignments and vulnerabilities. Our findings reveal that while most safety-aligned models effectively refuse to generate malicious code, they readily produce vulnerable code when asked. Furthermore, our analysis of various attack vectors demonstrated that multi-turn conversational attacks and single-turn role-playing scenarios are significantly more effective at bypassing safety measures than template-based prompts. These results underscore the importance of a structured benchmarking methodology and reveal key weaknesses in current LLMs, providing a clear path for future research in developing more robust and secure models. |
|---|---|
| Main Authors: | Donato, João Maria Campos |
| Subject: | Large Language Models (LLMs) Benchmarking Code generation Prompt Hacking Large Language Models (LLMs) Benchmarking Geração de código Prompt Hacking |
| Year: | 2025 |
| Country: | Portugal |
| Document type: | master thesis |
| Access type: | open access |
| Associated institution: | Universidade de Coimbra |
| Language: | English |
| Origin: | Estudo Geral - Universidade de Coimbra |
| Summary: | Over the past few years, Large Language Models (LLMs) have revolutionized various fields by leveraging their extensive capabilities. Their ability to generate meaningful and contextually coherent text from minimal inputs, has led to significant advancements in areas such as Natural Language Processing (NLP) and Generative Artificial Intelligence (GenAI). These models have also shown potential in assisting or even automating complex tasks, including code generation. However, as LLMs become increasingly integrated into real-world applications and are made more accessible to the general public, concerns about the security and safety of the content they output have grown significantly. Recent research has shown that these models are vulnerable to adversarial techniques, which can manipulate them into generating harmful or biased outputs. These risks highlight the need for robust evaluation methods to assess the resilience of LLMs against these threats. While some efforts have been done to benchmark LLMs’ robustness against adversarial prompt-based attacks, these approaches are often highly context-specific and lack the comprehensive components required for a robust and systematic evaluation.In this dissertation, we have explored the concepts of LLMs, security, safety, and benchmarking with the goal of establishing a systematic benchmarking methodology for evaluating and comparing the robustness of LLMs built-in security measures when faced with adversarial prompt-based attacks. The proposed methodology is composed of the key components that make up a reliable benchmark, including scenarios, workloads, metrics and attack loads and is designed to ensure its representativeness, usefulness and adaptability to different contexts.To demonstrate the usefulness of the proposed methodology, we have also conducted benchmarking campaign with a varied set of LLMs in the context of vulnerable and also malicious code generation. From this instantiation, we uncovered significant insights into the models’ safety alignments and vulnerabilities. Our findings reveal that while most safety-aligned models effectively refuse to generate malicious code, they readily produce vulnerable code when asked. Furthermore, our analysis of various attack vectors demonstrated that multi-turn conversational attacks and single-turn role-playing scenarios are significantly more effective at bypassing safety measures than template-based prompts. These results underscore the importance of a structured benchmarking methodology and reveal key weaknesses in current LLMs, providing a clear path for future research in developing more robust and secure models. |
|---|
Funded activities
No funded projects found.