Publicação
Security in Virtualized Systems: Contributions on Anomaly Detection and Intrusion Injection
| Resumo: | Virtualization technologies have become foundational in modern computing, providing an essential infrastructure for cloud and virtualized services. However, the benefits of virtualization come with a complex and large attack surface, particularly affecting the hypervisor, the core component responsible for managing and isolating virtual resources. As virtualization extends beyond traditional cloud environments into critical systems, ensuring their security is increasingly important. While considerable progress has been made in evaluating dependability and performance, comprehensive security assessments remain a significant challenge. Understanding the system behavior under compromise, including potential attacker capabilities after the system deviates from its intended semantics, is particularly complex. This thesis addresses these challenges by making four contributions to the security landscape of virtualized environments.The first contribution focuses on anomaly detection in virtualized environments. We propose a methodology that uses a three-phase approach, integrating system profiling, performance modeling, and real-time sequential anomaly detection specifically designed for complex, multi-tenant virtualized systems. Unlike traditional Intrusion Detection Systems, the proposed approach leverages performance-based signatures, using the sequential detection Bucket Algorithm, to identify anomalies with minimal system intrusion. The methodology enables tuning of false-positive rates and detection latency by modeling system behavior under typical workloads and calibrating the algorithm through statistical profiling. We conducted the validation in a cloud-representative environment that models some critical challenges of cloud workloads, such as transient performance variations and mixes services with different profiles. Results showed low false-positive detection of potential security threats in multi-tenant environments, providing insights into residual performance impacts due to anomalies.The second contribution is a study of the robustness and security of the Xen Hypervisor, combining mutation-based testing of the hypercall interface with a systematic vulnerability analysis. We evaluated 26 hypercalls (66.6% of the API) across 28,000 test cases under a database-centric workload, revealing that traditional mutation-based testing falls short due to the runtime-dependent behavior of hypercalls, highlighting the need to understand hypercall semantics. Given the layered trust relationships involved, we also found that the CRASH failure model is inadequate for virtualized systems. To complement this, we analyzed hypervisor vulnerabilities using metadata (e.g., CVEs, patches) and empirical models such as Vulnerability Discovery Models and Vulnerability Density, introducing trustworthiness evidence (artifacts that indirectly signal system security). A taxonomy links root causes to abusive functionalities and their security consequences, with memory-related flaws emerging as the most prevalent. While offering key insights into systemic weaknesses, this analysis also underscores the limitations of static approaches and the need to explore system behavior under active intrusion scenarios.The core contribution of this thesis is the concept, design, and implementation of intrusion injection for virtualized systems. Intrusion injection introduces a novel security evaluation paradigm focused on systematically inducing erroneous states, defined as states with security implications triggered by malicious activities, rather than exploiting specific vulnerabilities directly. This methodology leverages the abstraction provided by Intrusion Models (IMs), which encapsulate abusive functionalities and their associated security consequences. These models enable the controlled and repeatable emulation of post-intrusion conditions, facilitating comprehensive security assessments even in systems that have been patched or are undergoing active development, without relying on known exploits. Experimental validation of the proposed approach demonstrates its feasibility by accurately replicating erroneous states and security violations observed in real-world exploit scenarios.Finally, we present a structured methodology for the definition and instantiation of Intrusion Models (IMs). Leveraging concepts from weird machines theory and system modeling principles, we formalize IMs as tuples comprising the source, interface, target, abstraction, violated security property, abusive functionality, and erroneous state. We demonstrate the methodology’s practicality by applying it to the memory management subsystem of the Xen hypervisor, resulting in the definition of over 50 distinct IMs derived from real-world Xen Security Advisories (XSAs). Furthermore, we developed an extended injector prototype to illustrate how to reuse test cases across different intrusion models. Validation was performed across multiple versions of Xen, highligh |
|---|---|
| Autores principais: | Gonçalves, Charles Ferreira |
| Assunto: | Virtualization Hypervisors Security Anomaly Detection Intrusion Injection Virtualização Hipervisores Segurança Deteção de Anomalias Injeção de Intrusões |
| Ano: | 2025 |
| País: | Portugal |
| Tipo de documento: | tese de doutoramento |
| Tipo de acesso: | acesso aberto |
| Instituição associada: | Universidade de Coimbra |
| Idioma: | inglês |
| Origem: | Estudo Geral - Universidade de Coimbra |
| Resumo: | Virtualization technologies have become foundational in modern computing, providing an essential infrastructure for cloud and virtualized services. However, the benefits of virtualization come with a complex and large attack surface, particularly affecting the hypervisor, the core component responsible for managing and isolating virtual resources. As virtualization extends beyond traditional cloud environments into critical systems, ensuring their security is increasingly important. While considerable progress has been made in evaluating dependability and performance, comprehensive security assessments remain a significant challenge. Understanding the system behavior under compromise, including potential attacker capabilities after the system deviates from its intended semantics, is particularly complex. This thesis addresses these challenges by making four contributions to the security landscape of virtualized environments.The first contribution focuses on anomaly detection in virtualized environments. We propose a methodology that uses a three-phase approach, integrating system profiling, performance modeling, and real-time sequential anomaly detection specifically designed for complex, multi-tenant virtualized systems. Unlike traditional Intrusion Detection Systems, the proposed approach leverages performance-based signatures, using the sequential detection Bucket Algorithm, to identify anomalies with minimal system intrusion. The methodology enables tuning of false-positive rates and detection latency by modeling system behavior under typical workloads and calibrating the algorithm through statistical profiling. We conducted the validation in a cloud-representative environment that models some critical challenges of cloud workloads, such as transient performance variations and mixes services with different profiles. Results showed low false-positive detection of potential security threats in multi-tenant environments, providing insights into residual performance impacts due to anomalies.The second contribution is a study of the robustness and security of the Xen Hypervisor, combining mutation-based testing of the hypercall interface with a systematic vulnerability analysis. We evaluated 26 hypercalls (66.6% of the API) across 28,000 test cases under a database-centric workload, revealing that traditional mutation-based testing falls short due to the runtime-dependent behavior of hypercalls, highlighting the need to understand hypercall semantics. Given the layered trust relationships involved, we also found that the CRASH failure model is inadequate for virtualized systems. To complement this, we analyzed hypervisor vulnerabilities using metadata (e.g., CVEs, patches) and empirical models such as Vulnerability Discovery Models and Vulnerability Density, introducing trustworthiness evidence (artifacts that indirectly signal system security). A taxonomy links root causes to abusive functionalities and their security consequences, with memory-related flaws emerging as the most prevalent. While offering key insights into systemic weaknesses, this analysis also underscores the limitations of static approaches and the need to explore system behavior under active intrusion scenarios.The core contribution of this thesis is the concept, design, and implementation of intrusion injection for virtualized systems. Intrusion injection introduces a novel security evaluation paradigm focused on systematically inducing erroneous states, defined as states with security implications triggered by malicious activities, rather than exploiting specific vulnerabilities directly. This methodology leverages the abstraction provided by Intrusion Models (IMs), which encapsulate abusive functionalities and their associated security consequences. These models enable the controlled and repeatable emulation of post-intrusion conditions, facilitating comprehensive security assessments even in systems that have been patched or are undergoing active development, without relying on known exploits. Experimental validation of the proposed approach demonstrates its feasibility by accurately replicating erroneous states and security violations observed in real-world exploit scenarios.Finally, we present a structured methodology for the definition and instantiation of Intrusion Models (IMs). Leveraging concepts from weird machines theory and system modeling principles, we formalize IMs as tuples comprising the source, interface, target, abstraction, violated security property, abusive functionality, and erroneous state. We demonstrate the methodology’s practicality by applying it to the memory management subsystem of the Xen hypervisor, resulting in the definition of over 50 distinct IMs derived from real-world Xen Security Advisories (XSAs). Furthermore, we developed an extended injector prototype to illustrate how to reuse test cases across different intrusion models. Validation was performed across multiple versions of Xen, highligh |
|---|
Atividades financiadas
Não foram encontrados projetos financiados.