Publicação
Multimodal perception for fault tolerant human-robot interaction
| Resumo: | Ensuring safety and reliability in human–robot interaction (HRI) requires the timely detection of unexpected events that could lead to failures or unsafe behaviours. Anomaly detection therefore plays a crucial role in enabling robots to recognise and respond to deviations from normal operation during collaborative tasks. While reconstruction-based methods have been widely explored for anomaly detection in HRI, approaches that operate directly on semantic feature representations remain relatively unexplored. In this work, we propose an anomaly detection framework that transforms video streams into semantically meaningful feature vectors and performs reconstruction-based anomaly detection on these representations. The approach is further extended by incorporating additional modalities, including the robot’s internal sensor readings and scene graph representations, allowing the model to capture both external visual anomalies and internal robotic failures. To evaluate the framework, we collected a custom multimodal dataset comprising a simple pick-and-place robotic task performed under normal and anomalous conditions. Experimental results show that reconstruction based on vision-derived feature vectors is effective for identifying anomalies, and that integrating multiple modalities further improves detection performance. These findings highlight the advantages of multimodal feature reconstruction for achieving more robust and reliable anomaly detection in human–robot collaborative environments. |
|---|---|
| Autores principais: | Ribeiro,Guilherme Rei dos Santos |
| Assunto: | Human-robot interaction Deep Learning Anomaly Detection Computer Vision |
| Ano: | 2026 |
| País: | Portugal |
| Tipo de documento: | dissertação de mestrado |
| Tipo de acesso: | acesso aberto |
| Instituição associada: | Universidade de Lisboa |
| Idioma: | inglês |
| Origem: | Repositório da Universidade de Lisboa |
| Resumo: | Ensuring safety and reliability in human–robot interaction (HRI) requires the timely detection of unexpected events that could lead to failures or unsafe behaviours. Anomaly detection therefore plays a crucial role in enabling robots to recognise and respond to deviations from normal operation during collaborative tasks. While reconstruction-based methods have been widely explored for anomaly detection in HRI, approaches that operate directly on semantic feature representations remain relatively unexplored. In this work, we propose an anomaly detection framework that transforms video streams into semantically meaningful feature vectors and performs reconstruction-based anomaly detection on these representations. The approach is further extended by incorporating additional modalities, including the robot’s internal sensor readings and scene graph representations, allowing the model to capture both external visual anomalies and internal robotic failures. To evaluate the framework, we collected a custom multimodal dataset comprising a simple pick-and-place robotic task performed under normal and anomalous conditions. Experimental results show that reconstruction based on vision-derived feature vectors is effective for identifying anomalies, and that integrating multiple modalities further improves detection performance. These findings highlight the advantages of multimodal feature reconstruction for achieving more robust and reliable anomaly detection in human–robot collaborative environments. |
|---|
Atividades financiadas
Não foram encontrados projetos financiados.