This dissertation investigates the automatic recognition of human emotions from facial expressions, one of the main challenges in human-computer interaction, particularly relevant to sensitive domains such as robot-assisted therapy, personalized education, and emotional well-being monitoring. The task is complicated by factors including inter- and intra-subject variability, frequent occlusions, dynamic poses, changes in lighting conditions, class imbalance, and the subtle nature of spontaneous expressions.To address these challenges, a hybrid and modular framework is proposed, combining the representational strength of deep convolutional neural network embeddings (e.g., InceptionResNetV1 and lightweight alternatives) with the descriptive richness of handcrafted visual features and OpenFace action units. Dimensionality reduction and alignment techniques, including Principal Component Analysis (PCA), autoencoders, and BoVW clustering, are employed to align, standardize, and compact heterogeneous feature spaces before fusion. Multiple fusion strategies are explored, including early, intermediate, and late fusion, as well as more advanced mechanisms such as Feature-wise Linear Modulation (FiLM), α-gating, and cross-modal transformers, with all strategies evaluated under comparable structural and parametric conditions to ensure fairness.Temporal aspects of facial expressions are explicitly modeled using a variety of sequence-aware modules, including LSTM heads over frame-level embeddings, 3D CNNs such as R(2+1)D, Temporal Convolutional Networks (TCNs), transformer-based temporal models, and hybrid configurations. An empirical evaluation is conducted across several benchmark datasets, comparing unimodal and hybrid models, analyzing the impact of different fusion techniques and feature sets, and contrasting frame-level and sequence-level modeling approaches. Performance is assessed using a wide range of metrics, including accuracy, macro and micro F1 scores, area under the curve (AUC), and per-class precision and recall. Robustness is further evaluated through cross-validation and per-class threshold analyses.Interpretability is prioritized throughout the study. Modality contributions are quantified through attention scores, gating weights, and FiLM parameters, while spatial and representational relevance are visualized using Grad-CAM and t-SNE. Diagnostic analyses help identify common failure modes and dataset-specific biases.Overall, the proposed hybrid framework consistently outperforms CNN-only and handcrafted-only baselines, with temporal modeling providing additional gains in recognizing subtle, spontaneous expressions in FER datasets. Additionally, the system serves as a flexible ablation tool, enabling direct comparisons between fusion techniques and feature sets with minimal modifications. The approach supports a wide range of deployment scenarios by revealing meaningful trade-offs between accuracy and efficiency, and incorporates practical considerations such as model compression, latency, dataset curation, and ethical implications. This work contributes a reproducible and interpretable pipeline for facial affect recognition, offering insights into effective fusion and temporal modeling strategies, and promoting the development of emotion-aware systems for real-world human–computer interaction.

Gesture recognition technology has emerged as a transformative solution for natural and intuitive human–computer interaction (HCI), offering touch-free operation across diverse fields such as healthcare, gaming, and smart home systems. In mobile contexts, where hygiene, convenience, and the ability to operate under resource constraints are critical, hand gesture recognition provides a compelling alternative to traditional touch based interfaces. However, implementing effective gesture recognition in real-world mobile settings involves challenges such as limited computational power, varying environmen tal conditions, and the requirement for robust offline–online data management. In this study, we introduce ThumbsUp, which is a gesture-driven system, and employ a partially systematic literature review approach (inspired by core PRISMA guidelines) to identify the key research gaps in mobile gesture recognition. By incorporating insights from deep learning–based methods (e.g., CNNs and Transformers) while focusing on low resource consumption, we leverage Google’s MediaPipe in our framework for real-time detection of 21 hand landmarks and adaptive lighting pre-processing, enabling accurate recogni tion of a “thumbs-up” gesture. The system features a secure queue-based offline–cloud synchronization model, which ensures that the captured images and metadata (encrypted with AES-GCM) remain consistent and accessible even with intermittent connectivity. Ex perimental results under dynamic lighting, distance variations, and partially cluttered environments confirm the system’s superior low-light performance and decreased resource consumption compared to baseline camera applications. Additionally, we highlight the feasibility of extending ThumbsUp to incorporate AI-driven enhancements for abrupt lighting changes and, in the future, electromyographic (EMG) signals for users with mo tor impairments. Our comprehensive evaluation demonstrates that ThumbsUp maintains robust performance on typical mobile hardware, showing resilience to unstable network conditions and minimal reliance on high-end GPUs. These findings offer new perspectives for deploying gesture-based interfaces in the broader IoT ecosystem, thus paving the way toward secure, efficient, and inclusive mobile HCI solutions.

Parkinson’s disease (PD) is a neurodegenerative disorder characterized by motor and non-motor symptoms. The MoveONParkinson project aims to enhance exercise engagement among people with Parkinson’s Disease (PwPD) in the Portuguese context through the ONParkinson digital platform, which provides mobile and web interfaces. While the broader MoveONParkinson project has been previously described from a health-focused perspective, this study specifically focuses on the development and integration of an AI-driven conversational agent (CA) for the Portuguese language, called PANDORA, within the mobile interface of the solution to assist and motivate PwPD in their exercise routines. PANDORA (Parkinson Assistant in Natural Dialogue and Oriented by Rules and Assessments), designed based on Self-Determination Theory (SDT), addresses the psychological needs of autonomy, competence, and relatedness. A preliminary study involving 20 PwPD, 10 caregivers, and 5 healthcare professionals informed the design requirements for PANDORA. The development process involved four main phases: (1) Design of the Chatbot’s Motivation Model, (2) Design and implementation of the conversational agent, (3) Technical Performance Evaluation, and (4) User Experience Evaluation. Technical Performance Evaluation, conducted with three physiotherapists, assessed domain coverage, coherence response capacity, and dialog management capacity, achieving 100% accuracy in domain coverage and coherence response capacity and 89% in dialog management capacity. The User Experience Study involved eight PwPD users recruited from Portuguese healthcare units performing predefined tasks, with user satisfaction scores ranging from 4.2 to 4.9 on a five-point Likert scale. The findings indicate that integrating a conversational agent with motivational cues tends to increase patient engagement. However, further studies are required to determine PANDORA’s impact on exercise engagement in PwPD.