In the last few years, the computing systems processing capabilities have increased significantly, changing from single-core to multi-core and even many-core systems. Accompanying this evolution, local networks have also become faster, with multi-gigabit technologies like Infiniband, Myrinet and 10G Ethernet. Parallel/distributed programming tools and standards, like POSIX Threads, OpenMP and MPI, have helped to explore these technologies and have been frequently combined, giving rise to Hybrid Programming Models. Recently, co-processors like GPUs and FPGAs, started to be used as accelerators, requiring specialized frameworks (like CUDA for NVIDIA GPUs). Presented with so much heterogeneity, the industry formulated the OpenCL specification, as a standard to explore heterogeneous systems. However, in the context of cluster computing, one problem surfaces: OpenCL only enables a developer to use the devices that are present in the local machine. With many processor devices scattered across cluster nodes (CPUs, GPUs and other co-processors), it then became important to enable software developers to take full advantage of the full cluster device set. This dissertation demonstrates and evaluates an OpenCL extension, named clOpenCL, which supports the simple deployment and efficient running of OpenCL-based parallel applications that may span several cluster nodes, thus expanding the original single-node OpenCL model. The main contributions are that clOpenCL i) offers a transparent approach to the porting of traditional OpenCL applications to cluster environments and ii) provides significant performance increases over classical (non-)hybrid parallel approaches. Nos últimos anos, a capacidade de processamento dos sistemas de computação aumentou significativamente, passando de CPUs com um núcleo para CPUs multi-núcleo. Acompanhando esta evolução, as redes locais também se tornaram mais rápidas, com tecnologias multi-gigabit como a Infiniband, Myrinet e 10G Ethernet. Ferramentas e standards paralelos/distribuídos, como POSIX Threads, OpenMP e MPI, ajudaram a explorar esses sistemas, e têm sido frequentemente combinados dando origem a Modelos de Programação Híbrida. Mais recentemente, co-processadores como GPUs e FPGAs, começaram a ser utilizados como aceleradores, exigindo frameworks especializadas (como o CUDA para GPUs NVIDIA). Deparada com tanta heterogeneidade, a indústria formulou a especificação OpenCL, como sendo um standard para exploração de sistemas heterogéneos. No entanto, no contexto da computação em cluster, um problema surge: o OpenCL só permite ao desenvolvedor utilizar dispositivos presentes na máquina local. Com tantos dispositivos de processamento espalhados pelos nós de um cluster (CPUs, GPUs e outros co-processadores), tornou-se assim importante habilitar os desenvolvedores de software, a tirarem o máximo proveito do conjunto total de dispositivos do cluster. Esta dissertação demonstra e avalia uma extensão OpenCL, chamada clOpenCL, que suporta a implementação simples e execução eficiente de aplicações paralelas baseadas em OpenCL que podem estender-se por vários nós do cluster, expandindo assim o modelo original de um único nó do OpenCL. As principais contribuições referem-se a que o clOpenCL i) oferece uma abordagem transparente à portabilidade de aplicações OpenCL tradicionais para ambientes cluster e ii) proporciona aumentos significativos de desempenho sobre abordagens paralelas clássicas (não) híbridas.

As our systems become larger and their interactions more complex, there is a potential for learning techniques that adapt tovarying system conditions as workloads evolve. The framework for sequential decision making provided by Reinforcement Learning (RL) fits well with resource management problems, as recent literature indicates. Yet, when we consider the use of Deep Learning for the scheduling of batch jobs in High Performance Computing (HPC) systems,we see that work in the literature either focuses on smaller problems, such as deciding which heuristic to use at a given time, or on simplified instances of the problem. In this dissertation proposal, we investigate Markov Decision Process (MDP) models to solve the problem of scheduling batch HPC jobs, presenting an approach for faster and reproducible experimentation. With this foundation,we investigate how different agents behave under this framework, while identifying deficiencies both in the representation of the problem and how learning proceeds in such a setting. Among the contributions of this work, we propose a software system for developing and experimenting with RL agents, and we evaluate different state-of-the-art algorithms from the literature in this environment, achieving performance equivalent to that of purpose-built algorithms, with comparable performance.

High-Performance Computing (HPC) systems face increasing challenges in energy efficiency and resource management as computational demands grow. A central concern is the carbon footprint of irregular tasks, which exhibit unpredictable runtime and power consumption patterns, complicating job scheduling. Efficient scheduling requires differentiating between regular, predictable jobs and irregular ones, allowing for customized scheduling that reduces energy consumption and enhances performance. Furthermore, there is a notable lack of real-world HPC data, which necessitates the creation of synthetic datasets. These datasets are essential for enabling predictive modeling of job characteristics, as machine learning models require substantial data volumes for optimal accuracy. To address these challenges, this study explores machine learning-based techniques for predicting job characteristics in HPC systems. By generating and analyzing a synthetic dataset derived from existing job metrics, predictive models capable of classifying regular versus irregular jobs and estimating key performance metrics was developed. Specifically, an ExtraTreesClassifier model, achieving 96.7% accuracy in job classification. Additionally, subsequent regression models predict runtime and power consumption, attaining a Mean Absolute Error (MAE) of 495 seconds and 280 watts, respectively. These predictions support more efficient scheduling strategies, optimizing both resource utilization and energy demands in HPC systems.

Floating-point computing with more than one TFLOP of peak performance is already a reality in recent Field-Programmable Gate Arrays (FPGA). General-Purpose Graphics Processing Units (GPGPU) and recent many-core CPUs have also taken advantage of the recent technological innovations in integrated circuit (IC) design and had also dramatically improved their peak performances. In this paper, we compare the trends of these computing architectures for high-performance computing and survey these platforms in the execution of algorithms belonging to different scientific application domains. Trends in peak performance, power consumption and sustained performances, for particular applications, show that FPGAs are increasing the gap to GPUs and many-core CPUs moving them away from high-performance computing with intensive floating-point calculations. FPGAs become competitive for custom floating-point or fixed-point representations, for smaller input sizes of certain algorithms, for combinational logic problems and parallel map-reduce problems. © 2014 Technical University of Munich (TUM).