Publicação
Evaluating the Feasibility of Local LLMs in Privacy-Sensitive Environments: A Comparative Performance Framework
| Resumo: | In recent years, large language models (LLMs) have made great advancements in natural language processing (NLP) and other artificial intelligence (AI) applications and have been used in many industries such as healthcare, finance and law, which work with sensitive, confidential and proprietary data. However, most state-of-the-art LLMs are cloud-based, raising concerns about data privacy and security in these sectors. A safe alternative to this could be the implementation of local LLMs, which offer a promising solution by enabling organizations to keep control of their data whilst benefiting from the same AI capabilities. This study evaluates the feasibility of implementing these models in such industries through development of a framework to benchmark LLMs capabilities in diverse NLP tasks. The two most researched local LLMs and the two most researched cloud-based LLMs are tested in this benchmark, to assess the local models’ capabilities, compared to state-of-the-art models. After analyzing the results, we concluded that, despite being competitive in some of the benchmarks, the local LLMs weren’t consistent through the entire evaluation process and consistently underperformed compared to the cloud-based models. These suggest that local models, although not as accurate or versatile as the proprietary LLMs, could be viable for lightweight NLP workloads where latency, cost-efficiency, or privacy are primary concerns. Tasks involving simple fact retrieval or comprehension can often be handled by these models, but they can be relied on for complex and high-stakes applications. |
|---|---|
| Autores principais: | Anjos, Afonso Bruno dos |
| Assunto: | Local LLMs Cloud-based LLMs NLP Framework Benchmark Llama 3.3 Claude 3.7 Sonnet GPT-4.1 Mistral Small SDG 4 - Quality education SDG 9 - Industry, innovation and infrastructure |
| Ano: | 2025 |
| País: | Portugal |
| Tipo de documento: | dissertação de mestrado |
| Tipo de acesso: | acesso aberto |
| Instituição associada: | Universidade Nova de Lisboa |
| Idioma: | inglês |
| Origem: | Repositório Institucional da UNL |
| Resumo: | In recent years, large language models (LLMs) have made great advancements in natural language processing (NLP) and other artificial intelligence (AI) applications and have been used in many industries such as healthcare, finance and law, which work with sensitive, confidential and proprietary data. However, most state-of-the-art LLMs are cloud-based, raising concerns about data privacy and security in these sectors. A safe alternative to this could be the implementation of local LLMs, which offer a promising solution by enabling organizations to keep control of their data whilst benefiting from the same AI capabilities. This study evaluates the feasibility of implementing these models in such industries through development of a framework to benchmark LLMs capabilities in diverse NLP tasks. The two most researched local LLMs and the two most researched cloud-based LLMs are tested in this benchmark, to assess the local models’ capabilities, compared to state-of-the-art models. After analyzing the results, we concluded that, despite being competitive in some of the benchmarks, the local LLMs weren’t consistent through the entire evaluation process and consistently underperformed compared to the cloud-based models. These suggest that local models, although not as accurate or versatile as the proprietary LLMs, could be viable for lightweight NLP workloads where latency, cost-efficiency, or privacy are primary concerns. Tasks involving simple fact retrieval or comprehension can often be handled by these models, but they can be relied on for complex and high-stakes applications. |
|---|
Atividades financiadas
Não foram encontrados projetos financiados.