Publicação
Transformers in Time Series Forecasting: A Systematic Literature Review
| Resumo: | The growing interest in transformers for time series forecasting has triggered an exponential surge in publications, making it increasingly challenging to organize and understand emerging trends and discern future directions. This thesis is a systematic literature review of the subject, that aims to study how the different components and aspects of the transformers evolved, finding trends, and highlighting gaps and future directions. The review focuses on papers in English, peer-reviewed, published in reputable sources, and that focus exclusively on transformer models for time series forecasting. It spans sources such as ACM, ARXiv, Elsevier, MDPI, OpenReview, AAAI, IJCAI, IEEEExplore, NeurIPS, Springer, epubs, and Jstage, from December of 2023 to April of 2024 – selecting a total of 99 papers. To assess the risk of bias, statements regarding conflicts of interest and sources of funding were examined. Publication bias was evaluated by comparing transformer results over time. The reviewed papers reveal a preference for nine datasets in specific but a lot of papers still use unique private datasets, with some variation of the time-horizons used as well as the metrics. Recent advancements showcase innovations in Input Representation (Positional Encoding, Segmentation, Decomposition, and Covariates), Modelling (Attention Mechanism, Feature Selection, Channel Dependence or Independence, Multi-scale and Hierarchical approaches, and Hybridization), Model Optimization (Regularization, Loss function, and Normalization), and Types of Learning. These advancements aim to enhance model adaptability, accuracy, robustness, and computational efficiency, as well as to improve the capture of local context and temporal relationships at different granularities. However, there is a limited understanding of transformer performance across diverse domains and datasets, as most studies focus on tailored models rather than cross-domain evaluations. The lack of standardized benchmarks complicates model assessment, necessitating more transparent and explainable models. Scalability and computational efficiency remain major concerns, especially for large-scale and real-time implementations. |
|---|---|
| Autores principais: | Silva, Diogo Miguel Goulart Ferreira Pereira da |
| Assunto: | Transformer Time Series Forecast Attention Mechanism Deep Learning |
| Ano: | 2024 |
| País: | Portugal |
| Tipo de documento: | dissertação de mestrado |
| Tipo de acesso: | acesso aberto |
| Instituição associada: | Universidade Nova de Lisboa |
| Idioma: | inglês |
| Origem: | Repositório Institucional da UNL |
| Resumo: | The growing interest in transformers for time series forecasting has triggered an exponential surge in publications, making it increasingly challenging to organize and understand emerging trends and discern future directions. This thesis is a systematic literature review of the subject, that aims to study how the different components and aspects of the transformers evolved, finding trends, and highlighting gaps and future directions. The review focuses on papers in English, peer-reviewed, published in reputable sources, and that focus exclusively on transformer models for time series forecasting. It spans sources such as ACM, ARXiv, Elsevier, MDPI, OpenReview, AAAI, IJCAI, IEEEExplore, NeurIPS, Springer, epubs, and Jstage, from December of 2023 to April of 2024 – selecting a total of 99 papers. To assess the risk of bias, statements regarding conflicts of interest and sources of funding were examined. Publication bias was evaluated by comparing transformer results over time. The reviewed papers reveal a preference for nine datasets in specific but a lot of papers still use unique private datasets, with some variation of the time-horizons used as well as the metrics. Recent advancements showcase innovations in Input Representation (Positional Encoding, Segmentation, Decomposition, and Covariates), Modelling (Attention Mechanism, Feature Selection, Channel Dependence or Independence, Multi-scale and Hierarchical approaches, and Hybridization), Model Optimization (Regularization, Loss function, and Normalization), and Types of Learning. These advancements aim to enhance model adaptability, accuracy, robustness, and computational efficiency, as well as to improve the capture of local context and temporal relationships at different granularities. However, there is a limited understanding of transformer performance across diverse domains and datasets, as most studies focus on tailored models rather than cross-domain evaluations. The lack of standardized benchmarks complicates model assessment, necessitating more transparent and explainable models. Scalability and computational efficiency remain major concerns, especially for large-scale and real-time implementations. |
|---|
Atividades financiadas
Não foram encontrados projetos financiados.