demo title by scholarjms

Q: Who are the authors of this article and where was it published?

test user. Published in Demo Journal from ScholarJMS, Vol. 2, No. 2, (2026).

Q: Is the full text of this article freely available?

Yes. This article is published open access under CC BY 4.0. The full text and PDF can be downloaded free of charge.

Q: What are the key topics covered in this article?

Large Language Models, Low-Rank Adaptation (LoRA), Sparse Attention, Energy-Efficient Training, Green, Received:, Revised:, Accepted:, Citation for the Paper:.

test user

Published

demo title by scholarjms

test user

Published in July - December 2026 (Vol. 2, Issue 2, 2026)

Keywords

Large Language Models Low-Rank Adaptation (LoRA)Sparse Attention Energy-Efficient Training Green Received:Revised:Accepted:Citation for the Paper:

Abstract

Large language models (LLMs) such as GPT and BERT have revolutionized natural language processing but impose enormous computational and energy costs due to their massive parameter sizes and quadratic attention complexity. This study introduces LoRA-S, a unified framework that combines Low-Rank Adaptation (LoRA) with sparse attention mechanisms to achieve energy-efficient and scaljournalable training of transformer-based LLMs. By freezing pretrained weights and injecting low-rank trainable matrices into attention and feed-forward layers, LoRA reduces the number of trainable parameters by over 90%, significantly lowering gradient computation and memory overhead. Simultaneously, sparse attention restricts token interactions to structured subsets, cutting attention-related FLOPs from 100 G to 7 G in WikiText-2 experiments. Comparative analysis across Full Fine-Tuning, LoRA, and LoRA-S demonstrates that LoRA-S achieves the lowest energy consumption of 22,380 J (6.22 Wh) while maintaining competitive task performance, with perplexity of 115.26 on WikiText-2 and sentiment classification accuracy of 73.90% on IMDB. Pareto frontier analysis confirms LoRA-S as an optimal trade-off between computational efficiency and predictive capability, enabling resource-constrained and eco-friendly model deployment. These results establish LoRA-S as a practical step toward Green AI, providing a novel, integrated approach to minimize FLOPs and parameter updates without substantially compromising LLM performance.

References

[1]Hadi, M. U., Qureshi, R., Shah, A., Irfan, M., Zafar, A., Shaikh, M. B., ... & Mirjalili, S. (2023). Large language models: a comprehensive survey of their applications, challenges, limitations, and future prospects. Authorea preprints, 1(3), 1-26.
[2]Raiaan, M. A. K., Mukta, M. S. H., Fatema, K., Fahad, N. M., Sakib, S., Mim, M. M. J., ... & Azam, S. (2024). A review on large language models: Architectures, applications, taxonomies, open issues, and challenges. IEEE Access, 12, 26839-26874.
[3]Jonnala, R., Yang, J., Lee, Y., Liang, G., & Cao, Z. (2025). Measuring and improving the efficiency of Python code generated by LLMs using cot prompting and fine-tuning. IEEE Access.
[4]Mussa, A., Tuimebayev, Z., & Mansurova, M. (2025). Make Large Language Models Efficient: A Review. IEEE Access.DOI: 10.1109/ACCESS.2025.3605110
[5]Wang, L., Chen, S., Jiang, L., Pan, S., Cai, R., Yang, S., & Yang, F. (2025). Parameter-efficient fine-tuning in large language models: a survey of methodologies. Artificial Intelligence Review, 58(8), 227.https://doi.org/10.1007/s10462-025-11236-4
[6]Yuan, Z., Sun, W., Liu, Y., Zhou, H., Zhou, R., Li, Y., ... & Ye, Y. (2025). EfficientLLM: Efficiency in Large Language Models. arXiv preprint arXiv:2505.13840.https://doi.org/10.48550/arXiv.2505.13840
[7]Usman, Y., Ihejirika, C. J., Offor, S. N., Robert, A., & Chataut, R. (2025). Green cybersecurity: leveraging AI, ML, and LLMs to optimize energy, threat detection, and sustainability Frameworks. IEEE Access.DOI: 10.1109/ACCESS.2025.3602451
[8]Shahzad, T., Mazhar, T., Tariq, M. U., Ahmad, W., Ouahada, K., & Hamam, H. (2025). A comprehensive review of large language models: issues and solutions in learning environments. Discover Sustainability, 6(1), 27.https://doi.org/10.1007/s43621-025-00815-8
[9]Wu, Y., Kan, S., Zeng, M., & Li, M. (2023, August). Singularformer: Learning to Decompose Self-Attention to Linearize the Complexity of Transformer. In IJCAI (pp. 4433-4441).
[10]Sarpietro, R. E., Pino, C., Coffa, S., Messina, A., Palazzo, S., Battiato, S., ... & Rundo, F. (2022). Explainable deep learning system for advanced silicon and silicon carbide electrical wafer defect map assessment. IEEE Access, 10, 99102-99128.DOI: 10.1109/ACCESS.2022.3204278
[11]Yin, D., Zhao, T. F., Fan, D. P., Li, S., Du, B., Sun, X., & Hu, S. M. (2025). Remote sensing tuning: A survey. Computational Visual Media.
[12]Sharma, S. (2024). Generalization and Fine-Tuning of Robotic Foundation Models.
[13]Taylor, N., Ghose, U., Rohanian, O., Nouriborji, M., Kormilitzin, A., Clifton, D. A., & Nevado-Holgado, A. (2024). Efficiency at scale: investigating the performance of diminutive language models in clinical tasks. Artificial intelligence in medicine, 157, 103002.https://doi.org/10.1016/j.artmed.2024.103002
[14]Nwaiwu, S. (2025). Parameter-efficient fine-tuning for low-resource text classification: a comparative study of LoRA, IA3, and ReFT. Frontiers in Big Data, 8, 1677331.https://doi.org/10.3389/fdata.2025.1677331
[15]Ayyat, M., Osman, M., & Nadeem, T. (2025). Opportunities and challenges of foundation models in industrial manufacturing. IEEE Access.
[16]Kumar, P. (2024). Large language models (LLMs): survey, technical frameworks, and future challenges. Artificial Intelligence Review, 57(10), 260.https://doi.org/10.1007/s10462-024-10888-y
[17]Tu, X., He, Z., Huang, Y., Zhang, Z. H., Yang, M., & Zhao, J. (2024). An overview of large AI models and their applications. Visual Intelligence, 2(1), 34.https://doi.org/10.1007/s44267-024-00065-8
[18]Fakhabi, M. M., Hamidian, S. M., & Aliehyaei, M. (2024). Exploring the role of the Internet of Things in green buildings. Energy Science & Engineering, 12(9), 3779-3822.DOI: 10.1002 /ese3.1840
[19]Barbierato, E., & Gatti, A. (2024). Toward green AI: A methodological survey of scientific literature. IEEE Access, 12, 23989-24013.
[20]Cong, S., & Zhou, Y. (2023). A review of convolutional neural network architectures and their optimizations. Artificial Intelligence Review, 56(3), 1905-1969.https://doi.org/10.1007/s10462-022-10213-5
[21]Ahmed, S. F., Alam, M. S. B., Hassan, M., Rozbu, M. R., Ishtiak, T., Rafa, N., ... & Gandomi, A. H. (2023). Deep learning modelling techniques: current progress, applications, advantages, and challenges. Artificial Intelligence Review, 56(11), 13521-13617.https://doi.org/10.1007/s10462-023-10466-8