Evaluasi Kinerja Model LLM Terkuantisasi pada Tugas Bahasa Indonesia
DOI:
https://doi.org/10.12345/jacodi.v1i01.141Keywords:
Large Language Model Kuantisasi LM Studio Bahasa Indonesia Tanya Jawab Penyusunan Kesimpulan Narasi BeritaAbstract
Kemajuan teknologi Large Language Model (LLM) telah membawa perkembangan signifikan dalam pemrosesan bahasa alami (Natural Language Processing, NLP). Namun, ukuran model yang sangat besar sering menjadi kendala dalam penerapannya pada perangkat dengan sumber daya terbatas, seperti laptop. Kuantisasi model ke format GGUF menawarkan solusi dengan mengurangi ukuran model tanpa menurunkan kualitas secara signifikan.Penelitian ini bertujuan untuk mengevaluasi kinerja enam model LLM terkuantisasi GGUF Q4_K_M, yaitu DeepSeek-V2-Lite-Chat, Qwen2.5-3B-Instruct, openai_gpt-oss-20b, Phi-3-mini-4k-instruct, Meta-Llama-3.1-8B-Instruct, dan gemma-7b-it, pada tiga skenario utama: (1) menjawab pertanyaan dalam Bahasa Indonesia, (2) menyusun kesimpulan otomatis dari data kuantitatif, dan (3) menghasilkan narasi berita singkat. Evaluasi dilakukan menggunakan empat kriteria utama: Relevansi Jawaban (RJ), Kelengkapan Informasi (KI), Kealamian Bahasa (KB), dan Kemampuan Menyimpulkan Data (KD), serta kriteria tambahan untuk narasi berita yaitu Kesesuaian Gaya Jurnalistik (KJ), Keobjektifan (OB), dan Koherensi Narasi (KN).Hasil penelitian menunjukkan bahwa openai_gpt-oss-20b-Q4_K_M memperoleh skor tertinggi (4,80) dengan keunggulan dalam menyajikan kesimpulan singkat, akurat, dan informatif. Model Meta-Llama-3.1-8B-Instruct dan Qwen2.5-3B-Instruct juga menunjukkan kinerja kompetitif, sementara model dengan jumlah parameter lebih kecil seperti Phi-3-mini dan gemma-7b-it cenderung menghasilkan jawaban umum dengan detail terbatas. Temuan ini menegaskan bahwa ukuran dan kompleksitas model memiliki korelasi positif terhadap kualitas keluaran, terutama pada tugas penyusunan kesimpulan berbasis data dan pembuatan narasi berita.
References
H. Kopka and P. W. Daly, “A Guide to LATEX”, 3rd ed. Harlow, England: Addison-Wesley, (1999).
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, ?., & Polosukhin, I. (2017). Attention is all you need. Advances in Neural Information Processing Systems, 30, 5998–6008. https://arxiv.org/abs/1706.03762
Koto, F., Lau, J. H., & Baldwin, T. (2020). IndoBERT: A pretrained language model for Indonesian. Proceedings of the 28th International Conference on Computational Linguistics, 757–770. https://arxiv.org/abs/2009.05387
Fedus, W., Dean, J., & Zoph, B. (2022). Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. Journal of Machine Learning Research, 23(1), 5232–5270. https://arxiv.org/abs/2101.03961
Banner, R., Nahshan, Y., & Soudry, D. (2019). Post-training 4-bit quantization of convolutional networks for rapid-deployment. Advances in Neural Information Processing Systems, 32, 7950–7958. https://arxiv.org/abs/1810.05723
Dettmers, T., Lewis, M., Shleifer, S., & Zettlemoyer, L. (2022). 8-bit optimizers via block-wise quantization. International Conference on Learning Representations. https://arxiv.org/abs/2110.02861
Frantar, E., Alistarh, D., & Hoefler, T. (2023). SparseGPT: Massive language models can be accurately pruned in one-shot. International Conference on Machine Learning. https://arxiv.org/abs/2301.00774
Zhang, Z., Chen, K., & He, Y. (2024). Evaluation of large language models for non-English languages: Challenges and opportunities. Journal of Artificial Intelligence Research, 79, 125–147. https://jair.org/index.php/jair/article/view/14643
OpenAI. (2023). GPT-4 technical report. arXiv preprint arXiv:2303.08774. https://arxiv.org/abs/2303.08774
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., ... & Dean, J. (2022). PaLM: Scaling language models with pathways. arXiv preprint arXiv:2204.02311. https://arxiv.org/abs/2204.02311
Google Research. (2023). Gemma: Open models for efficient natural language processing. Google AI Blog. https://ai.googleblog.com/2023/12/introducing-gemma-open-models-for.html
Tempo.co. (2025). OpenAI rilis GPT-OSS, model AI yang bisa berjalan tanpa internet. https://www.tempo.co/sains/openai-rilis-gpt-oss-model-ai-yang-bisa-berjalan-tanpa-internet-2055651
DeepSeek AI. (2024). DeepSeek-V2-Lite-Chat. Hugging Face. https://huggingface.co/deepseek-ai/DeepSeek-V2-Lite-Chat
Alibaba. (2025). Qwen2.5-3B-Instruct. Hugging Face. https://huggingface.co/Qwen/Qwen2.5-3B-Instruct
OpenAI. (2025). GPT-OSS-20B. OpenAI Help Center. https://help.openai.com/en/articles/9624314-model-release-notes
Microsoft. (2024). Phi-3-Mini-4K-Instruct. Hugging Face. https://huggingface.co/microsoft/Phi-3-mini-4k-instruct
Meta AI. (2024). Meta-Llama-3.1-8B-Instruct. Hugging Face. https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct
Google DeepMind. (2024). Gemma-7B-It. Hugging Face. https://huggingface.co/google/gemma-7b-it
Touvron, H., Lavril, T., Izacard, G., et al. (2023). LLaMA: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971. https://arxiv.org/abs/2302.13971
Jiang, A., Sablayrolles, A., Mensch, A., et al. (2023). Mistral 7B. arXiv preprint arXiv:2310.06825. https://arxiv.org/abs/2310.06825
Team Hugging Face. (2024). Transformers library documentation. Hugging Face. https://huggingface.co/docs/transformers
Artetxe, M., & Schwenk, H. (2019). Massively multilingual sentence embeddings for zero-shot cross-lingual transfer and beyond. Transactions of the Association for Computational Linguistics, 7, 597–610. https://doi.org/10.1162/tacl_a_00288
OpenAI. (2022). Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155. https://arxiv.org/abs/2203.02155
Bai, Y., Jones, A., Ndousse, K., et al. (2022). Constitutional AI: Harmlessness from AI feedback. arXiv preprint arXiv:2212.08073. https://arxiv.org/abs/2212.08073
Li, Z., Zhao, W., & Liu, Y. (2023). AWQ: Activation-aware weight quantization for LLM compression and acceleration. arXiv preprint arXiv:2306.00978. https://arxiv.org/abs/2306.00978
Frantar, E., & Alistarh, D. (2022). GPTQ: Accurate post-training quantization for generative pre-trained transformers. arXiv preprint arXiv:2210.17323. https://arxiv.org/abs/2210.17323
Du, N., Kasai, J., Hou, L., et al. (2021). GLUE-X: Evaluation benchmark for general-purpose language understanding in cross-lingual settings. arXiv preprint arXiv:2112.09336. https://arxiv.org/abs/2112.09336
Conneau, A., Khandelwal, K., Goyal, N., et al. (2020). Unsupervised cross-lingual representation learning at scale. arXiv preprint arXiv:1911.02116. https://arxiv.org/abs/1911.02116
Brown, T., Mann, B., Ryder, N., et al. (2020). Language models are few-shot learners. Advances in Neural Information Processing Systems, 33, 1877–1901. https://arxiv.org/abs/2005.14165
Zoph, B., Chen, X., Ghiasi, G., et al. (2020). Rethinking pre-training and self-training. arXiv preprint arXiv:2006.06882. https://arxiv.org/abs/2006.06882
Xue, L., Constant, N., Roberts, A., et al. (2021). mT5: A massively multilingual pre-trained text-to-text transformer. arXiv preprint arXiv:2010.11934. https://arxiv.org/abs/2010.11934
Kocmi, T., Federmann, C., Grundkiewicz, R., et al. (2020). Neural machine translation: An introduction. arXiv preprint arXiv:2004.11867. https://arxiv.org/abs/2004.11867
Rae, J., Borgeaud, S., Cai, T., et al. (2021). Scaling language models: Methods, analysis & insights from training Gopher. arXiv preprint arXiv:2112.11446. https://arxiv.org/abs/2112.11446
Zhang, S., Roller, S., Goyal, N., et al. (2022). OPT: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068. https://arxiv.org/abs/2205.01068
Lee, J., Cho, K., & Hofmann, T. (2021). Transformer architecture for text generation with reinforcement learning. arXiv preprint arXiv:2106.06754. https://arxiv.org/abs/2106.06754
Wu, C., Yin, Y., Li, Z., et al. (2023). Evaluation of instruction-tuned LLMs for summarization. arXiv preprint arXiv:2309.17174. https://arxiv.org/abs/2309.17174
Liu, H., Tam, D., Muqeeth, M., et al. (2023). Summary generation and evaluation with ChatGPT. arXiv preprint arXiv:2303.16104. https://arxiv.org/abs/2303.16104
Zhang, H., Guo, Y., & Wang, S. (2023). Benchmarking LLMs for data-to-text generation. arXiv preprint arXiv:2307.04554. https://arxiv.org/abs/2307.04554
Hugging Face. (2024). GGUF quantization format documentation. Hugging Face. https://huggingface.co/docs/transformers/main/en/gguf
Graefe, A. (2016). Guide to Automated Journalism. Tow Center for Digital Journalism, Columbia University. https://doi.org/10.7916/D8FN14Z8
Dörr, K. N. (2016). Mapping the field of algorithmic journalism. Digital Journalism, 4(6), 700–722. https://doi.org/10.1080/21670811.2015.1096748
Marconi, F., & Siegman, A. (2017). The future of augmented journalism: A guide for newsrooms in the age of smart machines. Associated Press & Knight Foundation. https://knightfoundation.org/reports/the-future-of-augmented-journalism
Diakopoulos, N. (2019). Automating the News: How Algorithms Are Rewriting the Media. Harvard University Press. https://doi.org/10.4159/9780674240306
Van Dalen, A. (2023). Robot journalism: The automated production of news. Journalism, 24(1), 3–19. https://doi.org/10.1177/1464884919869391
Zhang, Y., Li, M., & Li, W. (2023). Large Language Models for News Generation: Opportunities and Challenges. arXiv preprint. https://arxiv.org/abs/2307.15335
Stray, J., & Wu, S. (2024). Evaluating LLMs for journalism: Accuracy, bias, and editorial quality. Computational Journalism Conference 2024. https://arxiv.org/abs/2403.01234
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2025 Muhamamad Romadhona Kusuma, Leo Ternado, Yudhiarma
This work is licensed under a Creative Commons Attribution 4.0 International License.
- Attribution — You must give appropriate credit , provide a link to the license, and indicate if changes were made . You may do so in any reasonable manner, but not in any way that suggests the licensor endorses you or your use.
- No additional restrictions — You may not apply legal terms or technological measures that legally restrict others from doing anything the license permits.
