Peringkasan Teks Berita Berbahasa Indonesia Menggunakan LSTM dan Transformer

Christina Prilla Rosaria Ardyanti, Yudi Wibisono, Rani Megasari

Abstract

Abstrak

Pertumbuhan informasi di internet membuat volume data tekstual semakin besar. Hal ini membuat manusia kesulitan dalam mengolah informasi dengan cepat. Peringkasan teks dapat membantu manusia untuk memahami informasi dalam jumlah yang banyak dengan cepat. Pada penelitian ini, arsitektur encoder-decoder akan diimplementasikan pada dataset Indosum menggunakan Long Short Term Memory (LSTM) dengan tambahan  mekanisme atensi dan Transformer. Ujicoba juga dilakukan menggunakan  fine-tuning pada pre-trained model T5-Small dan BART-Small. Eksperimen juga dilakukan dengan membandingkan dataset yang menggunakan praproses dan tanpa praproses. Berdasarkan eksperimen, model LSTM-Atensi memiliki kinerja rendah  dengan nilai ROUGE-L sebesar 13.0 pada dataset yang menggunakan praproses. Sedangkan nilai ROUGE-tertinggi didapatkan dari hasil fine-tuning T5-Small dengan nilai sebesar 66.2.

===================================================

Abstract

The proliferation of information on the internet has led to an increasing volume of textual data. This presents a challenge for humans in processing information rapidly. Text summarization can aid humans in quickly comprehending large amounts of information. In this research, an encoder-decoder architecture will be implemented on the Indosum dataset using Long Short-Term Memory (LSTM) along with attention mechanisms and Transformer. Experiments will also involve fine-tuning pre-trained models T5-Small and BART-Small. The influence of preprocessing will also be studied through experiments. Based on the experiments, the LSTM-Attention model demonstrates poor performance with an ROUGE-L score of 13.0 on the preprocessed dataset. Conversely, the highest ROUGE score was achieved through fine-tuning T5-Small, scoring 66.2.

Keywords

Peringkasan Teks, Natural Language Processing, Deep Learning, LSTM, Transformer, Mekanisme Atensi, ROUGE

Full Text:

PDF

References

[1] F. Koto, J. H. Lau, and T. Baldwin, “Liputan6: {A} Large-scale Indonesian Dataset for Text Summarization,” CoRR, vol. abs/2011.0, 2020, [Online]. Available: https://arxiv.org/abs/2011.00679

[2] M. F. Mridha, A. A. Lima, K. Nur, S. C. Das, M. Hasan, and M. M. Kabir, “A survey of automatic text summarization: Progress, process and challenges,” IEEE Access, vol. 9, pp. 156043–156070, 2021.

[3] R. Wijayanti, M. L. Khodra, and D. H. Widyantoro, “Indonesian Abstractive Summarization using Pre-trained Model,” in 2021 3rd East Indonesia Conference on Computer and Information Technology (EIConCIT), 2021, pp. 79–84.

[4] D. Bahdanau, K. Cho, and Y. Bengio, “Neural Machine Translation by Jointly Learning to Align and Translate.” 2016.

[5] K. Kurniawan and S. Louvan, “Indosum: A new benchmark dataset for Indonesian text summarization,” 2018 Int. Conf. Asian Lang. Process., 2018, doi: 10.1109/ialp.2018.8629109.

[6] C. Raffel et al., “Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer,” J. Mach. Learn. Res., vol. 21, no. 140, pp. 1–67, 2020, [Online]. Available: http://jmlr.org/papers/v21/20-074.html

[7] P. M. Hanunggul and S. Suyanto, “The impact of local attention in lstm for abstractive text summarization,” in 2019 International Seminar on Research of Information Technology and Intelligent Systems (ISRITI), 2019, pp. 54–57.

[8] S. Song, H. Huang, and T. Ruan, “Abstractive text summarization using LSTM-CNN based deep learning,” Multimed. Tools Appl., vol. 78, pp. 857–875, 2019.

[9] Y. Liu and M. Lapata, “Text summarization with pretrained encoders,” arXiv Prepr. arXiv1908.08345, 2019.

[10] R. Wijayanti, M. L. Khodra, and D. H. Widyantoro, “Single Document Summarization Using BertSum and Pointer Generator Network,” Int. J. Electr. Eng. Informatics, vol. 13, no. 4, pp. 916–930, 2021.

[11] R. Dua and M. S. Ghotra, Keras Deep Learning Cookbook: Over 30 recipes for implementing deep neural networks in python. Packt Publishing, 2018.

[12] I. Sutskever, O. Vinyals, and Q. V Le, “Sequence to Sequence Learning with Neural Networks.” 2014.

[13] K. Cho, B. van Merrienboer, D. Bahdanau, and Y. Bengio, “On the Properties of Neural Machine Translation: Encoder-Decoder Approaches.” 2014.

[14] I. Vasilev, Advanced deep learning with python: Design and implement advanced next-generation AI solutions using tensorflow and pytorch. Packt, 2019.

[15] A. Vaswani et al., “Attention Is All You Need.” 2017.

DOI: https://doi.org/10.20961/ijai.v8i2.78856

Refbacks

  • There are currently no refbacks.