Analisis Perbandingan Metode Natural Language Processing (NLP) Klasik (Naïve Bayes, Support Vektor Machine) Dan Model Bidirectional Encoder Representations From  Transformers (BERT) Dalam Klasifikasi  Teks Berbahasa Indonesia

Ummul Farihah; Zainal Azis

Authors

Ummul Farihah Universitas Muhammadiyah Sumatera Utara
Zainal Azis Universitas Muhammadiyah Sumatera Utara

Keywords:

Analisis Sentimen, Klasifikasi Teks, IndoBERT, Support Vector Machine, Uji McNemar

Abstract

Pertumbuhan data teks berbahasa Indonesia di platform digital menuntut sistem klasifikasi sentimen yang akurat dan dapat dipertanggungjawabkan. Penelitian ini membandingkan Multinomial Naïve Bayes, Support Vector Machine (SVM) linear, dan IndoBERT pada dataset SmSA IndoNLU, sekaligus menguji signifikansi perbedaan performa dengan Uji McNemar. Dataset berjumlah 12.760 sampel yang terbagi menjadi 11.000 data latih, 1.260 data validasi, dan 500 data uji dengan distribusi kelas tidak seimbang. Model klasik menggunakan TF-IDF unigram-bigram dan preprocessing eksplisit, sedangkan IndoBERT menggunakan tokenisasi WordPiece. Macro F1-Score digunakan sebagai metrik utama. IndoBERT mencapai akurasi 88,60% dan Macro F1-Score 83,77%, diikuti SVM 76,20% dan 71,68%, serta Naïve Bayes 66,00% dan 60,99%. Pada kelas netral, F1-Score IndoBERT mencapai 67,67%, lebih tinggi daripada SVM 55,56% dan Naïve Bayes 46,48%. Seluruh perbedaan antarmodel signifikan secara statistik (p < 0,05). Hasil eksperimen diintegrasikan ke platform web Ummu NLP Lab yang menghasilkan metrik identik dengan notebook laboratorium serta mendukung reproduksibilitas eksperimen secara konsisten dan transparan.

References

We Are Social and Meltwater, Digital 2026: Global Overview Report. We Are Social Ltd., 2026.

I. Alfina, R. Mulia, M. I. Fanany, and Y. Ekanata, “Hate speech detection in the Indonesian language: A dataset and preliminary study,” in Proc. 2017 Int. Conf. Advanced Computer Science and Information Systems (ICACSIS), 2017, pp. 233–238.

A. McCallum and K. Nigam, “A comparison of event models for Naive Bayes text classification,” in AAAI-98 Workshop on Learning for Text Categorization, 1998, pp. 41–48.

C. Cortes and V. Vapnik, “Support-vector networks,” Machine Learning, vol. 20, no. 3, pp. 273–297, 1995.

J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” in Proc. NAACL-HLT 2019, 2019, pp. 4171–4186.

T. G. Dietterich, “Approximate statistical tests for comparing supervised classification learning algorithms,” Neural Computation, vol. 10, no. 7, pp. 1895–1923, 1998.

E. Alpaydin, “Combined 5×2 cv F test for comparing supervised classification learning algorithms,” Neural Computation, vol. 11, no. 8, pp. 1885–1892, 1999.

B. Wilie et al., “IndoNLU: Benchmark and resources for evaluating Indonesian natural language understanding,” in Proc. AACL-IJCNLP 2020, 2020, pp. 843–857.

F. Koto, A. Rahimi, J. H. Lau, and T. Baldwin, “IndoLEM and IndoBERT: A benchmark dataset and pre-trained language model for Indonesian NLP,” in Proc. COLING 2020, 2020, pp. 757–770.

M. Sokolova and G. Lapalme, “A systematic analysis of performance measures for classification tasks,” Information Processing & Management, vol. 45, no. 4, pp. 427–437, 2009.

J. Pineau, P. Vincent-Lamarre, K. Sinha, V. Larivière, A. Beygelzimer, F. d’Alché-Buc, E. Fox, and H. Larochelle, “Improving reproducibility in machine learning research: A report from the NeurIPS 2019 reproducibility program,” Journal of Machine Learning Research, vol. 22, no. 164, pp. 1–20, 2021.

G. Salton and C. Buckley, “Term-weighting approaches in automatic text retrieval,” Information Processing & Management, vol. 24, no. 5, pp. 513–523, 1988.

A. Vaswani et al., “Attention is all you need,” in Advances in Neural Information Processing Systems 30, 2017, pp. 5998–6008.

F. Pedregosa et al., “Scikit-learn: Machine learning in Python,” Journal of Machine Learning Research, vol. 12, pp. 2825–2830, 2011.

J. Bergstra and Y. Bengio, “Random search for hyper-parameter optimization,” Journal of Machine Learning Research, vol. 13, pp. 281–305, 2012.

Q. McNemar, “Note on the sampling error of the difference between correlated proportions or percentages,” Psychometrika, vol. 12, no. 2, pp. 153–157, 1947.

T. Joachims, “Text categorization with Support Vector Machines: Learning with many relevant features,” in Proc. 10th European Conf. Machine Learning (ECML 98), 1998, pp. 137–142.

S. Cahyawijaya, G. I. Winata, B. Wilie, K. Vincentio, X. Li, F. Koto, A. Rahimi, P. Bahar, and A. Purwarianti, “Indo4B: An open-source Indonesian language foundation model,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 29, pp. 3230–3242, 2021.