Penerapan Fine-Tuning Model Bert Untuk Efektivitas  Klasifikasi Teks

Mhd Syafiq Hasan Jambak; Mahardika Abdi Prawira Tanjung

Authors

Mhd Syafiq Hasan Jambak Universitas Muhammadiyah Sumatera Utara
Mahardika Abdi Prawira Tanjung Universitas Muhammadiyah Sumatera Utara

Keywords:

Analisis Sentimen, BERT, Fine-Tuning, Feature Extraction, Pemrosesan Bahasa Alami

Abstract

Klasifikasi sentimen teks menghadapi kompleksitas semantik dan ambiguitas konteks. BERT menawarkan representasi kontekstual dua arah, tetapi strategi adaptasi antara feature extraction dan end-to-end fine-tuning memerlukan pembuktian empiris dari sisi akurasi dan efisiensi komputasi. Penelitian ini menganalisis Model A (feature extraction) dan Model B (end-to-end fine-tuning) berbasis bert-base-uncased pada Stanford Sentiment Treebank (SST-2) dengan 872 sampel held-out test set. Eksperimen dijalankan pada enam random seed dan dievaluasi menggunakan akurasi, precision, recall, F1-score, uji McNemar, Wilcoxon Signed-Rank, Bootstrap 95% Confidence Interval, Cohen's d, konsumsi VRAM, latensi inferensi, dan ketahanan linguistik. Model B mencapai akurasi 92,78% ± 0,71% dan F1-score 92,94% ± 0,79%, sedangkan Model A mencapai 86,05% ± 0,69% dan 86,66% ± 0,43%. Perbedaan signifikan pada Wilcoxon (p=0,03125) dan McNemar (χ²=38,1635; p=6,51×10⁻¹⁰), dengan Cohen's d=10,11. Latensi GPU hampir identik, sedangkan Model A lebih hemat VRAM hingga 68,7% pada clean state. Model B juga lebih kuat pada ulasan panjang dan negasi biner. Hasil ini menunjukkan fine-tuning lebih efektif untuk akurasi dan pemahaman konteks, sementara feature extraction lebih sesuai pada lingkungan dengan keterbatasan memori GPU

References

S. Minaee, N. Kalchbrenner, E. Cambria, N. Nikzad, M. Chenaghlu, and J. Gao, “Deep learning-based text classification: A comprehensive review,” ACM Computing Surveys, vol. 54, no. 3, pp. 1–40, 2021.

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in Advances in Neural Information Processing Systems, 2017, pp. 5998–6008.

J. Devlin, M. W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” in Proc. NAACL-HLT, 2019, pp. 4171–4186, doi: 10.18653/v1/N19-1423.

A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman, “GLUE: A multi-task benchmark and analysis platform for natural language understanding,” in Proc. EMNLP Workshop BlackboxNLP, 2018, doi: 10.18653/v1/W18-5446.

R. Qasim, W. H. Bangyal, M. A. Alqarni, and A. Ali Almazroi, “A fine-tuned BERT-based transfer learning approach for text classification,” Journal of Healthcare Engineering, vol. 2022, Art. no. 3498123, 2022.

H. Zaman-Khan, M. Naeem, R. Guarasci, U. Khalid, M. Esposito, and F. Gargiulo, “Enhancing text classification using BERT: A transfer learning approach,” Computación y Sistemas, vol. 28, no. 4, pp. 2279–2295, 2024, doi: 10.13053/cys-28-4-5290.

R. Dror, G. Baumer, S. Shlomov, and R. Reichart, “The hitchhiker’s guide to testing statistical significance in natural language processing,” in Proc. 56th Annual Meeting of the Association for Computational Linguistics, 2018, pp. 1383–1392.

L. Galke, A. Diera, B. X. Lin, B. Khera, T. Meuser, T. Singhal, F. Karl, and A. Scherp, “Are we really making much progress in text classification? A comparative review,” ACM Computing Surveys, 2024.

Y. Hao, L. Dong, F. Wei, and K. Xu, “Investigating learning dynamics of BERT fine-tuning,” in Proc. AACL-IJCNLP, 2020, pp. 87–92.

C. Sun, X. Qiu, Y. Xu, and X. Huang, “How to fine-tune BERT for text classification?” in Chinese National Conference on Computational Linguistics, 2019, pp. 194–206.

R. Socher, A. Perelygin, J. Wu, J. Chuang, C. D. Manning, A. Ng, and C. Potts, “Recursive deep models for semantic compositionality over a sentiment treebank,” in Proc. EMNLP, 2013, pp. 1631–1642, doi: 10.18653/v1/D13-1170.

M. Sokolova and G. Lapalme, “A systematic analysis of performance measures for classification tasks,” Information Processing & Management, vol. 45, no. 4, pp. 427–437, 2009, doi: 10.1016/j.ipm.2009.03.002.

X. Bouthillier et al., “Accounting for variance in machine learning benchmarks,” Proceedings of Machine Learning and Systems, vol. 3, pp. 747–769, 2021.

Q. McNemar, “Note on the sampling error of the difference between correlated proportions or percentages,” Psychometrika, vol. 12, no. 2, pp. 153–157, 1947, doi: 10.1007/BF02295996.

F. Wilcoxon, “Individual comparisons by ranking methods,” Biometrics Bulletin, vol. 1, no. 6, pp. 80–83, 1945, doi: 10.2307/3001968.

B. Efron, “Bootstrap methods: Another look at the jackknife,” The Annals of Statistics, vol. 7, no. 1, pp. 1–26, 1979, doi: 10.1214/aos/1176344552.

J. Cohen, Statistical Power Analysis for the Behavioral Sciences, 2nd ed. Lawrence Erlbaum Associates, 1988.

M. T. Ribeiro, T. Wu, C. Guestrin, and S. Singh, “Beyond accuracy: Behavioral testing of NLP models with CheckList,” in Proc. 58th Annual Meeting of the Association for Computational Linguistics, 2020, pp. 4902–4912, doi: 10.18653/v1/2020.acl-main.442.

J. Brooke, “SUS: A quick and dirty usability scale,” in Usability Evaluation in Industry, P. W. Jordan, B. Thomas, B. A. Weerdmeester, and I. L. McClelland, Eds. Taylor & Francis, 1996, pp. 189–194.

T. Wolf et al., “Transformers: State-of-the-art natural language processing,” in Proc. 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, 2020, pp. 38–45, doi: 10.18653/v1/2020.emnlp-demos.6.

F. Å. Nielsen, “A new ANEW: Evaluation of a word list for sentiment analysis in microblogs,” in Proc. ESWC 2011 Workshop on Making Sense of Microposts, 2011, pp. 93–98.