Journal of King Saud University - Computer and Information Sciences · 2021 · 64 citations · 15 references
ComputingArabicComputer ScienceFeature EngineeringKnowledge TransferNatural Language ProcessingLanguage ModelingMultilingual Natural Language ProcessingRepresentation LearningText Feature EngineeringTransfer LearningWord EmbeddingsContextual EmbeddingsContextual Feature RepresentationContextual Knowledge RepresentationContextual Word EmbeddingsNeural Text RepresentationText CategorizationTransformer Language ModelsUniversal Sentence Representations
Despite that pre-trained word embedding models have advanced a wide range of natural language processing applications, they ignore the contextual information and meaning within the text. In this paper, we investigate the potential of the pre-trained Arabic BERT (Bidirectional Encoder Representations from Transformers) model to learn universal contextualized sentence representations aiming to showcase its usefulness for Arabic text Multi-class categorization. We propose to exploit the pre-trained AraBERT for contextual text representation learning in two different ways, transfer learning model and feature extractor. On the one hand, we employ the Arabic BERT (AraBERT) model after fine-tuning its parameters on the OSAC datasets to transfer its knowledge for the Arabic text categorization. On the other hand, we inquire into AraBERT performance, as a feature extractor model, by combining it with several classifiers, including CNN, LSTM, Bi-LSTM, MLP, and SVM. Finally, we conduct an exhaustive set of experiments comparing two BERT models, namely AraBERT and multilingual BERT. The findings show that the fine-tuned AraBERT model accomplishes state-of-the-art performance results and attains up to 99% in terms of F1-score and accuracy.
15
Advances in Neural Information Processing Systems 29
Onur Teymur, Kostas Zygalakis, Ben Calderhead · 2016 · 13.5K citations
Ashish Vaswani, Noam Shazeer, Niki Parmar et al. · 2025 · 6.6K citations · Full text
Distributed Representations of Sentences and Documents
Quoc V. Le, Tomáš Mikolov · arXiv (Cornell University) · 2014 · 5.1K citations · Full text
Farasa: A Fast and Furious Segmenter for Arabic
Ahmed Abdelalí, Kareem Darwish, Nadir Durrani et al. · 2016 · 372 citations · Full text