Performance comparison of data balancing techniques on hate speech detection in Turkish

Habibe Karayiğit; Ali Akdağli; Çiğdem İnan Acı

Article Details

Performance comparison of data balancing techniques on hate speech detection in Turkish

Authors:

Habibe Karayiğit | Ali Akdağli | Çiğdem İnan Acı

Publisher:

Pamukkale University

Main Topics:

Algoritmalar ve Hesaplama Kuramı | Elektrik Mühendisliği (Diğer)

Journal:

Pamukkale Üniversitesi Mühendislik Bilimleri Dergisi Vol.30 Issue 5, pp.610-621

Document Type:

Article

Publication Date:

10/2024

Document Language:

English

Document ID (Org No.):

2001330085

pISSN:

2147-5881

Keywords:

Abstract:

 Increasing hate speech on social media platforms causes psychological disorders and deep and negative effects. Automatic language classification models are needed to detect hate speech. When testing language models for hate speech, imbalanced datasets where one data class is represented much more frequently than the other can be a problem in language datasets. When the dataset is imbalanced, the classifier may be biased towards the majority class and may not perform well in the minority class. This can lead to incorrect or unreliable classification results. To solve this problem, data level balancing methods such as oversampling or undersampling are used to balance the class distribution before classifying the dataset. This study, it is aimed to achieve a successful classification model combination that detects hate speech by using data-level balancing methods. For this, a comprehensive study was carried out by applying the balancing method at eight data levels random oversampling, Synthetic Minority Oversampling Technique SMOTE, K-means SMOTE, Localized Random Affine Shadow Sample LoRAS, Text-based Generative AdversarialNetwork TextGAN, Nearmiss, Tomek Links ve Clustering-based to the Abusive Turkish Comments ATC dataset, which has an imbalanced distribution of labels, obtained from Instagram. Classification performances of data level balancing methods were evaluated with Basic Machine Learning BML and Convolutional Neural Network CNN methods. It has been observed that the CBoWCNN model based on the TextGAN data-level balancing method, as well as the Skip-gram CNN model, exhibited the best classification performance with a MacroAveraged F1 score of 0.972.

 Increasing hate speech on social media platforms causes psychological disorders and deep and negative effects. Automatic language classification models are needed to detect hate speech. When testing language models for hate speech, imbalanced datasets where one data class is represented much more frequently than the other can be a problem in language datasets. When the dataset is imbalanced, the classifier may be biased towards the majority class and may not perform well in the minority class. This can lead to incorrect or unreliable classification results. To solve this problem, data level balancing methods such as oversampling or undersampling are used to balance the class distribution before classifying the dataset. This study, it is aimed to achieve a successful classification model combination that detects hate speech by using data-level balancing methods. For this, a comprehensive study was carried out by applying the balancing method at eight data levels random oversampling, Synthetic Minority Oversampling Technique SMOTE, K-means SMOTE, Localized Random Affine Shadow Sample LoRAS, Text-based Generative AdversarialNetwork TextGAN, Nearmiss, Tomek Links ve Clustering-based to the Abusive Turkish Comments ATC dataset, which has an imbalanced distribution of labels, obtained from Instagram. Classification performances of data level balancing methods were evaluated with Basic Machine Learning BML and Convolutional Neural Network CNN methods. It has been observed that the CBoWCNN model based on the TextGAN data-level balancing method, as well as the Skip-gram CNN model, exhibited the best classification performance with a MacroAveraged F1 score of 0.972.

 Increasing hate speech on social media platforms causes psychological disorders and deep and negative effects. Automatic language classification models are needed to detect hate speech. When testing language models for hate speech, imbalanced datasets where one data class is represented much more frequently than the other can be a problem in language datasets. When the dataset is imbalanced, the classifier may be biased towards the majority class and may not perform well in the minority class. This can lead to incorrect or unreliable classification results. To solve this problem, data level balancing methods such as oversampling or undersampling are used to balance the class distribution before classifying the dataset. This study, it is aimed to achieve a successful classification model combination that detects hate speech by using data-level balancing methods. For this, a comprehensive study was carried out by applying the balancing method at eight data levels random oversampling, Synthetic Minority Oversampling Technique SMOTE, K-means SMOTE, Localized Random Affine Shadow Sample LoRAS, Text-based Generative AdversarialNetwork TextGAN, Nearmiss, Tomek Links ve Clustering-based to the Abusive Turkish Comments ATC dataset, which has an imbalanced distribution of labels, obtained from Instagram. Classification performances of data level balancing methods were evaluated with Basic Machine Learning BML and Convolutional Neural Network CNN methods. It has been observed that the CBoWCNN model based on the TextGAN data-level balancing method, as well as the Skip-gram CNN model, exhibited the best classification performance with a MacroAveraged F1 score of 0.972.

 Sosyal medya platformlarında artan nefret söylemleri, psikolojik rahatsızlıklara, derin ve olumsuz etkilere neden olmaktadır. Nefret söylemlerini tespit etmek için otomatik dil sınıflandırma modellerine ihtiyaç vardır. Nefret söylemleri için dil modelleri test edilirken, bir veri sınıfının diğerinden çok daha sık temsil edildiği dengesiz veri kümeleri dil verilerinde sorun teşkil edebilir. Veri kümesi dengesiz dağılıma sahip olduğunda, sınıflandırıcı çoğunluk sınıfına yönelik önyargılı olabilir ve azınlık sınıfında iyi performans göstermeyebilir. Bu, yanlış veya güvenilmez sınıflandırma sonuçlarına yol açabilir. Bu sorunu çözmek için veri kümesi sınıflandırılmadan önce oversampling veya undersampling gibi veri düzeyi dengeleme yöntemleri ile veri sınıfları dengelenir. Bu çalışmada, veri düzeyi dengeleme yöntemleri kullanılarak nefret söylemini tespit eden başarılı bir sınıflandırma modeli kombinasyonu elde etmek amaçlanmaktadır. Bu amaçla, Instagramdan elde edilmiş dengesiz etiket dağılımına sahip Abusive Turkish Comments ATC veri kümesine sekiz veri düzeyinde rastgele oversampling, Synthetic Minority Oversampling Technique SMOTE, Kmeans SMOTE, Localized Random Affine Shadow Sample LoRAS, Textbased Generative Adversarial Network TextGAN, Nearmiss, Tomek Links ve Clustering-based dengeleme yöntemi uygulanarak kapsamlı bir çalışma yapılmıştır. Veri düzeyi dengeleme yöntemlerinin sınıflandırma performansları Basic Machine Learning BML ve Convolutional Neural Network CNN yöntemleriyle değerlendirilmiştir. TextGAN veri düzeyi dengeleme yöntemine dayalı CBoWCNN modelinin ve Skip-gram CNN modelinin 0,972 Makro Ortalamalı F1 puanı ile en iyi sınıflandırma performansını sergilediği görülmüştür.

Article Details

Performance comparison of data balancing techniques on hate speech detection in Turkish

Authors:

Habibe Karayiğit | Ali Akdağli | Çiğdem İnan Acı

Publisher:

Pamukkale University

Main Topics:

Algoritmalar ve Hesaplama Kuramı | Elektrik Mühendisliği (Diğer)

Journal:

Pamukkale Üniversitesi Mühendislik Bilimleri Dergisi Vol.30 Issue 5, pp.610-621

Document Type:

Article

Publication Date:

10/2024

Document Language:

English

Document ID (Org No.):

2001330085

pISSN:

2147-5881

Data balancing

Social media

Machine learning

Deep learning

Natural language processing

Hate speechVeri dengeleme

Sosyal medya

Makine öğrenmesi

Derin öğrenme

Doğal dil işleme

nefret söylemi

Issues

Article Details

Authors: Habibe Karayiğit | Ali Akdağli | Çiğdem İnan Acı

Publisher: Pamukkale University

Main Topics: Algoritmalar ve Hesaplama Kuramı | Elektrik Mühendisliği (Diğer)

Journal: Pamukkale Üniversitesi Mühendislik Bilimleri Dergisi Vol.30 Issue 5, pp.610-621

Document Type: Article

Publication Date: 10/2024

Document Language: English

Document ID (Org No.): 2001330085

pISSN: 2147-5881

Issues

Authors:

Habibe Karayiğit | Ali Akdağli | Çiğdem İnan Acı

Publisher:

Pamukkale University

Main Topics:

Algoritmalar ve Hesaplama Kuramı | Elektrik Mühendisliği (Diğer)

Journal:

Pamukkale Üniversitesi Mühendislik Bilimleri Dergisi Vol.30 Issue 5, pp.610-621

Document Type:

Article

Publication Date:

10/2024

Document Language:

English

Document ID (Org No.):

2001330085

pISSN:

2147-5881