Investigation of ChatGPT and Real Raters in Scoring Open-Ended Items in Terms of Inter-Rater Reliability

Seda Demir

Article Details

Investigation of ChatGPT and Real Raters in Scoring Open-Ended Items in Terms of Inter-Rater Reliability

Authors:

Seda Demir

Publisher:

Tokat Gaziosmanpasa University

Main Topics:

Eğitimde ve Psikolojide Ölçme Teorileri ve Uygulamaları

Journal:

Uluslararası Türk Eğitim Bilimleri Dergisi Vol.2023 Issue 21, pp.1072-1099

Document Type:

Article

Publication Date:

10/2023

Document Language:

English

Document ID (Org No.):

2001303547

pISSN:

2148-2314

Keywords:

Abstract:

 The aim of this study is to examine the inter-rater reliability of the responses to open-ended items scored by ChatGPT, an artificial intelligence-based tool, and two real raters according to the scoring keys. The study group consists of 30 students, aged between 13 and 15, studying in Eskişehir province in the 2022-2023 academic year. The data of the study were collected face-to-face with the help of 16 open-ended items selected from the sample questions published in the International Student Assessment Program-PISA Reading Skills. Correlation, percentage of agreement and the Generalizability theory were used to determine inter-rater reliability. SPSS 25 was used for correlation analysis, Excel for percentage of agreement analysis, and EduG 6.1 for the Generalizability theory analysis. The results of the study showed that there was a positive and high level of correlation between the raters, the raters showed a high level of agreement, and the reliability G coefficients calculated using the Generalizability theory were lower than the correlation values and percentage of agreement. In addition, it was determined that all raters showed excellent positive correlation and full agreement with each other in the scoring of the answers given to the short-answer items whose answers were directly in the text. In addition, according to the results of the Generalizability theory, it was found out that the items i explained the total variance the most among the main effects and the student-item interaction sxi explained the most among the interaction effects. As a result, it can be suggested to educators to get support from artificial intelligence-based tools such as ChatGPT when scoring open-ended items that take a long time to score, especially in crowded classes or when time is limited.

 The aim of this study is to examine the inter-rater reliability of the responses to open-ended items scored by ChatGPT, an artificial intelligence-based tool, and two real raters according to the scoring keys. The study group consists of 30 students, aged between 13 and 15, studying in Eskişehir province in the 2022-2023 academic year. The data of the study were collected face-to-face with the help of 16 open-ended items selected from the sample questions published in the International Student Assessment Program-PISA Reading Skills. Correlation, percentage of agreement and the Generalizability theory were used to determine inter-rater reliability. SPSS 25 was used for correlation analysis, Excel for percentage of agreement analysis, and EduG 6.1 for the Generalizability theory analysis. The results of the study showed that there was a positive and high level of correlation between the raters, the raters showed a high level of agreement, and the reliability G coefficients calculated using the Generalizability theory were lower than the correlation values and percentage of agreement. In addition, it was determined that all raters showed excellent positive correlation and full agreement with each other in the scoring of the answers given to the short-answer items whose answers were directly in the text. In addition, according to the results of the Generalizability theory, it was found out that the items i explained the total variance the most among the main effects and the student-item interaction sxi explained the most among the interaction effects. As a result, it can be suggested to educators to get support from artificial intelligence-based tools such as ChatGPT when scoring open-ended items that take a long time to score, especially in crowded classes or when time is limited.

 The aim of this study is to examine the inter-rater reliability of the responses to open-ended items scored by ChatGPT, an artificial intelligence-based tool, and two real raters according to the scoring keys. The study group consists of 30 students, aged between 13 and 15, studying in Eskişehir province in the 2022-2023 academic year. The data of the study were collected face-to-face with the help of 16 open-ended items selected from the sample questions published in the International Student Assessment Program-PISA Reading Skills. Correlation, percentage of agreement and the Generalizability theory were used to determine inter-rater reliability. SPSS 25 was used for correlation analysis, Excel for percentage of agreement analysis, and EduG 6.1 for the Generalizability theory analysis. The results of the study showed that there was a positive and high level of correlation between the raters, the raters showed a high level of agreement, and the reliability G coefficients calculated using the Generalizability theory were lower than the correlation values and percentage of agreement. In addition, it was determined that all raters showed excellent positive correlation and full agreement with each other in the scoring of the answers given to the short-answer items whose answers were directly in the text. In addition, according to the results of the Generalizability theory, it was found out that the items i explained the total variance the most among the main effects and the student-item interaction sxi explained the most among the interaction effects. As a result, it can be suggested to educators to get support from artificial intelligence-based tools such as ChatGPT when scoring open-ended items that take a long time to score, especially in crowded classes or when time is limited.

 Bu araştırmanın amacı, açık uçlu maddelere verilen yanıtlar için yapay zekâ tabanlı bir araç olan ChatGPT ve iki gerçek puanlayıcı tarafından puanlama anahtarlarına göre yapılan puanlamanın puanlayıcılar arası güvenirlik bakımından incelenmesidir. Araştırmanın çalışma grubunu, 2022-2023 eğitim öğretim yılında Eskişehir ilinde öğrenim gören 13-15 yaş grubundan 30 öğrenci oluşturmaktadır. Araştırmanın verileri, Uluslararası Öğrenci Değerlendirme Programı-PISA Okuma Becerileri alanında yayımlanmış örnek sorular arasından seçilen 16 açık uçlu madde yardımıyla yüz yüze toplanmıştır. Puanlayıcılar arası güvenirliği belirlemek amacıyla korelasyon, uyuşma yüzdesi ve Genellenebilirlik kuramından yararlanılmıştır. Korelasyon analizlerinde SPSS 25, uyuşma yüzdesinin analizlerinde Excel ve genellenebilirlik kuramı analizlerinde EduG 6.1 programları kullanılmıştır. Araştırma sonuçları, puanlayıcılar arasında pozitif yönlü ve yüksek düzeyde bir ilişki olduğunu, puanlayıcıların yüksek oranda uyuşma gösterdiğini ve Genellenebilirlik kuramı kullanılarak hesaplanan güvenirlik G katsayılarının, korelasyon değerleri ve uyuşma yüzdelerine kıyasla daha düşük olduğunu göstermiştir. Bunun yanı sıra cevabı doğrudan metnin içinde geçen ve kısa cevaplı olan maddelere verilen yanıtların puanlanmasında tüm puanlayıcıların birbirleriyle mükemmel pozitif korelasyon ve tam uyuşma gösterdiği belirlenmiştir. Ayrıca Genellenebilirlik kuramı sonuçlarına göre toplam varyansı ana etkiler arasından en çok maddelerin m, etkileşim etkileri arasından ise en çok öğrenci-madde etkileşiminin öxm açıkladığı görülmüştür. Sonuçta, uygulamaya dönük olarak eğitimcilere, kalabalık sınıflarda veya zamanın kısıtlı olduğu durumlarda özellikle puanlaması uzun zaman alan açık uçlu maddeler puanlanırken ChatGPT gibi yapay zekâ tabanlı araçlardan destek almaları önerilebilir.

Article Details

Investigation of ChatGPT and Real Raters in Scoring Open-Ended Items in Terms of Inter-Rater Reliability

Authors:

Seda Demir

Publisher:

Tokat Gaziosmanpasa University

Main Topics:

Eğitimde ve Psikolojide Ölçme Teorileri ve Uygulamaları

Journal:

Uluslararası Türk Eğitim Bilimleri Dergisi Vol.2023 Issue 21, pp.1072-1099

Document Type:

Article

Publication Date:

10/2023

Document Language:

English

Document ID (Org No.):

2001303547

pISSN:

2148-2314

Inter-rater Reliability

Scoring Open-Ended Items

Generalizability Theory

ChatGPTPuanlayıcılar Arası Güvenirlik

Açık Uçlu Maddelerin Puanlanması

Genellenebilirlik Kuramı

ChatGPT

Issues

Article Details

Authors: Seda Demir

Publisher: Tokat Gaziosmanpasa University

Main Topics: Eğitimde ve Psikolojide Ölçme Teorileri ve Uygulamaları

Journal: Uluslararası Türk Eğitim Bilimleri Dergisi Vol.2023 Issue 21, pp.1072-1099

Document Type: Article

Publication Date: 10/2023

Document Language: English

Document ID (Org No.): 2001303547

pISSN: 2148-2314

Issues

Authors:

Seda Demir

Publisher:

Tokat Gaziosmanpasa University

Main Topics:

Eğitimde ve Psikolojide Ölçme Teorileri ve Uygulamaları

Journal:

Uluslararası Türk Eğitim Bilimleri Dergisi Vol.2023 Issue 21, pp.1072-1099

Document Type:

Article

Publication Date:

10/2023

Document Language:

English

Document ID (Org No.):

2001303547

pISSN:

2148-2314