Artificial intelligence as an automated essay scoring tool: A focus on ChatGPT

Ahmet Can Uyar; Dilek Büyükahıska

Article Details

Artificial intelligence as an automated essay scoring tool: A focus on ChatGPT

Authors:

Ahmet Can Uyar | Dilek Büyükahıska

Publisher:

İzzet KARA

Main Topics:

Measurement and Evaluation in Education (Other)

Journal:

International Journal of Assessment Tools in Education Vol.12 Issue 1, pp.20-32

Document Type:

Article

Publication Date:

2/2025

Document Language:

English

Document ID (Org No.):

2001331419

pISSN:

2148-7456

Keywords:

Abstract:

 This study explores the effectiveness of using ChatGPT, an Artificial Intelligence AI language model, as an Automated Essay Scoring AES tool for grading English as a Foreign Language EFL learners essays. The corpus consists of 50 essays representing various types including analysis, compare and contrast, descriptive, narrative, and opinion essays written by 10 EFL learners at the B2 level. Human raters and ChatGPT 4o mini version scored the essays using the International English Language Testing System IELTS TASK 2 Writing band descriptors. Adopting a quantitative approach, the Wilcoxon signed-rank tests and Spearman correlation tests were employed to compare the scores generated, revealing a significant difference between the two methods of scoring, with human raters assigning higher scores than ChatGPT. Similarly, significant differences with varying degrees were also evident for each of the various types of essays, suggesting that the genre of the essays was not a parameter affecting the agreement between human raters and ChatGPT. After all, it was discussed that while ChatGPT shows promise as an AES tool, the observed disparities suggest that it has not reached sufficient proficiency for practical use. The study emphasizes the need for improvements in AI language models to meet the nuanced nature of essay evaluation in EFL contexts.

 This study explores the effectiveness of using ChatGPT, an Artificial Intelligence AI language model, as an Automated Essay Scoring AES tool for grading English as a Foreign Language EFL learners essays. The corpus consists of 50 essays representing various types including analysis, compare and contrast, descriptive, narrative, and opinion essays written by 10 EFL learners at the B2 level. Human raters and ChatGPT 4o mini version scored the essays using the International English Language Testing System IELTS TASK 2 Writing band descriptors. Adopting a quantitative approach, the Wilcoxon signed-rank tests and Spearman correlation tests were employed to compare the scores generated, revealing a significant difference between the two methods of scoring, with human raters assigning higher scores than ChatGPT. Similarly, significant differences with varying degrees were also evident for each of the various types of essays, suggesting that the genre of the essays was not a parameter affecting the agreement between human raters and ChatGPT. After all, it was discussed that while ChatGPT shows promise as an AES tool, the observed disparities suggest that it has not reached sufficient proficiency for practical use. The study emphasizes the need for improvements in AI language models to meet the nuanced nature of essay evaluation in EFL contexts.

 This study explores the effectiveness of using ChatGPT, an Artificial Intelligence AI language model, as an Automated Essay Scoring AES tool for grading English as a Foreign Language EFL learners essays. The corpus consists of 50 essays representing various types including analysis, compare and contrast, descriptive, narrative, and opinion essays written by 10 EFL learners at the B2 level. Human raters and ChatGPT 4o mini version scored the essays using the International English Language Testing System IELTS TASK 2 Writing band descriptors. Adopting a quantitative approach, the Wilcoxon signed-rank tests and Spearman correlation tests were employed to compare the scores generated, revealing a significant difference between the two methods of scoring, with human raters assigning higher scores than ChatGPT. Similarly, significant differences with varying degrees were also evident for each of the various types of essays, suggesting that the genre of the essays was not a parameter affecting the agreement between human raters and ChatGPT. After all, it was discussed that while ChatGPT shows promise as an AES tool, the observed disparities suggest that it has not reached sufficient proficiency for practical use. The study emphasizes the need for improvements in AI language models to meet the nuanced nature of essay evaluation in EFL contexts.

 This study explores the effectiveness of using ChatGPT, an Artificial Intelligence AI language model, as an Automated Essay Scoring AES tool for grading English as a Foreign Language EFL learners essays. The corpus consists of 50 essays representing various types including analysis, compare and contrast, descriptive, narrative, and opinion essays written by 10 EFL learners at the B2 level. Human raters and ChatGPT 4o mini version scored the essays using the International English Language Testing System IELTS TASK 2 Writing band descriptors. Adopting a quantitative approach, the Wilcoxon signed-rank tests and Spearman correlation tests were employed to compare the scores generated, revealing a significant difference between the two methods of scoring, with human raters assigning higher scores than ChatGPT. Similarly, significant differences with varying degrees were also evident for each of the various types of essays, suggesting that the genre of the essays was not a parameter affecting the agreement between human raters and ChatGPT. After all, it was discussed that while ChatGPT shows promise as an AES tool, the observed disparities suggest that it has not reached sufficient proficiency for practical use. The study emphasizes the need for improvements in AI language models to meet the nuanced nature of essay evaluation in EFL contexts.

Article Details

Artificial intelligence as an automated essay scoring tool: A focus on ChatGPT

Authors:

Ahmet Can Uyar | Dilek Büyükahıska

Publisher:

İzzet KARA

Main Topics:

Measurement and Evaluation in Education (Other)

Journal:

International Journal of Assessment Tools in Education Vol.12 Issue 1, pp.20-32

Document Type:

Article

Publication Date:

2/2025

Document Language:

English

Document ID (Org No.):

2001331419

pISSN:

2148-7456

Automated essay scoring

Artificial intelligence

ChatGPT

Foreign language writing

Writing evaluationAutomated essay scoring

Artificial intelligence

ChatGPT

Foreign language writing

Writing evaluation

Issues

Article Details

Authors: Ahmet Can Uyar | Dilek Büyükahıska

Publisher: İzzet KARA

Main Topics: Measurement and Evaluation in Education (Other)

Journal: International Journal of Assessment Tools in Education Vol.12 Issue 1, pp.20-32

Document Type: Article

Publication Date: 2/2025

Document Language: English

Document ID (Org No.): 2001331419

pISSN: 2148-7456

Issues

Authors:

Ahmet Can Uyar | Dilek Büyükahıska

Publisher:

İzzet KARA

Main Topics:

Measurement and Evaluation in Education (Other)

Journal:

International Journal of Assessment Tools in Education Vol.12 Issue 1, pp.20-32

Document Type:

Article

Publication Date:

2/2025

Document Language:

English

Document ID (Org No.):

2001331419

pISSN:

2148-7456