Comparison of the accuracy performances of the Gemini Advanced, the GPT-4, the Copilot, and the GPT-3.5 models in medical imaging systems: A Zero-shot prompting analysis

Alpaslan Koç; Ayşe Betül Öztiryaki

Article Details

Comparison of the accuracy performances of the Gemini Advanced, the GPT-4, the Copilot, and the GPT-3.5 models in medical imaging systems: A Zero-shot prompting analysis

Authors:

Alpaslan Koç | Ayşe Betül Öztiryaki

Publisher:

Nigde Omer Halisdemir University

Main Topics:

Doğal Dil İşleme | Planlama ve Karar Verme | Biyomedikal Bilimler ve Teknolojiler

Journal:

Niğde Ömer Halisdemir Üniversitesi Mühendislik Bilimleri Dergisi Vol.13 Issue 4, pp.1216-1223

Document Type:

Article

Publication Date:

10/2024

Document Language:

English

Document ID (Org No.):

2001329709

pISSN:

2564-6605

Keywords:

Abstract:

 Large Language Models LLMs have gained popularity across healthcare and attracted the attention of researchers of various medical specialties. Determining which model performs well in which circumstances is essential for accurate results. This study aims to compare the accuracy of recently developed LLMs for medical imaging systems and to evaluate the reliability of LLMs in terms of correct responses. A total of 400 questions were divided into four categories: X-ray, ultrasound, magnetic resonance imaging, and nuclear medicine. LLMs responses were evaluated with a zero-prompting approach by measuring the percentage of correct answers. McNemar tests were used to evaluate the significance of differences between models, and Cohen kappa statistics were used to determine the reliability of the models. Gemini Advanced, GPT-4, Copilot, and GPT-3.5 resulted in accuracy rates of 86.25, 84.25, 77.5, and 59.75, respectively. There was a strong correlation between Gemini Advanced and the GPT-4 compared with other models, К0.762. This study is the first that analyzes the accuracy of responses of recently developed LLMs: Gemini Advanced, GPT-4, Copilot, and GPT-3.5 on questions related to medical imaging systems. And a comprehensive dataset with three question types was created within medical imaging systems, which was evenly distributed from various sources.

 Large Language Models LLMs have gained popularity across healthcare and attracted the attention of researchers of various medical specialties. Determining which model performs well in which circumstances is essential for accurate results. This study aims to compare the accuracy of recently developed LLMs for medical imaging systems and to evaluate the reliability of LLMs in terms of correct responses. A total of 400 questions were divided into four categories: X-ray, ultrasound, magnetic resonance imaging, and nuclear medicine. LLMs responses were evaluated with a zero-prompting approach by measuring the percentage of correct answers. McNemar tests were used to evaluate the significance of differences between models, and Cohen kappa statistics were used to determine the reliability of the models. Gemini Advanced, GPT-4, Copilot, and GPT-3.5 resulted in accuracy rates of 86.25, 84.25, 77.5, and 59.75, respectively. There was a strong correlation between Gemini Advanced and the GPT-4 compared with other models, К0.762. This study is the first that analyzes the accuracy of responses of recently developed LLMs: Gemini Advanced, GPT-4, Copilot, and GPT-3.5 on questions related to medical imaging systems. And a comprehensive dataset with three question types was created within medical imaging systems, which was evenly distributed from various sources.

 Large Language Models LLMs have gained popularity across healthcare and attracted the attention of researchers of various medical specialties. Determining which model performs well in which circumstances is essential for accurate results. This study aims to compare the accuracy of recently developed LLMs for medical imaging systems and to evaluate the reliability of LLMs in terms of correct responses. A total of 400 questions were divided into four categories: X-ray, ultrasound, magnetic resonance imaging, and nuclear medicine. LLMs responses were evaluated with a zero-prompting approach by measuring the percentage of correct answers. McNemar tests were used to evaluate the significance of differences between models, and Cohen kappa statistics were used to determine the reliability of the models. Gemini Advanced, GPT-4, Copilot, and GPT-3.5 resulted in accuracy rates of 86.25, 84.25, 77.5, and 59.75, respectively. There was a strong correlation between Gemini Advanced and the GPT-4 compared with other models, К0.762. This study is the first that analyzes the accuracy of responses of recently developed LLMs: Gemini Advanced, GPT-4, Copilot, and GPT-3.5 on questions related to medical imaging systems. And a comprehensive dataset with three question types was created within medical imaging systems, which was evenly distributed from various sources.

 Büyük dil modelleri LLMler sağlık hizmetlerinde popülerlik kazanmış ve çeşitli tıbbi uzmanlık alanlarındaki araştırmacıların ilgisini çekmektedir. Doğru sonuçlar için hangi modelin hangi koşullarda iyi performans gösterdiğini belirlemek önemlidir. Bu çalışma, yeni geliştirilen büyük dil modellerinin tıbbi görüntüleme sistemleri için doğruluklarını karşılaştırmayı ve bu modellerin verdikleri doğru yanıtlar açısından birbirleri arasındaki uyumluluklarını değerlendirmeyi amaçlamaktadır. Bu değerlendirme için toplam 400 soru X-ray, ultrason, manyetik rezonans görüntüleme ve nükleer tıp görüntüleme olarak dört kategoriye ayrılmıştır. Büyük dil modellerinin yanıtları, doğru yanıtların yüzdesi ölçülerek sıfır-atışlı yönlendirme yaklaşımıyla değerlendirilmiştir. Modeller arasındaki farkların anlamlılığını değerlendirmek için McNemar testi, modellerin güvenilirliğini belirlemek için ise Cohen kappa istatistiği kullanılmıştır. Gemini Advanced, GPT-4, Copilot ve GPT-3.5 için sırasıyla 86.25, 84.25, 77.5 ve 59.75 doğruluk oranları elde edilmiştir. Diğer modellerle karşılaştırıldığında Gemini Advanced ve GPT-4 arasında güçlü bir korelasyon bulunmuştur, К0,762. Bu çalışma, yakın zamanda geliştirilen Gemini Advanced, GPT-4, Copilot ve GPT-3.5in tıbbi görüntüleme sistemleriyle ilgili sorulara verdiği yanıtların doğruluğunu analiz eden ilk çalışmadır. Ayrıca bu çalışma ile tıbbi görüntüleme sistemleri ile ilgili çeşitli kaynaklardan üç soru tipinden oluşan kapsamlı bir veri seti oluşturulmuştur.

Article Details

Comparison of the accuracy performances of the Gemini Advanced, the GPT-4, the Copilot, and the GPT-3.5 models in medical imaging systems: A Zero-shot prompting analysis

Authors:

Alpaslan Koç | Ayşe Betül Öztiryaki

Publisher:

Nigde Omer Halisdemir University

Main Topics:

Doğal Dil İşleme | Planlama ve Karar Verme | Biyomedikal Bilimler ve Teknolojiler

Journal:

Niğde Ömer Halisdemir Üniversitesi Mühendislik Bilimleri Dergisi Vol.13 Issue 4, pp.1216-1223

Document Type:

Article

Publication Date:

10/2024

Document Language:

English

Document ID (Org No.):

2001329709

pISSN:

2564-6605

Large language models

Medical imaging systems

Generative ai

Comparison of the accuracy

Foundation modelsBüyük dil modelleri

Tıbbi görüntüleme sistemleri

Üretken yapay zeka

Doğruluğun karşılaştırılması

Alt yapı modelleri

Issues

Article Details

Authors: Alpaslan Koç | Ayşe Betül Öztiryaki

Publisher: Nigde Omer Halisdemir University

Main Topics: Doğal Dil İşleme | Planlama ve Karar Verme | Biyomedikal Bilimler ve Teknolojiler

Journal: Niğde Ömer Halisdemir Üniversitesi Mühendislik Bilimleri Dergisi Vol.13 Issue 4, pp.1216-1223

Document Type: Article

Publication Date: 10/2024

Document Language: English

Document ID (Org No.): 2001329709

pISSN: 2564-6605

Issues

Authors:

Alpaslan Koç | Ayşe Betül Öztiryaki

Publisher:

Nigde Omer Halisdemir University

Main Topics:

Doğal Dil İşleme | Planlama ve Karar Verme | Biyomedikal Bilimler ve Teknolojiler

Journal:

Niğde Ömer Halisdemir Üniversitesi Mühendislik Bilimleri Dergisi Vol.13 Issue 4, pp.1216-1223

Document Type:

Article

Publication Date:

10/2024

Document Language:

English

Document ID (Org No.):

2001329709

pISSN:

2564-6605