ABSTRACT

Objective

Large language models, such as Chat Generative Pre-Trained Transformer 4o (ChatGPT-4o), are increasingly used by both patients and medical professionals to access health-related information. Myasthenia gravis (MG) is a chronic autoimmune neuromuscular disorder requiring long-term treatment. Therefore, timely access to accurate medical information about MG is important. This study aimed to evaluate the accuracy, completeness, clarity, appropriateness for the target audience, risk of misinformation or harm, and readability of ChatGPT-4o-generated responses to queries about MG from patients and neurology residents, in both English and Turkish.

Methods

We developed four sets of 20 questions, frequently asked by patients and neurology residents about MG in both English and Turkish, covering pathophysiology and symptoms, diagnosis, treatment, prognosis, and daily management. ChatGPT-4o responses were generated in separate sessions on March 29, 2025. Two neurologists independently evaluated the responses using a 5-point Likert scale across five domains. Readability was assessed using the Flesch Reading Ease score, Flesch-Kincaid grade level, and Gunning-Fog index for English, and the Ateşman readability index for Turkish.

Results

Scores for accuracy, clarity, appropriateness, and risk of misinformation or harm were consistently above 4 in both languages, with clarity rated as 5 in all responses. Completeness received the lowest scores (3.5-5.0), particularly in Turkish responses to resident-directed questions. Readability was higher in Turkish. English responses to resident queries were extremely difficult to read, while patient-directed ones remained in the “difficult” to “very difficult” range. Several discrepancies were observed in specific contents between English and Turkish outputs, such as differences in differential diagnosis lists, treatment options, contraindicated medications, and thymectomy indications.

Conclusion

ChatGPT-4o produced high-quality responses overall to MG-related queries in both languages. However, language-specific inconsistencies and content omissions highlight the need for further model refinement, particularly in multilingual and professional-use contexts.

INTRODUCTION

In recent years, artificial intelligence (AI) has been increasingly used in the medical field by both patients and healthcare professionals. In particular, large language models (LLMs), a type of machine learning model designed to understand, analyze, generate, and manipulate human language, have gained popularity as fast, easy, and accessible tools for addressing a broad range of inquiries, from everyday concerns to complex academic questions (1, 2).

Myasthenia gravis (MG) is the most common disorder of the neuromuscular junction and is characterized by fatigable skeletal muscle weakness (3, 4). As a chronic autoimmune condition that necessitates long-term treatment, MG significantly impacts patients’ quality of life and requires effective patient education and engagement (5). Patients frequently use the internet to search for information about the signs and symptoms of the disease, available treatment options, and strategies for the daily management of myasthenic symptoms (6, 7). Similarly, medical professionals increasingly rely on LLMs for rapid access to information about a variety of medical conditions, including MG (6, 7). However, the accuracy and reliability of the AI-generated content vary, requiring careful evaluation, as LLMs may have a tendency to hallucinate, resulting in misinformation (8-11).

Chat Generative Pre-Trained Transformer 4 (ChatGPT-4), one of the widely used LLMs, was launched by OpenAI in March 2023 (1). ChatGPT-4 is capable of processing both text and image inputs and performing complex tasks (1). More recently, an advanced version, ChatGPT-4o, was released by OpenAI in May 2024 (12). ChatGPT-4o is superior to ChatGPT-4 in terms of speed, cost-efficiency, multimodal functionality, and multilingual performance (2, 12, 13). While ChatGPT-4o is available for free with usage limitations and also as a paid version with extended features, ChatGPT-4 is not freely accessible (13).

In this study, we aimed to evaluate and compare the accuracy, completeness, clarity, appropriateness for the target audience, risk of misinformation or harm, and readability of ChatGPT-4o-generated responses to queries about MG from patients and neurology residents, in both English and Turkish.

METHODS

Study Design and Analysis of Responses

In this cross-sectional study, we developed four sets of queries consisting of frequently asked questions about MG from patients and neurology residents, in both English and Turkish. The questions were identical across the English and Turkish versions, and the content of the questions was similar in the patient and neurology resident groups. However, while the questions were phrased in a scientific tone in the neurology resident group, plain language was preferred for the patient group. Each set of queries consisted of 20 questions, including five on pathophysiology and symptoms, three on diagnosis, four on treatment modalities, four on prognosis, and four on the daily management of MG.

Each set of queries was submitted to ChatGPT-4o separately in a new chat window on 29 March 2025. The responses generated by ChatGPT-4o were independently evaluated by two neurologists specialized in neuromuscular diseases, based on the current literature about MG (4). Evaluations were conducted across five domains: accuracy, completeness, clarity, appropriateness for the target audience, and risk of misinformation or harm. Each domain was assessed using a 5-point Likert scale, ranging from 1 (poor) to 5 (excellent). The mean of the scores given by the two experts for each domain was calculated and used for statistical analysis.

We analyzed the readability of the responses in English using Readable software (Readable.com, Horsham, United Kingdom) (14), applying the Flesch Reading Ease score (FRES), Flesch-Kincaid grade level, and Gunning-Fog index. The readability of the responses in Turkish was evaluated using the Ateşman readability index (15, 16).

Statistical Analysis

The normality of the data was assessed using the Shapiro-Wilk test. Descriptive statistics were given as mean±standard deviation or median (minimum-maximum) for continuous variables, and as frequency (percentage) for categorical variables. Group comparisons were performed using the Mann-Whitney U test or the independent samples t-test. All statistical analyses were conducted using SPSS for Windows, version 23.0 (IBM Corp., Armonk, NY, USA). Statistical significance was set as a p-value <0.05.

Ethical Approval

This cross-sectional study did not involve human participants, human tissue, or individually identifiable data. Therefore, informed consent and ethical approval were waived in accordance with institutional and national guidelines.

RESULTS

The accuracy, clarity, appropriateness for the target audience, and low risk of misinformation or harm scores for all ChatGPT-4o-generated responses were above 4 (very good) in both Turkish and English. All responses received a score of 5 points (excellent) for clarity from both experts. Among all evaluated domains, completeness received the lowest scores, ranging from 3.5 to 5 points. Although statistical analysis was not performed for the subgroups (pathophysiology and symptoms, diagnosis, treatment modalities, prognosis, and daily life management) due to the small sample size, we observed that completeness scores were higher for responses related to prognosis and daily life management than for those addressing pathophysiology and symptoms, diagnosis, and treatment modalities.

The scores for accuracy, completeness, clarity, appropriateness for the target audience, and risk of misinformation or harm were comparable between the patient and resident groups in the English responses. However, in the Turkish responses, the patient group demonstrated higher completeness scores than the resident group (Tables 1 and 2).

In the assessment of readability in English, responses to the patient group were found to be easier to read (Table 1). Readability scores were similar between the patient and neurology resident groups in Turkish responses (Table 2).

When comparing English and Turkish responses, the scores for accuracy, completeness, clarity, appropriateness, and risk of misinformation or harm did not differ in responses to patient queries. However, the completeness scores for the resident group were significantly lower in Turkish than in English (Table 3).

DISCUSSION

In this study, we examined the performance of ChatGPT-4o in responding to frequently asked queries about MG, both in English and Turkish, by patients, and neurology residents. Rather than focusing solely on quantitative scores, we also analyzed qualitative aspects such as clarity, completeness of the content, contextual appropriateness for the target audiences, and linguistic differences.

ChatGPT-4o exhibited high overall response quality and reliability in our study, as reflected by high accuracy, clarity, appropriateness scores, and low risk of misinformation. However, the performance of ChatGPT models has shown variability depending on the complexity of content and disease subtypes (17-20). ChatGPT-3.5 achieved stronger results in peripheral nerve and cerebrovascular diseases, while its performance was weaker in neuromuscular junction disorders and multiple sclerosis (17). Although ChatGPT-4’s performance matched or even surpassed physicians in multiple-choice and board-style exams (18, 19), its capacity for clinical reasoning and higher-order decision making remained limited (18). Furthermore, concerns regarding overconfidence and factual inconsistency have been raised, particularly in open-ended questions or complex medical scenarios (18-20).

The completeness scores were the lowest among all domains, which exhibited a language-dependent variation, prompting further evaluation. Completeness scores were notably lower in Turkish responses to resident queries compared to their English counterparts. This finding is consistent with prior studies suggesting that ChatGPT-4 exhibits higher performance in English, likely due to the predominance of English-language training data in the model’s training corpus (21-24). Studies evaluating the performance of ChatGPT-4 in bilingual examinations showed that the model achieved significantly higher accuracy in English than in Chinese (21), Arabic (23), and Korean (24). Language-related discrepancies in ChatGPT-4 performance have also been reported in clinical and public health domains. In a study evaluating ChatGPT’s multilingual performance in clinical nutrition advice, ChatGPT-4 produced significantly lower quality outputs in Kazakh compared to English and Russian (22). These findings collectively suggest that language-based performance biases remain a challenge and underscore the need for multilingual fine-tuning and dataset diversification to address such disparities. Although ChatGPT-4o has been promoted as a multilingually improved version of its predecessors (2, 12), to the best of our knowledge, no study has yet evaluated its multilingual performance in any medical context, including MG.

In addition to the differences in completeness scores between English and Turkish responses, our study also identified linguistic discrepancies in specific content elements. For instance, in response to the patient group’s question, “Can MG be mistaken for other conditions?”, the differential diagnoses listed differed between the English and Turkish versions. Similarly, the response to the question on treatment options included eculizumab in English but omitted it in Turkish. The list of contraindicated medications in MG also varied between the two languages. Similar discrepancies were observed in resident-directed queries. The differential diagnoses of MG were not consistent across languages, and the indications for thymectomy included different age thresholds in the English and Turkish responses. Moreover, the predictors of spontaneous remission varied between languages. These findings suggest potential inconsistencies in how ChatGPT-4o retrieves and generates language-specific medical content.

Notably, in our study, ChatGPT-4o achieved its highest completeness scores when responding to questions related to prognosis and the daily management of MG, suggesting the model’s strength in patient-centered communication. This finding is consistent with previous studies across various medical conditions (9, 11, 25). ChatGPT-4o has been shown to provide accurate and reliable responses in contexts such as keratoconus (25), prostate cancer (9), and postmenopausal osteoporosis (11), particularly when addressing follow-up care, treatment adherence, and lifestyle recommendations.

Readability remains a significant barrier to patient accessibility in AI-generated medical content. In our study, responses to both patient and resident-directed questions in Turkish were fairly easy to read. However, in their English counterparts, responses to resident-directed questions were extremely difficult to read based on the FRES, whereas responses to patient-directed questions showed higher scores, indicating relatively better accessibility. Nevertheless, the overall readability for responses to patient-directed questions remained within the “difficult” to “very difficult” range, aligning with previous studies that have highlighted the limited readability of ChatGPT-4o’s outputs in various clinical contexts (8-11,25). Encouragingly, several studies have shown that prompting ChatGPT-4o to simplify its language can significantly improve readability without compromising accuracy (9, 10). Moreover, patient perceptions of understandability may not always correspond to objective readability indices, suggesting that future models should integrate real-time feedback and personalization to improve communication effectiveness (9).

Study Limitations

This study has several limitations. First, the number of questions was limited, restricting the application of statistical analyses in certain comparisons. Second, the evaluation was based on predefined questions, which may not fully reflect the variability of real-world questions posed by patients or residents. Additionally, only two languages were assessed, limiting the generalizability of the findings to other languages. Finally, readability was evaluated using standard indices that may not accurately represent actual patient comprehension.

CONCLUSION

ChatGPT-4o demonstrated high overall performance in responding to medical queries about MG, providing accurate, clear, and contextually appropriate answers in both English and Turkish. Although minor language-related differences were observed, particularly in the completeness of complex responses and certain factual discrepancies, ChatGPT-4o shows strong potential as a supportive tool for both patient education and professional reference.