Comparative Analysis of the Readability of Artificial Intelligence-Supported Educational Materials for Menengitis: Chatgpt and Gemini

Abstract

Background: Artificial intelligence (AI)-based large language models (LLMs) are increasingly used to generate patient educational materials. However, the readability of AI-generated information is a key determinant of patient comprehension and health literacy. This study compared the readability of educational materials on meningitis generated by ChatGPT and Gemini.
Methods: This cross-sectional comparative study evaluated 80 AI-generated educational texts on meningitis, including 40 responses generated by ChatGPT and 40 by Gemini using an identical standardized prompt. Readability was assessed using nine validated readability indices: Automated Readability Index (ARI), Flesch Reading Ease, Flesch-Kincaid Grade Level, Gunning Fog Index, Coleman-Liau Index, SMOG Index, Linsear Write Formula, Dale-Chall Readability Score, and Spache Readability Formula. Continuous variables were expressed as mean ± standard deviation, and comparisons between groups were performed using appropriate statistical tests. A p-value <0.05 was considered statistically significant.
Results: Most readability indices did not differ significantly between ChatGPT and Gemini. ARI, Flesch Reading Ease, Gunning Fog Index, SMOG Index, Linsear Write Formula, Dale-Chall Readability Score, and Spache Readability Formula were comparable between the two platforms (all p >0.05). However, ChatGPT-generated materials demonstrated significantly higher Flesch-Kincaid Grade Level scores than Gemini (9.25 ± 2.64 vs. 7.66 ± 2.42, p <0.001) and significantly higher Coleman-Liau Index scores (16.74 ± 2.11 vs. 12.76 ± 2.12, p =0.03), indicating greater reading complexity. Overall, both AI models generated educational materials above the sixth-grade reading level recommended for patient education.
Conclusions: ChatGPT and Gemini produced patient educational materials on meningitis with generally similar readability; however, ChatGPT generated significantly more complex texts according to the Flesch-Kincaid Grade Level and Coleman-Liau Index. Despite their potential as educational tools, both AI platforms produced content that exceeded recommended readability levels for the general population. Further refinement of AI-generated health information is needed to improve accessibility and support patients with diverse health literacy levels.