Aims: To evaluate the diagnostic accuracy of Claude-3, a large language model, in detecting pathological features and diagnosing retinitis pigmentosa and cone-rod dystrophy using pattern electroretinography data.
Methods: A subset of pattern electroretinography measurements from healthy individuals, patients with retinitis pigmentosa and cone-rod dystrophy was randomly selected from the PERG-IOBA dataset. The pattern electroretinography and clinical data, including age, gender, visual acuities, were provided to Claude-3 for analysis and diagnostic predictions. The model’s accuracy was assessed in two scenarios: “first choice,” evaluating the accuracy of the primary differential diagnosis and “top 3,” evaluating whether the correct diagnosis was included within the top three differential diagnoses.
Results: A total of 46 subjects were included in the study: 20 healthy individuals, 13 patients with retinitis pigmentosa, 13 patients with cone-rod dystrophy. Claude-3 achieved 100% accuracy in detecting the presence or absence of pathology. In the “first choice” scenario, the model demonstrated moderate accuracy in diagnosing retinitis pigmentosa (61.5%) and cone-rod dystrophy (53.8%). However, in the “top 3” scenario, the model’s performance significantly improved, with accuracies of 92.3% for retinitis pigmentosa and 76.9% for cone-rod dystrophy.
Conclusion: This is the first study to demonstrate the potential of large language models, specifically Claude-3, in analyzing pattern electroretinography data to diagnose retinal disorders. Despite some limitations, the model’s high accuracy in detecting pathologies and distinguishing between specific diseases highlights the potential of large language models in ocular electrophysiology. Future research should focus on integrating multimodal data, and conducting comparative analyses with human experts.