An Intelligent Multimodal Medical Diagnosis System based on Patients’ Medical Questions and Structured Symptoms for Telemedicine

Abstract The massive increase in health-related digital data has revolutionized the power of machine learning algorithms to produce more salient information. Digital health data consists of various information, including diagnoses, treatments, and medications. Diagnosis is a fundamental service provided by healthcare agents for improving patient health. However, diagnosis errors result in treating the patient incorrectly or at an improper time causing harm to them. Computer-aided diagnosis systems are intelligent methods that help clinicians in making correct decisions by mitigating the potential of clinical cognitive errors. This paper proposes an intelligent diagnosis decision support system as part of a telemedicine 1 platform for serving the Middle East and North Africa (MENA) region. The proposed system utilizes a huge health-related dataset curated by the Altibbi company, which includes numerous unstructured patient questions written in different dialects of the Arabic language, and structured symptoms identified by general practitioners (GPs). The system encompasses a fusion of machine learning models trained based on two modalities: the symptoms and the medical questions of the patients. Various feature representation techniques (i.e., statistical and word embeddings) and machine learning classifiers, including Logistic Regression (LR), Random Forest (RF), Stochastic Gradient Descent Classifier (SGDClassifier), and variants of the Multilayer Perceptron (MLP) classifier have been used for experiments. The output of the combination of the two modalities has shown promising predictive ability in terms of the classification accuracy, which is 84.9%. The obtained results indicate the potential of the model in predicting the diagnosis of possible patient conditions based on the given symptoms and patients’ questions, which consequently can aid doctors in making the right decisions.