Research and Publications
Objective and peer-reviewed research can raise the industry bar for quality, safety, and performance. Here's some research that looks at the performance and efficacy of Ada's technology.
Scroll through or navigate to a topic of interest below.
Safety and accuracy
- Breadth of coverage and accuracy of Chinese language digital symptom assessment apps for suggesting conditions and urgency advice: A clinical vignettes evaluation
- China’s 8 most popular symptom checkers showed a range of coverage, safety, and accuracy, factors relevant to user and physician choice.
- Ada offered the highest coverage (99.5%) and advice safety (99.5%) compared with the average (69.6% and 90.3%, respectively).
- Ada’s accuracy was significantly higher, and more than twice the average (61.5% vs 28.7%).
Chinese Journal of General Practitioners, 2023 - Comparison of physician and artificial intelligence-based symptom checker diagnostic accuracy
- Ada was significantly more accurate in the recognition of inflammatory rheumatic diseases (IRD) than physicians (70% vs physicians' 54%), which included experienced rheumatology physicians.
- Ada was more accurate than physicians in suggesting the correct final diagnosis (54% of cases vs physicians' 32%).
- Highlights the potential of using symptom checkers early in the patient journey and the advantages of considering complete patient information to establish a correct diagnosis.
- Shows the potential of symptom checkers to make accurate triage and diagnostic decisions in rheumatology.
Rheumatology International, 2022 - Online symptom checkers lack diagnostic accuracy for skin rashes
- Ada outperformed all other symptom checkers in suggesting the correct diagnosis for skin rashes.
- Ada's top condition suggestion matched the correct diagnosis in 73.3% of cases, vs an average of 30.8%.
- The correct diagnosis was provided within Ada's top three condition suggestions in 93.3% of instances vs an average of 43.3%.
Journal of the American Academy of Dermatology, 2022 - Safety of Triage Self-Assessment Using a Symptom Assessment App for Walk-in Patients in the Emergency Care Setting: Observational Prospective Cross-sectional Study
- Real-world hospital ED study of 378 patients, comparing the safety of Ada’s urgency advice to that of the Manchester Triage System (MTS).
- Ada demonstrated a high rate of safety (94.7%) over all medical specialties of the Emergency Department, with a focus on Internal Medicine, Orthopedics and Trauma, and Neurology.
- From the 3 lowest MTS categories, 43.4% could have safely accessed lower urgency care such as seeing a GP or managing their symptoms at home.
- Ada has the potential to relieve pressure on Emergency Departments by guiding patients to lower urgency care if used at home.
JMIR mHealth and uHealth, 2022 - Diagnostic Performance of an App-Based Symptom Checker in Mental Disorders: Comparative Study in Psychotherapy Outpatients
- One of Ada's top 5 condition suggestions matched therapist's formal diagnoses in 69% of cases, and Ada's top suggestion matched in 51% of cases.
- Symptom checkers have potential to complement the mental health diagnostic process, but further investigation is recommended.
- The usability of Ada was rated as high in the System Usability Scale.
JMIR Mental Health, 2022 - The Diagnostic Efficacy of an App-based Diagnostic Health Care Application in the Emergency Room: eRadaR-Trial. A Prospective, Double-Blinded, Observational Study
- When used in combination with ER physicians assessments, Ada significantly increased diagnostic accuracy (87.3%) compared with an ER physician alone (80.9%).
- Patients with early diagnosis and rapid treatment allocation exhibited significantly reduced complications and length of hospital stay.
- AI tools have the potential to benefit the diagnostic efficacy of clinicians and improve quality of care.
Annals of Surgery, 2022 - Evaluation of Diagnostic and Triage Accuracy and Usability of a Symptom Checker in an Emergency Department: Observational Study
- One of Ada’s top 5 condition suggestions matched the correct diagnosis in 70% of cases, close to physicians' top 3 average of 68.9%.
- 90% of patients found Ada easy to use, and 83% considered Ada to be an efficient way to document their medical problems quickly.
- Ada’s correct ‘diagnoses’ were nearly identical to one of the board-certified emergency physicians'.
JMIR Mhealth Uhealth, 2022 - Prospective Feasibility Study of a Novel Diagnostic Decision Support System (DDSS) for Medical Professionals
- Ada’s DDSS prototype could support doctors’ decision-making by showing them new pathways and potential diagnoses.
- Ada provided accurate decision support in the clinical inpatient setting for people presenting with dyspnea.
JMIR Formative Research, 2022 - Actionable absolute risk prediction of atherosclerotic cardiovascular disease: a behavior-management approach based on data from 464,547 UK Biobank participants
- Using a dataset from 464,547 patients, Ada's CVD risk prediction tool achieved superior performance to several established general CVD risk prediction models.
- Ada could be used to direct individuals to app-based behavior change recommendations for risk reduction and guidance.
PLOS ONE, 2021 - The quality of condition suggestions and urgency advice provided by the Ada symptom assessment app assessed with independently generated vignettes optimized for Australia
- Ada identified the correct condition in the top 3 suggestions in 83% of cases, compared to the next best of 77%, including Australia-specific scenarios.
- 63% of Ada’s urgency advice matched the gold standard, compared to the next highest app’s score of 52%
Australian Journal of Primary Health, 2021 - Accuracy of online symptom checkers and the potential impact on service utilisation
- Ada had 73% accuracy compared to a 38% all-app average.
- Ada's exhibited 97% disposition safety compared to an 83% all-app average.
PLOS ONE, 2021 - How accurate are digital symptom assessment apps for suggesting conditions and urgency advice?: A clinical vignettes comparison to GPs
- The utility of symptom checkers relies on safety, coverage, and accuracy.
- Ada’s advice was safe in 97% of cases, on par with GPs and with 99% condition coverage for the clinical scenarios.
- Ada showed 70% accuracy for top 3 suggestion fit vs competitor average of 38%.
BMJ Open, 2020 - Use Characteristics and Triage Acuity of a Digital Symptom Checker in a Large Integrated Health System: Population-Based Descriptive Study
- 46.4% of 26,646 Ada assessments were completed outside primary care clinic hours.
- Ada’s recommendations were comparable to triage nurses’.
Journal of Medical Internet Research, 2020 - Accuracy of a chatbot (Ada) in the diagnosis of mental disorders: Comparative case study with lay and expert users
- Ada could help identify mental health disorders in adults.
JMIR Formative Research, 2019 - From symptom to diagnosis-symptom checkers re-evaluated : Are symptom checkers finally sufficient and accurate to use? An update from the ENT perspective
- Ada ranked highly on precision out of the 24 tested in the ear, nose, and throat (ENT) discipline.
HNO, 2019 - What happened when Pulse tested symptom checker apps
- Ada was the most accurate of the 4 apps tested.
- Ada was found to be fast and easy to use.
Pulse Today, 2019 - Can you really trust the medical apps on your phone?
- Ada was "by far the best" app out of the 4 tested, asking clear questions and providing the best condition suggestions.
Wired UK, 2017
Patient experience
- AI-based Symptom Assessment as the Digital Front Door to Care and to Enhanced Physician/Patient Interaction: Real-World Evidence from Portugal and the Netherlands
- 93% of Ada’s assessments provided useful information prior to consultation with 83% of Ada’s top condition suggestions matching the physician’s primary diagnosis.
- In 52% of appointments, Ada saved more than 1 minute of primary care consultation time.
- Most physicians agreed that Ada is safe and effective, guiding patients to the correct care, providing useful information, and saving time in consultations.
Healthcare Information and Management Systems Society (HIMSS), 2023 - Improving emergency department patient-doctor conversation through an artificial intelligence symptom-taking tool: an action-oriented design pilot study
- Ada supported conversation and improved rapport for 90% of patients, 73% of physicians, and 100% of nurses.
- Patients reported high usability and understood the tool’s questions.
JMIR Formative Research, 2022 - Optimization of Patient Flow in Urgent Care Centers Using a Digital Tool for Recording Patient Symptoms and History: Simulation Study
- Ada could decrease the average triage nurse waiting times by 54%.
JMIR Formative Research, 2021 - Patients’ utilization and perception of an artificial intelligence-based symptom assessment and advice technology in a British primary care waiting room: Exploratory pilot study
- 97.8% of patients rated Ada very or quite easy to use.
- 12.8% or patients would have delayed their appointment or used lower-intensity care.
JMIR Human Factors, 2020
Global health and universal coverage
- Addressing the Challenges of Adapting an AI-Powered Symptom Assessment and Care Navigation Tool for South African Users
- The readability of Ada's consumer-facing medical text was reduced from grade 11, to grade 7 without compromising on medical quality.
- AI companies can reduce the burden on vulnerable healthcare systems, but should be designed and adapted to the needs of the local population.
- Efforts should be made to reduce bias in AI by using white-box systems.
Pending publication, The Lancet, 2022 - Investigating the Potential for Clinical Decision Support in Sub-Saharan Africa With AFYA (Artificial Intelligence-Based Assessment of Health Symptoms in Tanzania): Protocol for a Prospective, Observational Pilot Study
- Protocol for a clinical study set in a Tanzanian district hospital outpatient clinic to investigate the usefulness of a prototype diagnosis decision support system (DDSS) in the hands of a mid-level healthcare worker.
- The diagnostic accuracy and qualitative data on the usability, usefulness, and acceptance of the prototype DDSS will provide insights on the appropriateness of the prototype DDSS interface.
JMIR Research Protocols, 2022 - Study protocol for a pilot prospective, observational study investigating the condition suggestion and urgency advice accuracy of a symptom assessment app in Sub-Saharan Africa: the AFYA-'Health' study
- Protocol for first-of-type clinical study set in a Tanzanian district hospital outpatient clinic to investigate the usefulness of symptom assessment applications in LMICs.
- Evaluation of Ada's compared to gold-standard differential diagnoses and triage advice for participants with various conditions and age groups, including children.
BMJ Open, 2022 - Addressing the challenges of adapting an AI-powered symptom assessment and care navigation tool for South African (SA) users
- Ada’s knowledge base was adapted to account for differences in incidence and disease presentation in South Africa, including optimizing 25 maternal and 25 pediatric conditions.
- Ada’s readability was lowered from grade 11 level to below grade 8 level (7.4) while still maintaining medical accuracy to improve accessibility and uptake.
- White-box AI like Ada, based on interpretable models that can be explained and easily understood by a human, can control and reduce data bias.
2022 - Challenges and solutions in adapting an AI-powered health management tool for South Africa
- AI solutions can only improve health outcomes in LMICs if they are designed and adapted with the needs of the local population in mind.
- To account for regional differences in incidence and disease presentation, 51 maternal and pediatric health conditions were optimized in Ada for South Africa.
- Ada’s content readability score has been reduced from grade 11 to below grade 8 (7.4±0.8) while still maintaining medical accuracy.
- White-box AI-approach like Ada’s makes it possible to capture the differences in underrepresented populations and low-income countriea.
European Journal of Public Health, 2021
Rare diseases
- Leveraging medical-AI to speed up Cold Agglutinin Disease detection
- The rare Cold Agglutinin Disease model was created and incorporated into Ada’s medical knowledge and reasoning engine.
- In the first 30 days, CAD was suggested among the top 3 conditions in 48 assessments.
- Integrating CAD in a symptom assessment can help speed up disease detection, potentially providing quicker access to healthcare systems.
Gesellschaft für Thrombose- und Hämostaseforschung (GTH), 2023 - Medical expert knowledge meets AI: How expert knowledge can improve symptom assessment apps - a new approach in rare diseases
- A new approach of conducting expert interviews to create 11 clinical vignettes on lysosomal storage diseases (LSDs) which were then used to update Ada’s condition models.
- 15 LSD patients and 9 LSD experts will rate both new and old disease models in Ada to compare efficacy.
- This novel approach can potentially shorten the ‘time to diagnosis’ for rare diseases.
European Conference on Rare Disease (ECRD), 2022 - What is confirmed in the diagnostics of autoinflammatory fever diseases? (German language)
- Ada could help detect genetic auto-inflammatory disorders, including the important group of Periodic Fever Syndromes.
The Internist (Der Internist), 2021 - Health economic benefits through the use of diagnostic support systems and expert knowledge
- Retrospective analysis of the diagnostic costs of rare inflammatory systemic diseases and the impact of a prototype Ada as a diagnostic decision support system (DDSS).
- If this Ada DDSS prototype were released, it has the potential to lead to earlier diagnosis and could save between 50% and 70% of diagnostic costs.
BMC Health Services Research, 2021 - Zebras or horses? How common are rare diseases on a medical symptom checker, Ada?
- Rare diseases were suggested as the most likely condition in 4% of Ada symptom assessments and among the top five suggestions in 17% of assessments.
- This is in line with epidemiological studies suggesting that ~2–4% of the population is living with a rare disease.
- Ada shows the potential to shorten the diagnostic odyssey of people living with rare diseases.
RE(ACT) Congress - International Rare Diseases Research Consortium, Berlin, 2020 - Can a decision support system accelerate rare disease diagnosis? Evaluating the potential impact of Ada DX in a retrospective study
- 89.25% of Ada’s top suggestions matched the confirmed diagnosis.
- 33.3% of participants could have received their rare disease diagnosis in the first documented clinical visit with Ada.
Orphanet Journal of Rare Diseases, 2019
Public health and surveillance
- How can symptom assessment apps contribute to early cancer detection?
- Ada has the potential to help surface users who should seek care, including possible early cases and users who do not participate in screening programs.
- Correlations between Ada’s cancer frequency in Germany and the German cancer registry incidence data were statistically significant.
- Between 23% and 94% of Ada assessments where cancer was suggested reported no advanced signs, and between 19% and 41% reported first signs and no advanced signs.
AI in Cancer Diagnostics: from research to clinical practice, European Association for Cancer Research (EACR), 2021 - Novel Methods in the Surveillance of Influenza-Like Illness in Germany Using Data From a Symptom Assessment App (Ada): Observational Case Study
- Ada uncovered similar ILI trends to an official German surveillance system and could help identify health trends in countries without population-based monitoring systems.
JMIR Public Health Surveillance, 2021 - The Potential of Digital Symptom-based Screening to Reduce the Transmission of SARS-CoV-2: A Modelling Study
- Digital symptom-based screening tools can substantially reduce the transmission of COVID-19.
- Applications like Ada (combined with testing and self-isolation) could prevent an additional 8% of the population being infected.
Pending publication, JMIR, 2021 - Syndromic surveillance insights from a symptom assessment app before and during COVID-19 measures in Germany and the United Kingdom: Results from repeated cross-sectional analyses
- A review of data from 950,000 users found that Ada could help uncover the impact of public health policies like COVID-19 lockdown measures.
JMIR mHealth and uHealth, 2020
Commentary and industry
- Letter to the Editor on 'Comment on “Accuracy and usability of a diagnostic decision support system in the diagnosis of three representative rheumatic diseases: a randomized controlled trial among medical students'BMC Arthritis Research & Therapy, 2022
- Letter to the Editor on 'The quality of diagnosis and triage advice provided by free online symptom checkers and apps in Australia'
Medical Journal of Australia (MJA), 2021 - Algorithm change protocols in the regulation of adaptive machine learning-based medical devices
- Proposals by the EU and FDA have set the regulatory groundwork for algorithm change protocols. The FDA proposals are detailed and actionable but are yet to be finalized. The EU has not yet detailed its plan. The authors recommended the EU set out an approach that is coordinated, actionable, and consultative.
- The public, manufacturers, and medical community should be engaged to inform requirements of algorithm change, with a focus on post market surveillance, RWP measurement, clinical evaluation, and labeling.
JMIR Formative Research, 2021 - Letter to the editor on 'Do Dr. Google and Health Apps Have (Comparable) Side Effects? An Experimental Study'
Clinical Psychological Science, 2021 - Letter to the editor on 'Periodic Manual Algorithm Updates and Generalizability: A Developer’s Response. Comment on “Evaluation of Four Artificial Intelligence–Assisted Self-Diagnosis Apps on Three Diagnoses: Two-Year Follow-Up Study”'Journal of Medical Internet Research, 2021
- Rare diseases 2030: How augmented AI will support diagnosis and treatment of rare diseases in the future
- Ada could help detect genetic autoinflammatory disorders, including familial Mediterranean fever.
Annals of the Rheumatic Diseases, 2020 - Introduction of a pathophysiology-based diagnostic decision support system (DDSS) and its potential impact on the use of AI in healthcare
- Ada Health successfully demonstrates the feasibility of a pathophysiology-based DDSS with Medical Deep Reasoning (MDR).
- MDR extends current DDSS capabilities by utilizing pathophysiology to describe the dynamic components of disease pathogenesis to create an individualized, high-resolution model for personal healthcare.
- Facilitating the integration of this kind of health-related data will significantly advance personalized medicine.
MedInfo, World Congress of Medical and Health Informatics, 2019