Comparative evaluation of LLMs in clinical oncology.

Nicholas R. Rydzewski; Deepak Dinakaran; Shuang G. Zhao; Eytan Ruppin; Baris Turkbey; Deborah E. Citrin; Krishnan R. Patel

Study

Comparative evaluation of LLMs in clinical oncology.

May 8, 2024

Rydzewski NR, Dinakaran D, Zhao SG, et al. NEJM AI. 2024;1(5):AIoa2300151.

View more articles from the same authors.

Large language models (LLM) are being developed to improve diagnostic accuracy. This study compared five LLMs on their accuracy of oncology diagnoses. Accuracy ranged from no better than random chance to similar to resident physicians. Notably, all models exhibited poor performance on women-predominant malignancies, suggesting a bias in training materials. This highlights the importance of partnerships between developers and medical professionals to co-develop reliable training sets.

Available at

Related Resources From the Same Author(s)

Families as partners in hospital error and adverse event surveillance.

March 8, 2017

Implementation of the I-PASS handoff program in diverse clinical environments: a multicenter prospective effectiveness implementation study.

November 16, 2022

Patient safety after implementation of a coproduced family centered communication programme: multicenter before and after intervention study.

December 19, 2018

American College of Surgeons' Committee on Trauma performance improvement and patient safety program: maximal impact in a mature trauma center.

December 14, 2011

Seroprevalence of SARS-CoV-2 among frontline health care personnel in a multistate hospital network--13 academic medical centers, April-June 2020.

September 23, 2020

Association of diagnostic stewardship for blood cultures in critically ill children with culture rates, antibiotic use, and patient outcomes: results of the Bright STAR Collaborative.

May 18, 2022

Changes in medical errors after implementation of a handoff program.

November 12, 2014

Doing right by our patients when things go wrong in the ambulatory setting.

February 12, 2014

Multistate point-prevalence survey of health care-associated infections.

April 9, 2014

Development of a pediatric adverse events terminology.

March 15, 2017

Surgeons' disclosures of clinical adverse events.

July 27, 2016

Validity of selected AHRQ Patient Safety Indicators based on VA National Surgical Quality Improvement program data.

October 15, 2008

Detection of potential look-alike/sound-alike medication errors using Veterans Affairs administrative databases.

November 28, 2018

Declines in hospitalizations for acute cardiovascular conditions during the COVID-19 pandemic: a multicenter tertiary care experience.

July 1, 2020

Common errors in computer electrocardiogram interpretation.

January 25, 2006

Effects of state opioid prescribing laws on use of opioid and other pain treatments among commercially insured U.S. adults.

March 30, 2022

Race, ethnicity, and 60-day outcomes after hospitalization with COVID-19.

November 17, 2021

It is time to define antimicrobial never events.

February 27, 2019

Surgical teams' attitudes about surgical safety and the surgical safety checklist at 10 years: a multinational survey.

November 17, 2021

Optimizing Pediatric Patient Safety in the Emergency Care Setting.

October 19, 2022

Impact of pharmacist previsit input to providers on chronic opioid prescribing safety.

March 14, 2018

Response of practicing chiropractors during the early phase of the COVID-19 pandemic: a descriptive report.

July 8, 2020

The effects of crew resource management on teamwork and safety climate at Veterans Health Administration facilities.

December 6, 2017

The impact of anti-infective drug shortages on hospitals in the United States: trends and causes.

February 8, 2012

Effect of health information exchange on recognition of medication discrepancies is interrupted when data charges are introduced: results of a cluster-randomized controlled trial.

December 6, 2017

STARD 2015 guidelines for reporting diagnostic accuracy studies: explanation and elaboration.

October 14, 2016

Strategies to prevent central line-associated bloodstream infections in acute-care hospitals: 2022 Update.

June 22, 2022

Development and testing of an objective structured clinical exam (OSCE) to assess socio-cultural dimensions of patient safety competency.

December 17, 2014

The frequency of intravenous medication administration errors related to smart infusion pumps: a multihospital observational study.

March 16, 2016

The relationship between organizational leadership for safety and learning from patient safety events.

April 14, 2010

Development of a measure of patient safety event learning responses.

September 23, 2009

Integrating patient safety into health professionals' curricula: a qualitative study of medical, nursing and pharmacy faculty perspectives.

February 19, 2014

'Not another safety culture survey': using the Canadian patient safety climate survey (Can-PSCS) to measure provider perceptions of PSC across health settings.

November 13, 2013

Improved prophylaxis and decreased rates of preventable harm with the use of a mandatory computerized clinical decision support tool for prophylaxis for venous thromboembolism in trauma.

February 20, 2013

Self-reported patient safety competence among new graduates in medicine, nursing and pharmacy.

January 9, 2013

The H-PEPSS: an instrument to measure health professionals' perceptions of patient safety competence at entry into practice.

June 13, 2012

A multi-hospital before–after observational study using a point-prevalence approach with an infusion safety intervention bundle to reduce intravenous medication administration errors.

June 27, 2018

Advancing measurement of patient safety culture.

October 15, 2008

Pharmacist medication assessments in a surgical preadmission clinic.

June 13, 2007

Where errors occur in the preparation and administration of intravenous medicines: a systematic review and Bayesian analysis.

January 27, 2010

Eliminating central line-associated bloodstream infections: a national patient safety imperative.

January 15, 2014

Declaring uncertainty: using quality improvement methods to change the conversation of diagnosis.

March 17, 2021

Making Health Care Safer II: An Updated Critical Analysis of the Evidence for Patient Safety Practices.

March 6, 2013

Changes in prevalence of health care-associated infections in U.S. hospitals.

November 14, 2018

Defining the critical role of nurses in diagnostic error prevention: a conceptual framework and a call to action.

December 6, 2017

Work system barriers and facilitators in inpatient care transitions of pediatric trauma patients.

March 11, 2020

Use and implementation of standard operating procedures and checklists in prehospital emergency medicine: a literature review.

April 5, 2017

Changes in hospital safety following penalties in the US Hospital Acquired Condition Reduction Program: retrospective cohort study.

July 24, 2019

Using the patient safety indicators to detect potential safety events among US veterans with psychotic disorders: clinical and research implications.

August 1, 2012

Safety climate associated with adverse events in nursing homes: a national VA study.

August 26, 2020

Organizational readiness to change as a leverage point for improving safety: a national nursing home survey.

September 8, 2021

Tracking rates of patient safety indicators over time: lessons from the Veterans Administration.

September 6, 2006

The 5th National Audit Project (NAP5) on accidental awareness during general anaesthesia: patient experiences, human factors, sedation, consent and medicolegal issues.

November 12, 2014

5th National Audit Project (NAP5) on accidental awareness during general anaesthesia: protocol, methods, and analysis of data.

September 24, 2014

Implementing a surgical checklist: more than checking a box.

November 7, 2012

Evaluating the Patient Safety Indicators: how well do they perform on Veterans Health Administration data?

September 7, 2005

Design and implementation of an application and associated services to support interdisciplinary medication reconciliation efforts at an integrated healthcare delivery network.

December 6, 2006

Enabling a learning healthcare system with automated computer protocols that produce replicable and personalized clinician actions.

August 4, 2021

Changes in safety attitude and relationship to decreased postoperative morbidity and mortality following implementation of a checklist-based surgical safety intervention.

February 9, 2011

Association of adverse effects of medical treatment with mortality in the United States: a secondary analysis of the Global Burden of Diseases, Injuries, and Risk Factors study.

January 30, 2019

Patient safety culture and the association with safe resident care in nursing homes.

April 18, 2012

Radiologists make more errors interpreting off-hours body CT studies during overnight assignments as compared with daytime assignments.

September 9, 2020

Association of hospital participation in a quality reporting program with surgical outcomes and expenditures for Medicare beneficiaries.

February 18, 2015

Effects of the introduction of the WHO "Surgical Safety Checklist" on in-hospital mortality: a cohort study.

January 30, 2005

Communication failures in the operating room: an observational classification of recurrent types and effects.

March 6, 2005

Using social and behavioural science to support COVID-19 pandemic response.

June 3, 2020

Establishing a simulation center for surgical skills: what to do and how to do it.

September 12, 2007

Identification of in-hospital complications from claims data. Is it valid?

March 6, 2005

The nature of reported safety events related to care coordination in the operating room setting in a tertiary academic center.

November 17, 2021

Physician behaviors associated with increased physician and nurse communication during bedside interdisciplinary rounds.

October 18, 2023

Patient falls in the operating room setting: an analysis of reported safety events.

May 11, 2022

Incidence and outcomes of non-ventilator-associated hospital-acquired pneumonia in 284 US hospitals using electronic surveillance criteria.

June 7, 2023

Medication errors: the impact of prescribing and transcribing errors on preventable harm in hospitalised patients.

March 4, 2009

Racial differences in antibiotic prescribing by primary care pediatricians.

March 20, 2013

Increased mortality associated with after-hours and weekend admission to the intensive care unit: a retrospective analysis.

April 20, 2011

Association of the Hospital Readmissions Reduction Program implementation with readmission and mortality outcomes in heart failure.

November 22, 2017

Prescription opioid dose reductions and potential adverse events: a multi-site observational cohort study in diverse US health systems.

November 29, 2023

A model for the departmental quality management infrastructure within an academic health system.

September 28, 2016

Experience with a trigger tool for identifying adverse drug events among older adults in ambulatory primary care.

June 17, 2009

Evaluation of the culture of safety: survey of clinicians and managers in an academic medical center.

March 6, 2005

Effect of Patient and Family Centered I-PASS on adverse event rates in hospitalized children with complex chronic conditions.

March 8, 2023

Creating a framework to integrate residency program and medical center approaches to quality improvement and patient safety training

January 13, 2021

Using smart IV infusion pumps outside of patient rooms.

February 2, 2022

Type 1 diabetes defined by severe insulin deficiency occurs after 30 years of age and is commonly treated as type 2 diabetes.

May 22, 2019

Assessing resident safety culture in nursing homes: using the nursing home survey on resident safety.

December 2, 2009

Safety culture of nursing homes: opinions of top managers.

April 6, 2011

Does clinical supervision of health professionals improve patient safety? A systematic review and meta-analysis.

June 29, 2016

Use of HIT for adverse event reporting in nursing homes: barriers and facilitators.

January 23, 2013

Measuring administrators' and direct care workers' perceptions of the safety culture in assisted living facilities.

August 1, 2012

Is hand hygiene before putting on nonsterile gloves in the intensive care unit a waste of health care worker time? A randomized controlled trial.

September 4, 2013

Venous thromboembolism after trauma: a never event?

October 10, 2012

Hand hygiene deficiency citations in nursing homes.

September 26, 2012

Relationship between nursing home safety culture and Joint Commission accreditation.

May 2, 2012

Nursing home patient safety culture perceptions among US and immigrant nurses.

January 24, 2018

Adverse events following an emergency department visit.

February 28, 2007

Nursing home administrators' opinions of the resident safety culture in nursing homes.

February 7, 2007

Nurse aides' ratings of the resident safety culture in nursing homes.

September 27, 2006

Association of polypharmacy and potential drug-drug interactions with adverse treatment outcomes in older adults with advanced cancer.

July 19, 2023

Anticoagulant medication errors in hospitals and primary care: a cross-sectional study.

November 14, 2018

Workplace factors associated with burnout of family physicians.

May 31, 2017

Related Resources

Addressing electronic health record contributions to diagnostic error.

May 15, 2024

The role for policy in AI-assisted medical diagnosis.

May 8, 2024

Oncologist perceptions of racial disparity, racial anxiety, and unconscious bias in clinical interactions, treatment, and outcomes.

April 24, 2024

Feeling safe in the context of digitalization in healthcare: a scoping review.

April 10, 2024

Patients' and doctors' views and experiences of the patient safety trajectory of breast cancer care.

March 27, 2024

Question answering systems for health professionals at the point of care - a systematic review.

March 20, 2024

Assessing diagnostic performance.

February 14, 2024

Assessing the impact of an electronic chemotherapy order verification checklist on pharmacist reported errors in oncology infusion centers of a health-system.

January 24, 2024

An analysis of medical malpractice claims against medical oncologists from a national database: implications for safer practice.

January 10, 2024

Uncovering the risks of anticancer therapy through incident report analysis using a newly developed medical oncology incident taxonomy.

December 13, 2023

"ChatGPT, can you help me save my child's life?" - Diagnostic accuracy and supportive capabilities to lay rescuers by ChatGPT in prehospital basic life support and paediatric advanced life support cases - an in-silico analysis.

December 6, 2023

The PRIDx framework to engage payers in reducing diagnostic errors in healthcare.

November 1, 2023

Assessing the utility of ChatGPT throughout the entire clinical workflow: development and usability study.

September 13, 2023

Pediatric Diagnostic Safety: State of the Science and Future Directions.

September 13, 2023

Final Report on Prioritization of Patient Safety Practices for a New Rapid Review or Rapid Response. Making Healthcare Safer IV Series.

August 9, 2023

Experimental evidence for structured information-sharing networks reducing medical errors.

August 9, 2023

Redesign of health care systems to reduce diagnostic errors: leveraging human experience and artificial intelligence.

August 9, 2023

Hospitalization due to adverse drug events in older adults with cancer: a retrospective analysis.

August 2, 2023

Burden of serious harms from diagnostic error in the USA.

July 28, 2023

Mitigating bias in AI at the point of care.

July 26, 2023

Association of polypharmacy and potential drug-drug interactions with adverse treatment outcomes in older adults with advanced cancer.

July 19, 2023

What causes delays in diagnosing blood cancers? A rapid review of the evidence.

July 5, 2023

Drugmakers are abandoning cheap generics, and now US cancer patients can’t get meds.

July 5, 2023

As cancer drug shortages grow, some doctors are forced to ration doses or delay care.

June 7, 2023

Optimizing measurement of misdiagnosis-related harms using symptom-disease pair analysis of diagnostic error (SPADE): comparison groups to maximize SPADE validity.

May 3, 2023

Annual Perspective

Improving Diagnostic Safety and Quality

April 26, 2023

Cancer drug shortages are creating dire circumstances for some patients.

April 26, 2023

STAMP: a 5-year project to reduce paediatric prescribing errors.

April 19, 2023

Using A.I. to detect breast cancer that doctors miss.

March 15, 2023

The impact of medication reconciliation and review in patients using oral chemotherapy.

March 1, 2023

See More About The Topic

Ambulatory Care

Medical Oncology

Pediatric Medical Oncology

Clinical Misdiagnosis

Chemotherapeutic Agents

Comparative evaluation of LLMs in clinical oncology.

Connect With Us

Sign up for Email Updates

Agency for Healthcare Research and Quality

Comparative evaluation of LLMs in clinical oncology.

Connect With Us

Sign up for Email Updates

Agency for Healthcare Research and Quality

Submit Your Innovations

Continue as a Guest

Continue Logged In

Submit Your Innovations

Continue as a Guest

Continue logged in

New Users to the PSNet site