Assessing the utility of ChatGPT throughout the entire clinical workflow: development and usability study.

Arya Rao; Michael Pang; John Kim; Meghana Kamineni; Winston Lie; Anoop K Prasad; Adam Landman; Keith J. Dreyer; Marc D Succi

Study

Assessing the utility of ChatGPT throughout the entire clinical workflow: development and usability study.

September 13, 2023

Rao A, Pang M, Kim J, et al. J Med Internet Res. 2023;25:e48659.

View more articles from the same authors.

Interest in testing ChatGPT as a clinical tool is increasing. This study asked ChatGPT to provide a differential diagnosis, diagnostic testing, final diagnosis, and care management for 36 previously published clinical vignettes. ChatGPT had an overall accuracy of 72%, with the highest level of accuracy at the final diagnosis stage (77%).

PubMed citation

Free full text

Related Resources From the Same Author(s)

Unscheduled radiologic examination orders in the electronic health record: a novel resource for targeting ambulatory diagnostic errors in radiology.

February 12, 2020

Society of Critical Care Medicine Guidelines on Recognizing and Responding to Clinical Deterioration Outside the ICU: 2023.

February 7, 2024

Errare humanum est: frequency of laterality errors in radiology reports.

June 10, 2009

Optimizing Pediatric Patient Safety in the Emergency Care Setting.

October 19, 2022

Significant reduction in preanalytical errors for nonphlebotomy blood draws after implementation of a novel integrated specimen collection module.

December 14, 2016

Families as partners in hospital error and adverse event surveillance.

March 8, 2017

Effect of restriction of the number of concurrently open records in an electronic health record on wrong-patient order errors: a randomized clinical trial.

May 29, 2019

Implementation of the I-PASS handoff program in diverse clinical environments: a multicenter prospective effectiveness implementation study.

November 16, 2022

Readiness of US general surgery residents for independent practice.

October 4, 2017

Liquid medication errors and dosing tools: a randomized controlled experiment.

October 5, 2016

Pictograms, units and dosing tools, and parent medication errors: a randomized study.

July 19, 2017

Liquid medication dosing errors by Hispanic parents: role of health literacy and English proficiency.

May 31, 2017

A randomized trial of nighttime physician staffing in an intensive care unit.

June 5, 2013

The impact of anti-infective drug shortages on hospitals in the United States: trends and causes.

February 8, 2012

Association of display of patient photographs in the electronic health record with wrong-patient order entry errors.

December 2, 2020

Using social and behavioural science to support COVID-19 pandemic response.

June 3, 2020

John M. Eisenberg Patient Safety Awards. Safety, effectiveness, and efficiency: a Web-based virtual anticoagulation clinic.

March 6, 2005

A multistep approach to improving biopsy site identification in dermatology: physician, staff, and patient roles based on a Delphi consensus.

March 26, 2014

Patient safety after implementation of a coproduced family centered communication programme: multicenter before and after intervention study.

December 19, 2018

Preventability and causes of readmissions in a national cohort of general medicine patients.

May 4, 2016

Guidelines for opioid prescribing in children and adolescents after surgery: an expert panel opinion.

December 2, 2020

The Lancet Commission on lessons for the future from the COVID-19 pandemic.

October 12, 2022

Clinical data sharing improves quality measurement and patient safety.

March 24, 2021

Potentially inappropriate opioid prescribing, overdose, and mortality in Massachusetts, 2011–2015.

October 3, 2018

Preventing harm in the ICU—building a culture of safety and engaging patients and families.

July 12, 2017

Increasing adoption of computerized provider order entry, and persistent regional disparities, in US emergency departments.

August 31, 2011

Incidence and OR team awareness of “near-miss” and retained surgical sharps: a national survey on United States operating rooms.

June 23, 2021

A new professionalism? Surgical residents, duty hours restrictions, and shift transitions.

November 17, 2010

Association of diagnostic stewardship for blood cultures in critically ill children with culture rates, antibiotic use, and patient outcomes: results of the Bright STAR Collaborative.

May 18, 2022

Structured override reasons for drug–drug interaction alerts in electronic health records.

May 15, 2019

Image Gently, Step Lightly: promoting radiation safety in pediatric interventional radiology.

November 17, 2010

Outpatient adverse drug events identified by screening electronic health records.

June 9, 2010

Evaluation and accurate diagnoses of pediatric diseases using artificial intelligence.

February 27, 2019

It is time to define antimicrobial never events.

February 27, 2019

Diagnostic accuracy of pediatric teledermatology using parent-submitted photographs: a randomized clinical trial.

November 29, 2017

February 26, 2020

Parents' medication administration errors: role of dosing instruments and health literacy.

February 10, 2010

Health apps and health policy: what is needed?

October 24, 2018

Analysis of errors in dictated clinical documents assisted by speech recognition software and professional transcriptionists.

July 25, 2018

Intraoperative adverse events in abdominal surgery: what happens in the operating room does not stay in the operating room.

November 23, 2016

Development of an emergency department trigger tool using a systematic search and modified Delphi process.

July 13, 2016

Potentially harmful medication dispenses after a fall or hip fracture: a mixed methods study of a commonly used quality measure.

March 9, 2022

Enabling a learning healthcare system with automated computer protocols that produce replicable and personalized clinician actions.

August 4, 2021

Changes in medical errors after implementation of a handoff program.

November 12, 2014

Effect of complementary interventions to redesign care on teamwork and quality for hospitalized medical patients: a pragmatic controlled trial.

November 15, 2023

Racial differences in antibiotic prescribing by primary care pediatricians.

March 20, 2013

Effect of work-hours regulations on intensive care unit mortality in United States teaching hospitals.

September 2, 2009

A new patient safety smartphone application for prevention of "forgotten" ureteral stents: results from a clinical pilot study in 194 patients.

July 26, 2017

Application of the aviation black box principle in pediatric cardiac surgery: tracking all failures in the pediatric cardiac operating room.

January 14, 2015

Beyond medication reconciliation: the correct medication list.

May 3, 2017

Outbreak investigation of COVID-19 among residents and staff of an independent and assisted living community for older adults in Seattle, Washington.

June 10, 2020

Preventable mortality after common urological surgery: failing to rescue?

September 3, 2014

Transforming the medication regimen review process using telemedicine to prevent adverse events.

December 16, 2020

Design and impact of a novel surgery-specific second victim peer support program.

January 29, 2020

The power of written word: reflection reduces errors of omission.

October 4, 2023

Seen through their eyes: residents' reflections on the cognitive and contextual components of diagnostic errors in medicine.

September 5, 2012

Contraindicated medication use in dialysis patients undergoing percutaneous coronary intervention.

December 16, 2009

Preventing home medication administration errors.

March 14, 2022

Physicians' perceptions, preparedness for reporting, and experiences related to impaired and incompetent colleagues.

July 28, 2010

Incidence and trends of sepsis in US hospitals using clinical vs claims data, 2009--2014.

September 27, 2017

Ambulatory computerized prescribing and preventable adverse drug events.

June 8, 2016

Effects on resident work hours, sleep duration and work experience in a Randomized Order Safety Trial Evaluating Resident-physician Schedules (ROSTERS).

June 26, 2019

Misdiagnosis, mistreatment, and harm - when medical care ignores social forces.

April 8, 2020

Teaching internal medicine residents quality improvement and patient safety: a lean thinking approach.

April 28, 2010

Procedural safety in emergency care: a conceptual model and recommendations.

October 24, 2012

EAU policy on live surgery events.

March 12, 2014

Access to prescription opioids—Primum Non Nocere: a teachable moment.

July 27, 2016

Safety climate and medical errors in 62 US emergency departments.

January 9, 2013

The safety of emergency care systems: results of a survey of clinicians in 65 US emergency departments.

March 4, 2009

The National Emergency Department Safety Study: study rationale and design.

January 9, 2008

Adverse drug events in surgical patients: an observational multicentre study.

November 6, 2013

Effectiveness of acute care remote triage systems: a systematic review.

February 5, 2020

Internally-developed online adverse drug reaction and medication error reporting systems.

June 7, 2006

Effect of clinical decision-support systems: a systematic review.

July 18, 2012

Pursuing professional accountability: an evidence-based approach to addressing residents with behavioral problems.

August 1, 2012

The nature of reported safety events related to care coordination in the operating room setting in a tertiary academic center.

November 17, 2021

The physiology of failure: identifying risk factors for mortality in emergency general surgery patients using a regional health system integrated electronic medical record.

September 7, 2022

Patient falls in the operating room setting: an analysis of reported safety events.

May 11, 2022

Improving patient safety through the involvement of patients: development and evaluation of novel interventions to engage patients in preventing patient safety incidents and protecting them against unintended harm.

November 16, 2016

Decreasing errors in pediatric continuous intravenous infusions.

May 24, 2006

Standardization as a mechanism to improve safety in health care.

March 6, 2005

The patient reporting and action for a safe environment (PRASE) intervention: a feasibility study.

December 21, 2016

Classifying errors in preventable and potentially preventable trauma deaths: a 9-year review using the Joint Commission's standardized methodology.

October 15, 2014

Automation failures and patient safety.

December 2, 2020

Simulation in the executive suite: lessons learned for building patient safety leadership.

January 6, 2016

A new paradigm for surgical procedural training.

January 18, 2012

Safety culture and complications after bariatric surgery.

November 14, 2012

Description and evaluation of an interprofessional patient safety course for health professions and related sciences students.

January 10, 2007

More than a tick box: medical checklist development, design, and use.

August 23, 2017

Assessment of physician sleep and wellness, burnout, and clinically significant medical errors.

January 13, 2021

The Diagnostic Error Index: a quality improvement initiative to identify and measure diagnostic errors.

February 10, 2021

Adverse drug event detection in pediatric oncology and hematology patients: using medication triggers to identify patient harm in a specialized pediatric patient population.

May 14, 2014

Evaluation of consistency in dosing directions and measuring devices for pediatric nonprescription liquid medications.

December 8, 2010

Variation in quality of urgent health care provided during commercial virtual visits.

April 20, 2016

Intraoperative sentinel events in the era of surgical safety checklists: results of a national survey.

March 3, 2021

Outcomes of missed diagnosis of pediatric appendicitis, new-onset diabetic ketoacidosis, and sepsis in five pediatric hospitals.

September 13, 2023

Board of pharmacy practices related to medication errors and their potential impact on patient safety.

August 1, 2018

Postoperative opioid prescribing and the pain scores on Hospital Consumer Assessment of Healthcare Providers and Systems survey.

June 7, 2017

Association of state opioid duration limits with postoperative opioid prescribing.

January 29, 2020

Return on investment for a computerized physician order entry system.

March 15, 2006

Related Resources

Diagnostic error in mental health: a review.

April 24, 2024

Unmasking bias in artificial intelligence: a systematic review of bias detection and mitigation strategies in electronic health record-based models.

April 17, 2024

Clinical reasoning of a generative artificial intelligence model compared with physicians.

April 17, 2024

Inattentional blindness in medicine.

April 17, 2024

Feeling safe in the context of digitalization in healthcare: a scoping review.

April 10, 2024

Patient Safety Research Summaries.

April 9, 2024

Question answering systems for health professionals at the point of care - a systematic review.

March 20, 2024

Machine learning in medication prescription: a systematic review.

February 21, 2024

Assessing diagnostic performance.

February 14, 2024

Current State of Diagnostic Safety: Implications for Research, Practice, and Policy.

February 7, 2024

Embracing the future-is artificial intelligence already better? A comparative study of artificial intelligence performance in diagnostic accuracy and decision-making.

February 7, 2024

Diagnostic accuracy of a large language model in pediatric case studies.

January 17, 2024

Diagnostic Quality Improvement Repository.

January 10, 2024

Error codes at autopsy to study potential biases in diagnostic error.

January 10, 2024

"ChatGPT, can you help me save my child's life?" - Diagnostic accuracy and supportive capabilities to lay rescuers by ChatGPT in prehospital basic life support and paediatric advanced life support cases - an in-silico analysis.

December 6, 2023

Advancing Diagnostic Excellence for Maternal Health Care: Proceedings of a Workshop–in Brief.

November 15, 2023

The PRIDx framework to engage payers in reducing diagnostic errors in healthcare.

November 1, 2023

Can AI help doctors come up with better diagnoses?

October 4, 2023

Preventing potential patient harm through clinical content interventions during oncology clinical trial implementation.

September 20, 2023

Understanding And Addressing Pre-Hospital Diagnostic Delays.

September 20, 2023

Patient and family contributions to improve the diagnostic process through the OurDX electronic health record tool: a mixed method analysis.

September 13, 2023

Pediatric Diagnostic Safety: State of the Science and Future Directions.

September 13, 2023

Diagnostic errors in uncommon conditions: a systematic review of case reports of diagnostic errors.

September 13, 2023

Reimagining Healthcare Teams: Leveraging the Patient-Clinician-AI Triad To Improve Diagnostic Safety.

August 16, 2023

Final Report on Prioritization of Patient Safety Practices for a New Rapid Review or Rapid Response. Making Healthcare Safer IV Series.

August 9, 2023

Experimental evidence for structured information-sharing networks reducing medical errors.

August 9, 2023

Redesign of health care systems to reduce diagnostic errors: leveraging human experience and artificial intelligence.

August 9, 2023

Mitigating bias in AI at the point of care.

July 26, 2023

Introducing second-year medical students to diagnostic reasoning concepts and skills via a virtual curriculum.

June 28, 2023

Context matters: toward a multilevel perspective on context in clinical reasoning and error.

June 21, 2023

See More About The Topic

Hospitals

Ambulatory Clinic or Office

Medicine

Clinical Misdiagnosis

Technologic Approaches

Assessing the utility of ChatGPT throughout the entire clinical workflow: development and usability study.

Connect With Us

Sign up for Email Updates

Agency for Healthcare Research and Quality

Assessing the utility of ChatGPT throughout the entire clinical workflow: development and usability study.

Connect With Us

Sign up for Email Updates

Agency for Healthcare Research and Quality

Submit Your Innovations

Continue as a Guest

Continue Logged In

Submit Your Innovations

Continue as a Guest

Continue logged in

New Users to the PSNet site