U.S. Department of Veterans Affairs Medical Center, Houston, TX, and Baylor College of Medicine Revised Safer Diagnosis (Safer Dx) Instrument

January 26, 2022

Innovation
Contact Innovator

Summary

The Revised Safer Dx Instrument provides a standardized list of questions to help users retrospectively identify and assess the likelihood of a missed diagnosis in a healthcare episode. Results of the assessment are intended for use in system-level safety improvement efforts, clinician feedback, and patient safety research.

The instrument consists of a series of questions that address five aspects of the diagnostic process: (1) the patient-provider encounter (history, physical examination, ordering tests/referrals based on assessment); (2) performance and interpretation of diagnostic tests; (3) follow-up and tracking of diagnostic information over time; (4) subspecialty and referral-specific factors; and (5) patient-related factors.¹ To answer the questions, the evaluator collects data from comprehensive electronic health records including information on a patient’s medical history, examination information, diagnostic test interpretation, and follow-up testing and diagnostic assessment. If the assessment indicates there was a likely diagnostic error (defined as a missed opportunity in diagnosis), users have the option to complete an additional process breakdown assessment as a guide designed to help identify factors contributing to the potential missed opportunity.

The original tool, the Safer Dx Instrument, was validated in a primary care setting, and results were published in 2016. In this study, the instrument yielded overall accuracy of 84%.² A study published in 2017 on use of the tool in a pediatric intensive care unit found the tool had inter-rater agreement of 93.6% (k, 0.72).³ The project team made minor revisions to the original tool to address feedback from the pilot studies, as well as from several national experts. Since the release of the revised iteration of the tool (i.e., the Revised Safer Dx Instrument), use of condition-specific adaptations of the tool (e.g., Safer Stroke Dx) have found it to yield accurate results.^4,5

For the best results, the project team suggests having multiple reviewers complete the assessment and discuss findings. Additionally, sites that wish to implement the tool may benefit from an existing safety environment that is supportive, with elements such as a patient safety culture, existing safety programs, and adequate staffing resources to implement the tool, including a multidisciplinary team with a dedicated safety analyst.

Innovation Patient Safety Focus

The Revised Safer Dx Instrument is designed to help identify potential diagnostic errors and guide safety improvement efforts that prevent future errors. The instrument is intended to maximize objectivity to address the challenge of inter-rater reliability, which has been a concern with previous assessments of diagnostic error.⁶

Resources Used and Skills Needed

The use of this tool requires relevant medical knowledge about the diagnostic process. Access to comprehensive electronic health records (progress notes, tests, referrals) and a shared understanding of important key terms (e.g., diagnostic error; missed opportunity) is needed. The tool could be used by a single clinician or by a multidisciplinary team with clinical input, depending on the need, setting, and extent of analysis. In most analyses of diagnostic error, a multidisciplinary perspective is needed, including input from physicians, nurses, and other health professionals who care for patients, as well as safety analysts and healthcare quality and risk management professionals.

To effectively use the tool, the implementing teams need logistical support and resources, such as available clinical staff. A culture focused on patient safety helps to identify underlying factors that lead to missed diagnoses.

Other important qualities and strategies include the following:

A patient safety culture that focuses on systems issues versus blaming individuals
An evaluation approach that incorporates:
- Multiple case reviewers with opportunity for discussion to fine-tune findings
- Evaluation of the diagnostic process rather than the patient outcome
- Evaluation of context to explain different perspectives and account for evolution and uncertainty of the diagnostic process
- Analysis of diagnostic process breakdowns that can help prevent diagnostic error in the future

Use By Other Organizations

Use of the tool has primarily been for research purposes to measure the prevalence of missed diagnoses and help identify their causes. Examples of research uses include the following:

A study on the incidence and origins of diagnostic errors in primary care in the United Kingdom, which was published in September 2021, used an earlier version of the tool.⁷
University of California San Francisco used the tool to identify the prevalence of diagnostic error at a hospital.⁸
An Agency for Healthcare Research and Quality (AHRQ) funded study is looking at the incidence of diagnostic error in inpatient settings, examining factors leading to diagnostic error, and exploring machine learning to retrospectively identify patients who were likely to have had an error in their diagnosis.⁹

A few organizations have used the tool on an ongoing basis in clinical practice. For example, it is being used by Geisinger,¹⁰ and safety analysts are using the tool at the Children’s Hospital of Philadelphia.

Date First Implemented

2013

Problem Addressed

The innovation addresses diagnostic error (including missed diagnoses) by helping users identify missed opportunities in diagnosis in an episode of care, and, if missed opportunities are found, contributory factors. Through a standard and systematic process, the instrument is intended to help circumvent the typical challenges and nuances in identifying diagnostic errors.

Description of the Innovative Activity

The Safer Diagnosis (Safer Dx) Instrument is a structured data collection instrument designed to help accurately assess diagnostic errors after a specific episode of care. The tool helps reviewers think through five main aspects of the diagnostic process: (1) the patient-provider encounter (history, physical examination, ordering tests/referrals based on assessment); (2) performance and interpretation of diagnostic tests; (3) follow-up and tracking of diagnostic information over time; (4) subspecialty and referral-specific factors; and (5) patient-related factors. The ultimate determination on the presence or absence of error depends on reviewers’ overall judgment after considering these items and sometimes discussing their findings with other clinician-reviewers.

After the initial assessment, if there is reason to believe there was a diagnostic error in the care episode, reviewers have the option to conduct a process evaluation to determine the root cause of the error. Findings from the initial evaluation and process assessment can be used to inform safety improvement efforts and can provide data for clinician feedback.

The instrument is relatively low cost to implement (i.e., primarily requires staff hours) and is available to the public.

Context of the Innovation

Wrong or delayed diagnoses cause more serious harms to patients than any other type of medical error.¹¹ Frequently-cited estimates are that diagnostic errors affect at least 12 million Americans each year,⁴ and about half of those errors may be harmful.¹² Research indicates that communication breakdowns during the patient-provider encounter are a leading contributor to diagnostic error.¹³ Other potential factors may relate to the healthcare system and care team, the provider, and/or the patient. Diagnostic error has been measured using autopsies, case reviews, surveys, malpractice claims, and incident reporting. In general, the burden of diagnostic errors has only recently become more visible, and there is still substantial progress that needs to be made in measuring their prevalence and the efficacy of interventions to reduce them. Congress authorized $2 million in fiscal year 2019 for the Agency for Healthcare Research and Quality (AHRQ) to initiate a research agenda to understand and solve the problem of diagnostic errors. In 2019, AHRQ awarded four grants that will more precisely define the scope of diagnostic errors, including an award to the Safer Dx team for detecting and measuring diagnostic errors in emergency department settings.¹⁴

The original Safer Dx Instrument was intended for research on diagnostic errors in primary care. Feedback following use of the instrument in several settings led to revisions to the tool. This allowed broadening of the settings in which the tool could be used and easier data collection.

The research team is exploring how other healthcare organizations can learn from diagnostic errors and improve safety.¹⁵

Results

The initial version of the tool was validated in a pediatric intensive care unit (PICU) and a primary care setting.

For the PICU study, overall initial reviewer agreement was 93.6% (k, 0.72). Infections and neurologic conditions were the most missed diagnoses across all high-risk cohorts (16/26).¹⁶

To validate the original version of the tool in a primary care setting, the Safer Dx team tested the accuracy of the tool against a sample of patient records (n = 389), with and without previously identified diagnostic errors (n = 129 and n = 260, respectively), and they found the tool had an overall accuracy of 84%, sensitivity of 71%, specificity of 90%, negative predictive value of 86%, and positive predictive value of 78%.¹⁷

Responding to feedback and expert input, the Safer Dx team revised and tested various iterations of the tool. Updates included changing the response scales from six to seven points so that there would be a neutral option, removing negatively phrased wording, and adding one question regarding missed follow-up on diagnostic data.¹⁸There are no published results on validation of the revised tool. More work is needed to measure associations between use of the instrument and patient outcomes.

Planning and Development Process

To plan implementation of use of the instrument, sites should review the open access guidance provided on key terms, such as diagnostic errors, missed opportunities, diagnostic harm, and diagnostic processes, to create a shared understanding. The guidance also provides recommendations on how to use the tool, as well as clinical case examples. It is also crucial to assemble a team to conduct review processes and analyze the information being gathered in order to inform next steps and interventions.

Depending on whether the goal is to do more research or to improve practice, the implementing team should develop and implement a rollout plan and goals for the initiative. Healthcare organizations may find a review of selected high-risk records useful to inform improvement opportunities. They may use the instrument in their peer review process. Clinicians can also use this tool for self-reflection and as a guide to review their own records. Sites can leverage their existing safety and quality improvement infrastructure and personnel to conduct case reviews on potential events identified through usual safety measurement processes such as incident reporting.

Resources Used and Skills Needed

Other important qualities and strategies include the following:

A patient safety culture that focuses on systems issues versus blaming individuals
An evaluation approach that incorporates:
- Multiple case reviewers with opportunity for discussion to fine-tune findings
- Evaluation of the diagnostic process rather than the patient outcome
- Evaluation of context to explain different perspectives and account for evolution and uncertainty of the diagnostic process
- Analysis of diagnostic process breakdowns that can help prevent diagnostic error in the future

Funding Sources

AHRQ
U.S. Department of Veterans Affairs (VA) Health Services Research and Development Service
VA National Center for Patient Safety
Private donors
Foundations

Getting Started with This Innovation

As noted, the implementing team should develop and launch a rollout plan and goals for the initiative. The plan should include reviewing materials and selecting ways in which the tool will be used.¹⁹ The team can first pilot the use of the tool prior to wider use. Clinicians can also use this tool for reflection on their own records. Health systems can leverage their existing safety and quality improvement infrastructure and personnel to create analysis and improvement initiatives.

Sustaining This Innovation

Sustainability is more likely if the work can be embedded into an existing patient safety program with patient safety specialists or risk managers.

Additionally, sustaining the innovation requires the following:

Prioritizing diagnosis-related initiatives
Adequate training of reviewers
Commitment from the team and leadership
A robust infrastructure and analysis effort to support and make the most of the initiative

References/Related Articles

Diagnostic safety and quality. Agency for Healthcare Research and Quality. Accessed October 20, 2021. https://www.ahrq.gov/topics/diagnostic-safety-and-quality.html

Fletcher TL, Helm A, Vaghani V, Kunik ME, Stanley MA, Singh H. Identifying psychiatric diagnostic errors with the Safer Dx Instrument. Int J Quality Health Care. 2020;32(6):405-411. doi:10.1093/intqhc/mzaa066

Grants to enable diagnostic excellence. Agency for Healthcare Research and Quality. November 2019. Updated January 2021. Accessed October 20, 2021. https://www.ahrq.gov/patient-safety/diagnostic-error-grants/index.html

Meyer AND, Upadhyay DK, Collins CA, et al. A program to provide clinicians with feedback on their diagnostic performance in a learning health system. Joint Commission J Quality Patient Safety. 2020;47(2):P120-P126. doi:10.1016/j.jcjq.2020.08.014

Safer Dx. Baylor College of Medicine, Department of Medicine. Accessed October 20, 2021. https://www.bcm.edu/departments/medicine/sections-and-divisions/health-services-research/research/safer-dx

Singh H, Bradford A, Goeschel C. Operational Measurement of Diagnostic Safety: State of the Science. Agency for Healthcare Research and Quality; 2020. Publication: 20-0040-1-EF. Accessed October 20, 2021. https://www.ahrq.gov/patient-safety/reports/issue-briefs/state-of-science.html

Singh H, Khanna A, Spitzmueller C, Meyer AND. Recommendations for using the Revised Safer Dx Instrument to help measure and improve diagnostic safety. Diagnosis (Berl). 2019;6(4):315-323. PMID: 31287795. doi:10.1515/dx-2019-0012

Singh H, Sittig DF. Advancing the science of measurement of diagnostic errors in healthcare: the Safer Dx framework. BMJ Qual Saf. 2015;24(2):103-110. Epub 2015 Jan 14. PMID: 25589094; PMCID: PMC4316850. doi:10.1136/bmjqs-2014-003675

Sittig DF, Ash JS, Singh H. The SAFER guides: empowering organizations to improve the safety and effectiveness of electronic health records. Am J Manag Care. 2014;20(5):418-423. PMID: 25181570.

Zwaan L, Staal J. Evidence on Use of Clinical Reasoning Checklists for Diagnostic Error Reduction. Agency for Healthcare Research and Quality; 2020. Publication: 20-0040-3-EF.
https://www.ahrq.gov/patient-safety/reports/issue-briefs/dxchecklists.html

Footnotes

Singh H, Khanna A, Spitzmueller C, Meyer AND. Recommendations for using the Revised Safer Dx Instrument to help measure and improve diagnostic safety. Diagnosis (Berl). 2019;6(4):315-323. PMID: 31287795. doi:10.1515/dx-2019-0012
Al-Mutairi A, Meyer AN, Thomas EJ, et al. Accuracy of the Safer Dx Instrument to identify diagnostic errors in primary care. J Gen Intern Med. 2016;31(6):602-608. doi:10.1007/s11606-016-3601-x
Davalos MC, Samuels K, Meyer AN, et al. Finding diagnostic errors in children admitted to the PICU. Pediatr Crit Care Med. 2017;18(3):265-271. doi:10.1097/PCC.0000000000001059
Fletcher TL, Helm A, Vaghani V, Kunik ME, Stanley MA, Singh H. Identifying psychiatric diagnostic errors with the Safer Dx Instrument. Int J Quality Health Care. 2020;32(6):405-411. doi:10.1093/intqhc/mzaa066
Saleh Velez FG, Alvarado-Dyer R, Pinto CB, et al. Safer Stroke-Dx Instrument: identifying stroke misdiagnosis in the emergency department. Circ Cardiovasc Qual Outcomes. 2021;14(7):e007758. doi:10.1161/CIRCOUTCOMES.120.007758
Singh H, Khanna A, Spitzmueller C, Meyer AND. Recommendations for using the Revised Safer Dx Instrument to help measure and improve diagnostic safety. Diagnosis (Berl). 2019;6(4):315-323. PMID: 31287795. doi:10.1515/dx-2019-0012
Cheraghi-Sohi S, Holland F, Singh H, et al. Incidence, origins and avoidable harm of missed opportunities in diagnosis: longitudinal patient record review in 21 English general practices. BMJ Qual Saf. 2021;30(12):977-985. Epub ahead of print. PMID: 34127547. doi:10.1136/bmjqs-2020-012594
Raffel KE, Kantor MA, Barish P, et al. Prevalence and characterisation of diagnostic error among 7-day all-cause hospital medicine readmissions: a retrospective cohort study. BMJ Qual Saf. 2020;29(12):971-979. doi:10.1136/bmjqs-2020-010896
Grants to enable diagnostic excellence. Agency for Healthcare Research and Quality. November 2019. Updated January 2021. Accessed November 30, 2021. https://www.ahrq.gov/patient-safety/diagnostic-error-grants/index.html
A learning health system approach to improve diagnosis. Geisinger. Accessed October 20, 2021. https://www.geisinger.edu/research/research-at-geisinger/learning-healthcare-system/safer-dx
Saber Tehrani AS, Lee H, Mathews SC, et al. 25-year summary of US malpractice claims for diagnostic errors 1986-2010: an analysis from the National Practitioner Data Bank. BMJ Qual Saf. 2013;22(8):672-680. doi:10.1136/bmjqs-2012-001550
Singh H, Meyer AN, Thomas EJ. The frequency of diagnostic errors in outpatient care: estimations from three large observational studies involving US adult populations. BMJ Qual Saf. 2014;23(9):727-731. Epub 2014 Apr 17. PMID: 24742777; PMCID: PMC4145460. doi:10.1136/bmjqs-2013-002627
Toolkit for Engaging Patients To Improve Diagnostic Safety . Content last reviewed November 2021. Agency for Healthcare Research and Quality, Rockville, MD. https://www.ahrq.gov/patient-safety/resources/diagnostic-safety/toolkit.html
Grants to enable diagnostic excellence. Agency for Healthcare Research and Quality. November 2019. Updated January 2021. Accessed November 30, 2021. https://www.ahrq.gov/patient-safety/diagnostic-error-grants/index.html
Singh H, Bradford A, Goeschel C. Operational measurement of diagnostic safety: state of the science. Diagnosis (Berl). 2020;8(1):51-65. Published 2020 Jul 24. doi:10.1515/dx-2020-0045
Davalos MC, Samuels K, Meyer AN, et al. Finding diagnostic errors in children admitted to the PICU. Pediatr Crit Care Med. 2017;18(3):265-271. doi:10.1097/PCC.0000000000001059
Al-Mutairi A, Meyer AN, Thomas EJ, et al. Accuracy of the Safer Dx Instrument to identify diagnostic errors in primary care. J Gen Intern Med. 2016;31(6):602-608. Epub 2016 Feb 22. PMID: 26902245; PMCID: PMC4870415. doi:10.1007/s11606-016-3601-x
Singh H, Khanna A, Spitzmueller C, Meyer AND. Recommendations for using the Revised Safer Dx Instrument to help measure and improve diagnostic safety. Diagnosis (Berl). 2019;6(4):315-323. PMID: 31287795. doi:10.1515/dx-2019-0012
Singh H, Khanna A, Spitzmueller C, Meyer AND. Recommendations for using the Revised Safer Dx Instrument to help measure and improve diagnostic safety. Diagnosis (Berl). 2019;6(4):315-323. PMID: 31287795. doi:10.1515/dx-2019-0012

Date Verified by Innovator

Date Verified by Innovator indicates the most recent date the innovator provided feedback during the review process.

January 18, 2022

FYI: You may notice that PSNet Innovations Exchange has recently been updated (June 2022) to remove the evidence rating section. For more information or questions, please email psnetsupport@ahrq.hhs.gov.

The inclusion of an innovation in PSNet does not constitute or imply an endorsement by the U.S. Department of Health and Human Services, the Agency for Healthcare Research and Quality, or of the submitter or developer of the innovation.