Cases & Commentaries

Deaths Not Foretold: Are Unexpected Deaths Useful Patient Safety Signals?

Commentary By Kaveh G. Shojania, MD

The Case

An 87-year-old woman with hypercholesterolemia,
osteoporosis, and mild dementia presented to the emergency department after
a mechanical fall and was found to have a hip fracture. The patient had
well-treated hypertension, good exercise tolerance, and no known heart
disease. On physical examination, her vital signs were stable and she
exhibited no evidence of delirium. She had normal renal function, a mild
anemia, and an electrocardiogram (EKG) with evidence of old Q waves but no
acute or dynamic changes. The patient was admitted for orthopedic surgery,
and a formal preoperative assessment placed her at "low risk" (based on the
Revised Cardiac Risk Index scale). The patient remained on metoprolol and
simvastatin in addition to enoxaparin for deep venous thrombosis (DVT)
prophylaxis until her scheduled operation.

The patient had an uneventful preoperative period but
suffered a pulseless electrical activity arrest in the operating room at
the time of wound closure. Though she responded to resuscitative measures
and remained in the intensive care unit the following day, she developed
acute renal failure and shock liver. Based on previously expressed wishes,
the family requested that the patient be made comfortable with no further
interventions. At the family's request, no autopsy was performed; however,
providers felt that the patient suffered a massive pulmonary embolus based
on an intraoperative echocardiogram that suggested right ventricular
strain. The death was unexpected, particularly given the patient's low
preoperative risk, and the family and providers struggled to explain the
outcome.

The Commentary

This patient's death certainly warrants the label
"unexpected." In a prospective regional study of 580 patients, no patient
who received venous thromboembolism prophylaxis at the time of hip fracture
surgery (as this patient received) developed a fatal pulmonary
embolism.(1) Even
if the patient in this case did not die of pulmonary embolism, she would be
considered a "low-risk" surgical candidate based on the Revised Cardiac
Risk Index instrument, implying a risk for all major complications (not
just death) of less than 0.5%.(2) What then should we infer from her unfortunate and
unexpected death: chance misfortune or a problem with her care?

The human mind generally favors causal explanations over
chance, which would lead many to infer a problem in her care. Many
hospitals traditionally review all maternal deaths precisely for this
reason—with a maternal mortality rate in the United States of
approximately 1 in 10,000, any death is unexpected.(3) Though less emotionally charged,
many other Diagnostic Related Groups (DRGs [a method developed for Medicare
as part of the prospective payment system to classify hospitalized patients
into one of many "groups" that are expected to have similar hospital
resource use because the patients within each group have the same condition
or underwent the same procedure]) are also associated with low expected
risks of death. In this context, "death in low-risk DRGs" has been
developed as a potential indicator of patient safety problems. In fact, it
is one of 20 "high-level hospital indicators" on the Agency for Healthcare
Research and Quality (AHRQ) Patient Safety Indicators (Table 1).(4)

One of the benefits of using death in low-risk DRGs as a
patient safety indicator is that it can be measured using administrative
data routinely collected for billing purposes. This is a far more efficient
method of identifying potential safety problems than most other methods,
such as reviewing charts. In the major chart review–based patient
safety studies (5,6), nurses screened charts for "triggers" that prompted a more
detailed review by physicians. These triggers included death, unplanned
admission to intensive care units, and some 15 other potential indicators
of quality problems. Among hospitalized patients whose records contained
one of these triggers, approximately 20% experienced an adverse event.(7) Outside of research
settings, even this event rate of 20% is probably too low: most hospitals
cannot afford to have physicians review hundreds of charts just to have a 1
in 5 chance of finding potential safety problems. Hospitals need safety
indicators that identify cases that will prove on investigation to involve
true safety problems more often than not; they do not have the resources to
investigate high numbers of false-positive triggers.

The efficiency of the ascertainment method and the rate
of false positives represent just two characteristics to consider in
choosing patient safety indicators, whether it is for local or national
consumption. My colleague Dr. Alan Forster and I have considered the
features listed in Table 2 as a framework for evaluating the advantages and
disadvantages of specific strategies for detecting patient safety problems.
Using this framework, "death in low-mortality DRG" would score highly on
clinical importance and (in part for that reason) might also elicit
reasonable buy-in from clinicians. But it would score low on sensitivity,
as many important safety problems would go undetected, as they often do not
result in death in any DRG, let alone low-risk ones. On the other hand, one
would expect this indicator to do well in terms of having a low
false-positive rate—it seems reasonable to posit that a substantial
portion of unexpected deaths will involve deficiencies in care.

So, the key question becomes the extent to which deaths
in low-mortality DRGs truly reflect safety problems. The literature review
for the Patient Safety Indicators (4) described only one study (8) that quantified the prevalence of
problems with care among patients who died in low-mortality DRGs. Among
8000 randomly selected deaths from New York hospitals, patients who died in
low-mortality DRGs (defined as a risk of death less than 0.5%) were 5.2
times more likely than all other patients who died to have received "care
that departed from professionally recognized standards."(8) In absolute terms,
however, the rate of quality problems was only 9.8% among patients who died
in low-mortality DRGs. Thus, for every 10 deaths in low-mortality DRGs,
careful review may reveal no apparent problems in care in nine of them. In
my judgment, based on the information presented, I would characterize the
case presented here as falling in the latter category: the patient was
appropriately risk stratified prior to surgery, she received prophylaxis
for venous thromboembolism, and she underwent the planned procedure without
any technical complications.

While the researcher (and purist) in me wants to
emphasize this high false-positive rate associated with death in
low-mortality DRGs—it seems unfair to identify 10 cases as being
potentially due to substandard care when the evidence suggests that only
one of them is—the clinician in me acknowledges that we often
assiduously pursue diagnoses that have less than a 10% chance of being
present in a given patient. Reviewing deaths in low-mortality DRGs falls
within the norm of clinical practice insofar as it is a moderately
expensive diagnostic process that targets a relatively unlikely condition.
One can justify using an indicator like death in a low-mortality DRG as a
screening test, that, when positive, leads an institution to follow up
using a more accurate test, namely a thorough investigation of the case
that occurs very soon after the death. However, I do not think one can
justify reporting deaths in low-mortality DRGs as prima facie
evidence of quality problems, as currently happens with the Patient Safety
Indicators.

Ultimately, as a matter of local quality improvement and
assessment, I come down on the side of the clinical perspective. Just as I
routinely rule out conditions in my patients that are relatively unlikely
(but important to find), I regard it as part of my professional
responsibility to investigate potential problems with my patients' care, at
both the individual provider and system level, even if the probability is
relatively low. Using this logic, hospitals may successfully convince
clinicians to participate in internal peer review processes driven by the
detection of unexpected deaths, in the spirit of professionalism.

Hospital administrators must make it clear that
indicators such as deaths in low-mortality DRGs are screening tests to
prompt internal investigations for potential safety problems. The message
should not in any way be conveyed that these events represent prima
facie
evidence of deficient care. While I think this message can be
conveyed successfully within hospitals, the chance of success with the
public and the media is low. In my view, death in low-mortality DRGs
carries too high a potential for misunderstanding and stigma in the eyes of
the public as evidence of egregious care to justify its use as a publicly
reported indicator, especially given its relatively low yield.

Admittedly, and notwithstanding its low sensitivity and
high false-positive rate, it is still difficult to shake the intuition that
unexpected deaths such as the one that occurred in the present case
represent a "canary in the coal mine." Given sufficient time, the hospital
with multiple safety problems will likely have more deaths in low-mortality
DRGs than the safer hospital, just as the casino with its slight
statistical advantage will always eventually beat the average bettor. In
any given year, however, a perfectly safe hospital may have some deaths in
low-mortality DRGs and a terrible hospital may have none. Capturing even
fairly substantial differences in safety between hospitals would require
many years of mortality data.

Another concern is that culprit quality problems in cases
such as the present one may be undetectable through chart review. For
instance, a technical problem with surgery or failure to administer
thromboembolism prophylaxis might be undetected by chart review and might
even require interviews with the personnel involved. Since the time
required to obtain data for patient safety indicators is often many months,
such detailed investigations would be severely compromised.

The situation here is analogous to the perspective
advocated by the modern patient safety movement: incompetent individuals
may cause many adverse events, but most adverse events involve competent
providers (working within poorly designed systems). If we design incident
reporting and other patient safety systems to catch the small percentage of
poor performing providers, we will demoralize the majority of competent
providers who are trying to deliver good care. Similarly, if we design (and
then publicly report) patient safety indicators to catch outlier hospitals
with gross safety problems, we will demoralize personnel at the vast
majority of hospitals whose average rates of deaths in low-mortality DRGs
do not actually indicate significant problems with care. Forcing hospitals
to investigate publicly reported patient safety indicators with high
false-positives rates, rather than using such indicators on an internal
basis only, will alienate hospital personnel and hinder patient safety
efforts, much like the traditional "blame and shame" attitude within
hospitals has traditionally hindered progress in patient safety.

Take-Home Points

  • Patient safety indicators drawn from
    administrative datasets present the possibility of efficiently
    identifying cases with a high potential for involving problems with
    care.
  • Although death in low-mortality DRGs has
    high intuitive appeal as an indicator of safety problems, investigation
    will reveal no deficiencies in care in the majority of such
    deaths.
  • This high rate of false positives makes
    deaths in low-mortality DRGs problematic as a publicly reported patient
    safety indicator.
  • Nevertheless, the yield of cases with safety
    problems is probably high enough to warrant thorough internal hospital
    investigation of all such deaths.

Kaveh G. Shojania, MD
Canada Research Chair in Patient Safety and Quality Improvement
Associate Professor of Medicine
University of Ottawa

References

1. Todd CJ, Freeman CJ, Camilleri-Ferrante C, et al.
Differences in mortality after fracture of hip: the east Anglian audit.
BMJ. 1995;310:904-908. [go to PubMed]

2. Lee TH, Marcantonio ER, Mangione CM, et al. Derivation
and prospective validation of a simple index for prediction of cardiac risk
of major noncardiac surgery. Circulation 1999;100:1043-1049. [go to PubMed]

3. Berg CJ, Chang J, Callaghan WM, Whitehead SJ.
Pregnancy-related mortality in the United States, 1991-1997. Obstet
Gynecol. 2003;101:289-296. [go to PubMed]

4. McDonald KM, Romano PS, Geppert J, et al. Measures of
Patient Safety Based on Hospital Administrative Data—The Patient
Safety Indicators. Technical Review 5. AHRQ Publication No. 02-0038; August
2002. Available online at: http://www.ahrq.gov/downloads/pub/evidence/pdf/psi/psi.pdf.
Accessed November 27, 2007.

5. Brennan TA, Leape LL, Laird NM, et al. Incidence of
adverse events and negligence in hospitalized patients. Results of the
Harvard Medical Practice Study I. N Engl J Med. 1991;324:370-376. [go to PubMed]

6. Thomas EJ, Studdert DM, Burstin HR, et al. Incidence
and types of adverse events and negligent care in Utah and Colorado. Med
Care. 2000;38:261-271. [go to PubMed]

7. Forster AJ, O'Rourke K, Shojania KG, van Walraven C.
Combining ratings from multiple physician reviewers helped to overcome the
uncertainty associated with adverse event classification. J Clin Epidemiol.
2007;60:892-901. [go to PubMed]

8. Hannan EL, Bernard HR, O'Donnell JF, Kilburn H, Jr. A
methodology for targeting hospital cases for quality of care record
reviews. Am J Public Health. 1989;79:430-436. [go to PubMed]

Tables

Table 1. Examples of AHRQ Patient Safety
Indicators*
Patient Safety Indicator Definition and Numerator Technical Specifications
Death in low-mortality
DRGs
Mortality rate for
admitting diagnoses or procedures with low risk for death
Patients with disposition
of "deceased" per 100 patients with discharges in DRGs associated
with mortality rate less than 0.5%, based on 1997 data from the
National Inpatient Sample

Excludes patients with any code for cancer, trauma, or an
immunocompromised state

Complications of
anesthesia
Complications attributable
to anesthesia
Discharges with ICD-9-CM
diagnosis codes for anesthesia complications in any secondary
diagnosis field per 100 surgical discharges

Excludes patients with any diagnosis code for drug dependence,
abuse of drugs, or self-inflicted injury

Decubitus ulcer Pressure sores in patients
hospitalized for 4 days or longer
Discharges with ICD-9-CM
code of 707.0 in any secondary diagnosis field per 100 discharges
for medical or surgical patients with a length of stay of at least
4 days

Excludes patients admitted from a long-term care facility and any
patient with a diagnosis of hemiplegia, paraplegia, or
quadriplegia

Failure to rescue Mortality rate among
patients who develop potentially life-threatening complications
while in hospital
All discharges with
disposition of "deceased" per 100 patients with discharge codes for
potential complications of care (e.g., cardiac arrest, blood clot,
gastrointestinal hemorrhage, renal failure)

Excludes patients transferred to hospital from another acute or
chronic care facility

Foreign body left in during
procedure
Surgical instrument,
sponge, or other device left inside patient after a procedure
Discharges with ICD-9-CM
codes for foreign body left in during procedure in any secondary
diagnosis field per 100 discharges
Iatrogenic
pneumothorax
Pneumothorax ("collapsed
lung") due to a procedure such as insertion of a central venous
catheter
Discharges with ICD-9-CM
code of 512.1 in any secondary diagnosis field per 100 medical and
surgical discharges
Postoperative hip
fracture
Fractured hip from falls
among hospitalized surgical patients
Discharges with ICD-9-CM
code for [hip fracture] in any secondary diagnosis among all
surgical discharges

Excludes patients who have principal diagnosis codes for trauma,
seizure, metastatic cancer, or other conditions that could cause
hip fracture

*A complete list of the AHRQ patient safety indicators,
their definitions, and technical specifications can be found in Appendix E
of reference 4.

Table 2. Important Characteristics of Strategies for
Detecting Patient Safety Problems
Sensitivity The patient safety
indicator or other detection strategy captures a substantial
proportion of the safety problems one would want to know
about.
Low false-positive
rate
The majority of cases
identified by the detection strategy turn out to involve true
safety problems.
Clinical importance The cases identified
involve safety problems that carry a high risk for substantial
morbidity, rather than relatively unimportant problems (e.g.,
missed doses of minor medications).
Conduciveness to
change
The detection method should
capture sufficient detail to understand the events and correct the
underlying problems.
Credibility and buy-in with
clinicians
The detection strategy has
credibility (e.g., hard clinical outcomes, not administrative data)
and does not place an inordinate burden of participation on
front-line clinicians (e.g., a system that requires nothing from
clinicians would score high on this dimension, whereas a process
that requires intensive chart reviews or incident reporting forms
that contain numerous questions to answer would score low).
Costs Expenditures to carry out
the detection strategy on a routine basis fall within the budget of
the average hospital.