Cases & Commentaries

Getting a Good Report Card: Unintended Consequences of the Public Reporting of Hospital Quality

Spotlight Case
Commentary By Peter Lindenauer, MD, MSc

Case Objectives

  • Understand the rationale for public
    reporting of hospital quality.
  • List the current measures set being used
    nationally.
  • Describe unintended consequence of
    public reporting and interventions to prevent these.

The Case

A 55-year-old woman with end stage renal
disease requiring hemodialysis and coronary artery disease (status
post-coronary artery bypass grafting and placement of a St. Jude
prosthetic valve) was admitted to the medical service with
palpitations and chest pain. She reported missing her scheduled
hemodialysis session, and her symptoms resolved with prompt
inpatient hemodialysis. In the course of her work-up, it was noted
that she was subtherapeutic on her anticoagulation with warfarin,
and a heparin drip was initiated with the plan to bridge her until
her INR was in the therapeutic range.

Mostly in response to increasing pressure from
the hospital's administration to improve compliance with publicly
reported quality measures, the attending physician recommended
pneumococcal vaccination, and it was administered. Later that day,
the patient complained of pain over her right upper arm. The
attending told the patient that this was a common complaint after
immunization and that it would resolve. The next day, the patient
reported that her pain was worse, and the team noted an
8-centimeter hematoma within the muscles of her upper arm. The
hematoma resolved spontaneously. There was no permanent
harm.

The Commentary

Since first proposed by pioneers like Ernest
Codman and Florence Nightingale, the public reporting of measures
of hospital quality has been viewed with skepticism and at times
outright scorn by members of the medical profession. Despite these
protests, there has been an explosion of public reporting over the
last few years, in America and around the world.

The public release of hospital quality measures
is typically undertaken with a number of explicit and implicit aims
in mind.(1)
Ideally, public reporting improves transparency and can empower
patients to make better choices about where to seek treatment. This
increases hospital accountability for quality of care and provides
a stimulus to improve for fear of losing market share to local
competitors. Public reporting also enables the stewards of the
public's health (including government, regulators, accreditors, and
payers) to track performance over time, and these data can be tied
to compensation through "pay-for-performance"
contracts.

Public reporting begins with the selection of
evidence-based processes or outcomes to be measured and the
identification of meaningful opportunities to improve practice.
There are many organizations currently involved in the design and
implementation of public reporting in the United States. The most
important of these is the Hospital Quality Alliance (HQA), a
national public-private collaboration established to encourage
hospitals to voluntarily collect and report hospital quality
performance information.(2) All
acute care hospitals in the United States have been invited to
participate in this initiative, and by linking participation in the
program to the annual Medicare payment update, the Centers for
Medicare and Medicaid Services has achieved participation rates of
greater than 98%. The HQA began in late 2003 with a limited set of
10 quality measures spread across three clinical conditions: heart
failure, acute myocardial infarction, and pneumonia. By 2006, the
program had expanded to include 21 measures (Table) related to the original three clinical areas,
with the addition of new measures focused on the quality of
surgical care.

Despite the natural appeal of public reporting,
relatively little is known about its actual impact on quality of
care or patient outcomes. After New York State began publicly
reporting hospital-specific mortality following coronary bypass
surgery, mortality rates were observed to fall, but the mechanism
through which this occurred continues to be debated.(3,4) Similarly, a report on the impact of the "Quality
Counts" program in Wisconsin found that hospitals involved in
public reporting of risk-adjusted outcomes engaged in more quality
improvement activities and were more likely to show improved
outcomes than control institutions.(5)
Although these limited data support one of the premises of
transparency—that they will catalyze hospitals' quality
improvement activities—the jury is still out about the impact
of public reporting on patient decisions regarding where to obtain
medical care. What little is known suggests that hospital market
share has been largely unaffected by these initiatives.(6)

Concerns over Public Reporting

Concerns expressed about public reporting can be
grouped into two major categories—those that question the
validity of the undertaking itself and those that raise questions
about its unintended consequences.(7-11)
Among the first group, it is often pointed out that the field of
quality measurement in health care is still in its infancy and
there are as of yet only a small number of measures based on sound
evidence. Further, measures that have been validated for one
purpose are sometimes used inappropriately for other purposes. For
example, patient safety indicators derived from administrative data
sources can be valuable tools for case identification and the
monitoring of rates at a single organization but should not be used
to compare rates across hospitals. For example, Romano and
colleagues reported that, when derived from administrative data
sources, more than half of the observed variation in risk-adjusted
rates of postoperative infections seen across hospitals could be
attributed to differences in coding practices and not actual
outcomes.(12)
Nonetheless, some health plans have begun to grant subscribers
access to data that compare hospitals in a region.

Additionally, attempts to compare outcomes across
institutions are hampered by both real and perceived challenges of
carrying out risk adjustment and by limited statistical power when
studying rare events. While this has led to a gradual shift toward
the use of process measures (what was done) instead of outcomes
(what happened to the patient), improving the performance of
process measures may fail to result in actual improvements in the
outcomes of care. This can occur when the association between the
process measure and the desired outcome is weak, or when there is
improvement in the documentation of the process but not in the
actual process itself. This latter example may lead hospitals to
believe that they have improved care when in fact they have simply
improved documentation.

The second concern expressed about the public
reporting of quality data relates to the phenomenon of unintended
consequences. One framework for classifying unintended consequences
distinguishes those that result in direct harm from those that
cause harm indirectly. There are several types of events that can
lead to direct harm. The development of an allergic reaction
following medication administration in a patient not previously
known to have an allergy is one example of direct harm. On the
other hand, direct harm is more predictable when errors are made or
guidelines are violated. The administration of a beta blocker to a
patient with heart block and the provision of multiple influenza
vaccinations to a patient during a single hospitalization are both
examples of this type of unintended consequence. Finally, direct
harm can occur when caregivers are faced with diagnostic or
therapeutic uncertainty, and this risk can be exacerbated by the
incentives created by public reporting. For example, an initiative
designed to increase the use of venous thromboembolism prophylaxis
that did not adequately specify contraindications to the use of
heparins is likely to produce excessive rates of bleeding. In
another example, the current standard that antibiotics should be
given to patients with pneumonia within 4 hours of arrival in the
ED may result in antibiotics being given to some patients who prove
to have heart failure or pulmonary embolism.(11)

While less apparent, the problem of indirect harm
may be just as great as or greater than the examples described
above. First, patients may have important health care matters
neglected when physicians or nurses shift their attention to those
aspects of care for which they are being asked to report publicly.
Consider the pyrrhic victory of achieving optimal glycemic control
during the hospitalization of a diabetic patient while failing to
intervene to correct untreated hyperlipidemia. Furthermore,
hospitals that "play to the test" by reallocating valuable human
and other resources to excel in public measures may overlook more
pressing opportunities to improve the care of patients with
conditions not subject to public reporting. For example, hospitals
may decide to hire additional clinical personnel to improve
performance from 98% to 100% on a measure such as the use of
aspirin in myocardial infarction while the quality of care for
patients with stroke or COPD is neglected. Another concern that has
been raised about public reporting, especially when outcomes such
as mortality or complications are being compared, is the risk that
hospitals will turn away high-risk patients who might tarnish their
scores. Finally, patients and payers may misinterpret quality
performance information and make poor choices about where to seek
care or direct patients. This is a significant risk given the
limited number of clinical areas for which public reporting has
been implemented and in light of the fact that strong performance
in one area does not predict such performance in other clinical
areas.(13)

Reducing the Burden of Unintended
Consequences

Given the many ways in which quality improvement
and public reporting can result in unintended consequences, no
single approach can be relied upon to prevent all problems. First
and foremost, hospitals must recognize that unintended consequences
are bound to occur following any effort to implement change.
Failure Mode Effects
Analysis (FMEA) is a systematic, proactive method for
evaluating a process to identify where and how it might fail, and
to assess the relative impact of different failures. In other
words, FMEA specifically concerns itself with attempting to
anticipate unintended consequences and should be incorporated into
the planning for any major quality improvement project.

Education is the foundation of good practice and
remains a necessary though not sufficient way to prevent errors.
Education must be supplemented with checklists, protocols, and
other reminder systems, and whenever possible, computers can and
should be employed to minimize the need to rely on human vigilance.
For example, electronic order entry systems can and are being used
to prevent the administration of medications to patients with
specific contraindications. Forcing
functions are another approach that can be used to reduce the
risk of unintended consequences. For example, a
medication-dispensing and administration system can be designed to
require the nurse to document a heart rate before releasing a beta
blocker for administration. Monitoring is an important and probably
underutilized approach to addressing the problem of unintended
consequences; underutilized because unintended consequences are not
always incorporated into the quality improvement planning process
and because quality improvement staffs are stretched thin complying
with current reporting requirements. However, the automated
detection of adverse events was described more than 15 years ago
and can dramatically reduce the burden of data
collection.(14)
For example, automated detection can be used to measure how often
patients with heart failure receive treatment with antibiotics
intended for treatment of pneumonia, or how frequently patient on
narcotics are administered opiate antagonists like naloxone to
reverse oversedation or respiratory depression.

Payers also have an important role to play in
ensuring that unintended consequences do not diminish the benefits
gained through public reporting. First, they must set and
communicate realistic goals. For many measures, a goal of 100%
performance is not possible without resulting in an unacceptable
rate of unintended effects. For example, it is unlikely that
hospitals can deliver antibiotics to 100% of patients with
pneumonia within 4 hours without exposing many other patients to
antibiotics unnecessarily. By instituting a broad range of
measures, by rotating them frequently, and by not announcing
beforehand which will be made public, payers may also be able to
prevent hospitals from gaming the system. Finally, where
appropriate, payers should consider incorporating predictable
unintended events into the standard public reporting measure
sets.

The Case Revisited

How can this particular case be viewed in light
of the previously outlined framework? First, the current
immunization measures apply only to patients with pneumonia, so
while this patient is likely to have benefited from immunization,
there was in fact no such reporting requirement, and the attending
physician was not correct in his or her belief that failure to
immunize would have affected the hospital's performance ratings.
Second, because anticoagulation is not considered a
contraindication to the administration of immunization (nor to
intramuscular injection more generally), this is probably best
viewed as a nonpreventable case of direct harm that occurred while
following clinical guidelines. Although hematoma formation and
frank bleeding are recognized risks following both subcutaneous and
intramuscular injections, the risk does not appear to be
significantly increased among patients on anticoagulants, and thus
the benefits of immunization outweighed the risk of
complication.(15-18)

Certainly, had the patient's partial
thromboplastin time (PTT) been extremely elevated, it would have
been prudent to wait until it had fallen back to a therapeutic
range before immunizing. More importantly, if there was reasonable
doubt on the part of the team responsible for this patient's care
that administration of the immunization was either relatively or
absolutely contraindicated, then professionalism dictates that they
set aside the hospital's interests in favor of the patient's. This
is especially important to appreciate given that specifications for
existing quality measures do not enumerate every possible scenario
that might make immunization contraindicated, as this is neither
possible nor practical from the standpoint of chart review. This is
another example of why performance rates of 100% are not always
possible or desirable. Finally, regulatory agencies must be willing
to modify problematic measures. For example, if a sufficient number
of hospitals reported that patients on heparins experienced
hematomas following immunization, then the measure ought to be
revised to reflect this.

Peter Lindenauer, MD, MSc
Medical Director, Clinical and Quality Informatics; Baystate
Health
Assistant Professor of Medicine; Tufts University School of
Medicine

Faculty Disclosure: Dr. Lindenauer has
declared that neither he, nor any immediate member of his family,
has a financial arrangement or other relationship with the
manufacturers of any commercial products discussed in this
continuing medical education activity. In addition, the commentary
does not include information regarding investigational or off-label
use of pharmaceutical products or medical devices.

References

1. Marshall MN, Shekelle PG, Leatherman S, Brook
RH. The public release of performance data: what do we expect to
gain? A review of the evidence. JAMA. 2000;283:1866-1874.

[go to PubMed]

2. Hospital Quality Alliance Web site. Available
at: http://www.aha.org/aha/key_issues/qualityalliance/index.html.
Accessed November 1, 2006.

3. Omoigui NA, Miller DP, Brown KJ, et al.
Outmigration for coronary bypass surgery in an era of public
dissemination of clinical outcomes. Circulation. 1996;93:27-33.

[go to PubMed]

4. Burack JH, Impellizzeri P, Homel P, Cunningham
JN Jr. Public reporting of surgical mortality: a survey of New York
State cardiothoracic surgeons. Ann Thorac Surg. 1999;68:1195-1200;
discussion 1201-1102.
[go to PubMed]

5. Hibbard JH, Stockard J, Tusler M. Hospital
performance reports: impact on quality, market share, and
reputation. Health Aff (Millwood). 2005;24:1150-1160.
[go to PubMed]

6. Jha AK, Epstein AM. The predictive accuracy of
the New York State coronary artery bypass surgery report-card
system. Health Aff (Millwood). 2006;25:844-855.
[go to PubMed]

7. Eddy DM. Performance measurement: problems and
solutions. Health Aff (Millwood). 1998;17:7-25.
[go to PubMed]

8. Epstein AM. Rolling down the runway: the
challenges ahead for quality report cards. JAMA.
1998;279:1691-1696.
[go to PubMed]

9. Casalino LP. The unintended consequences of
measuring quality on the quality of medical care. N Engl J Med.
1999;341:1147-1150.
[go to PubMed]

10. Werner RM, Asch DA. The unintended
consequences of publicly reporting quality information. JAMA.
2005;293:1239-1244.
[go to PubMed]

11. Wachter RM. Expected and unanticipated
consequences of the quality and information technology revolutions.
JAMA. 2006;295:2780-2783.
[go to PubMed]

12. Romano PS, Chan BK, Schembri ME, Rainwater
JA. Can administrative data be used to compare postoperative
complication rates across hospitals? Med Care. 2002;40:856-867.

[go to PubMed]

13. Jha AK, Li Z, Orav EJ, Epstein AM. Care in
U.S. hospitals—the Hospital Quality Alliance program. N Engl
J Med. 2005;353:265-274.
[go to PubMed]

14. Classen DC, Pestotnik SL, Evans RS, Burke JP.
Computerized surveillance of adverse drug events in hospital
patients. JAMA. 1991;266:2847-2851. Erratum in: JAMA.
1992;267:1922.
[go to PubMed]

15. Pneumovax 23 [package insert]. Whitehouse
Station, NJ: Merck & Co, Inc; 1986.

16. Raj G, Kumar R, McKinney WP. Safety of
intramuscular influenza immunization among patients receiving
long-term warfarin anticoagulation therapy. Arch Intern Med.
1995;155:1529-1531.
[go to PubMed]

17. Delafuente JC, Davis JA, Meuleman JR, Jones
RA. Influenza vaccination and warfarin anticoagulation: a
comparison of subcutaneous and intramuscular routes of
administration in elderly men. Pharmacotherapy. 1998;18:631-636.

[go to PubMed]

18. Treadwell T. Intramuscular injection site
injuries masquerading as pressure ulcers. Wounds.
2003;15:302-312.

Table

Hospital Quality Alliance: Current Measure
Set

Heart Attack (Acute Myocardial Infarction or AMI)

"          
Aspirin at arrival

"          
Aspirin at discharge

"          
ACE Inhibitor or ARB for Left Ventricular Systolic Dysfunction*

"          
Beta Blocker at arrival

"          
Beta Blocker at discharge

"          
Thrombolytic agent received within 30 minutes of hospital arrival

"          
Percutaneous Coronary Intervention (PCI) received within 120
minutes of hospital arrival

"          
Smoking cessation advice/counseling

 

Heart Failure

"          
Assessment of Left Ventricular Function

"          
ACE Inhibitor or ARB for Left Ventricular Systolic Dysfunction*

"          
Discharge Instructions

"          
Smoking cessation advice/counseling

 

Pneumonia

"          
Oxygenation Assessment

"          
Initial Antibiotic Timing

"          
Pneumococcal Vaccination

"          
Influenza vaccination

"          
Blood culture performed prior to first antibiotic received in
hospital

"          
Smoking cessation advice/counseling

"          
Appropriate Initial Antibiotic Selection

 

Surgical Infection Prevention

"          
Prophylactic Antibiotic Received Within 1 Hour Prior to Surgical
Incision

"          
Prophylactic Antibiotics Discontinued Within 24 Hours After Surgery
End

 

Available
at http://www.hospitalcompare.hhs.gov/Hospital/Static/Data-Professionals.asp?dest=NAV|Home|DataDetails|ProfessionalInfo#TabTop

-->

Heart Attack (Acute Myocardial Infarction or AMI)

  • Aspirin at arrival
  • Aspirin at discharge
  • ACE Inhibitor or ARB for Left Ventricular Systolic
    Dysfunction*
  • Beta Blocker at arrival
  • Beta Blocker at discharge
  • Thrombolytic agent received within 30 minutes of hospital
    arrival
  • Percutaneous Coronary Intervention (PCI) received within 120
    minutes of hospital arrival
  • Smoking cessation advice/counseling

Heart Failure

  • Assessment of Left Ventricular Function
  • ACE Inhibitor or ARB for Left Ventricular Systolic
    Dysfunction*
  • Discharge Instructions
  • Smoking cessation advice/counseling

Pneumonia

  • Oxygenation Assessment
  • Initial Antibiotic Timing
  • Pneumococcal Vaccination
  • Influenza vaccination
  • Blood culture performed prior to first antibiotic received in
    hospital
  • Smoking cessation advice/counseling
  • Appropriate Initial Antibiotic Selection

Surgical Infection Prevention

  • Prophylactic Antibiotic Received Within 1 Hour Prior to
    Surgical Incision
  • Prophylactic Antibiotics Discontinued Within 24 Hours After
    Surgery End

Available at http://www.hospitalcompare.hhs.gov/Hospital/Static/Data-Professionals.asp?dest=NAV|Home|DataDetails|ProfessionalInfo#TabTop

Figure

Example Screenshot of HQA Data Available to
the Public on the Hospital Compare Web Site [http://www.hospitalcompare.hhs.gov/].


Click to enlarge