Hospital Compare - A quality tool provided by Medicare

Information for Professionals on Data Collection

The data presented on this website comes from hospitals that volunteered to submit their data for public reporting. The clinical measures reported here focus on heart attack, heart failure, pneumonia, asthma (children only) and surgical care improvement project. Each rate calculation reported is based upon the hospital's relevant discharges. Detailed information concerning measure specifications and calculation of the rates is available at www.QualityNet.org

This site displays data provided by acute care "general" hospitals, children's hospitals, and "critical access" hospitals, which are small, generally geographically remote rural hospitals.

Beginning with discharges in 2004, eligible acute care hospitals could elect to report quality data in order to receive the incentive payment established by Section 501(b) of the Medicare Prescription Drug, Improvement and Modernization Act of 2003 (MMA). To obtain increased payment, the provision required eligible hospitals to report on an initial set of 10 quality performance measures (the "starter set") and to agree to have their data publicly displayed. Initially, almost all hospitals eligible for the payment incentive provided data for the 10 "starter set" measures, reflecting care delivered during 2004. Under Section 5001(a) of the Deficit Reduction Act of 2005, the set of measures included in the incentive was expanded, the magnitude of the incentive was increased, and the time-limit for the provision removed. More information about these provisions is available at Reporting Hospital Quality Data for Annual Payment Update.

The majority of these acute care hospitals are voluntarily providing data on additional measures that have been identified by the Hospital Quality Alliance (HQA). The HQA measure set currently consists of 26 measures:

• the ten "starter set" measures first published in October 2003;
• seven new measures first published in April 2005;
• three new measures first published in September 2005; and
• one measure first published in December 2006
• one measure first published in June 2007
• two measures first published in December 2007
• two measures first published in July 2008

For a full listing of the measures see measures list.

Critical access hospitals, which also provide acute care, also participate in the HQA. The critical access hospitals do not receive any financial incentive to report since they are not eligible for the incentive payment established by Section 501(b) of the MMA or 5001(a) of the DRA. These hospitals may elect to submit data for any or all of the measures in the measure set, and may elect to report data but not have it displayed on the website. While critical access hospitals play an important role in care in rural areas, measure design may affect the number of eligible cases reported by these hospitals.

The data collection approach was primarily retrospective. Data sources for required data elements included administrative data and medical record documents. Some hospitals may prefer to gather data concurrently by identifying patients in the population of interest. This approach provides opportunities for improvement at the point of care/service. However, complete documentation includes the principal and other ICD-9-CM diagnosis and procedure codes, which require retrospective data entry.

Data Quality Assurance

The QIO Clinical Data Warehouse stores provider-reported performance measure data for quality improvement and public reporting. Hospitals participating in the HQA initiative provide their data to the QIO Clinical Data Warehouse. It is the intent of CMS that the QIO Clinical Data Warehouse will ultimately provide four (4) quarters of data, updated quarterly, for each participating hospital.

The CMS Abstraction and Reporting Tool (CART) is a medical record data abstraction tool that builds on over 10 years of measurement activity, and currently generates both CMS/The Joint Commission aligned measures as well as the ten core CMS/The Joint Commission aligned measures, with full alignment slated for future development. Hospitals may use the CART to transmit abstracted data directly into the QIO Clinical Data Warehouse through www.QualityNet.org (a HIPAA-compliant, secure data transmission vehicle) or they may instruct a vendor to submit the data on their behalf using QNET. CART may also be used to transmit data directly to an ORYX vendor from a current Joint Commission-accredited hospital. The vendors transmit the data to the QIO Clinical Warehouse, if the hospital has authorized them to do so, on their behalf. Under The Joint Commission program, organizations that wish to be certified as ORYX vendors must pass certain tests that verify their capacity to correctly handle hospital data and calculate performance rates using the prescribed algorithms.

Both ORYX Vendors and CART data submissions include auditing procedures and edit checks, which assess whether data submitted is consistent with defined parameters for sample size, outliers, and missing data. In addition, CMS intends to validate data submitted to the Warehouse for the HQA initiative. The validation process will provide assurance that the hospital, or its designated agent, can accurately abstract patient medical records and accurately submit data to the QIO Clinical Data Warehouse. The data for this posting has been audited/edited, but not validated.

Because the children’s asthma care measures are collected by The Joint Commission and are shared with CMS, questions about children’s asthma care data quality assurance should be directed to The Joint Commission.

Confidence Interval

The information provided on the site enables the user to calculate confidence intervals for each reported measure.

Confidence intervals can be used to estimate the precision of the calculated rates for an individual hospital. A confidence interval is the range of values, within which an estimated value or rate is likely to fall. A confidence interval is a statistical determination of the degree of certainty associated with an estimated value. As can be seen in the table of estimated values (below), large differences between individual hospitals’ rates may be significant, and small differences between hospitals are usually not significant.

The smaller the sample size, the greater the difference in rates must be order for that difference to be statistically meaningful. Also, as sample size varies between hospitals, it is difficult to precisely compare their rates, without considering the confidence intervals.

Over time, as the quality data base is expanded, a full four quarters of data will ultimately be posted for each measure, so the number of cases used to determine hospitals' rates will likely increase, thereby increasing the reliability and stability of the rates.

Technical Appendix

Individual Hospital Performance Rate Calculations

The performance rate is calculated by dividing the numerator by the denominator. The denominator is the sum of all eligible cases (as defined in the measure specifications) submitted to the QIO Clinical Data Warehouse for the reporting period, while the numerator is the sum of all eligible cases submitted for the same reporting period where the recommended care was provided. The same data will be used for individual hospital, state and national rate calculations.

National and State Averages - Calculation of the Comparison Rates

A two-step process was used to calculate the national and state comparison group rates for all but the children's asthma care measures. The national and state comparison rates for each measure were calculated using all of the data submitted to the QIO Clinical Data Warehouse for hospitals with at least one case that met the measure’s inclusion criteria (that is, for which the denominator was greater than zero).

First, the individual hospital performance rates were calculated using the method described above for all hospitals. Next, hospitals with "0 patients" were excluded from the calculation. For the determination of the 90th percentile (or top 10%) of hospitals on a national basis, the individual rates were then rank ordered and the top 10th percentile score identified. For the national and state averages, a simple average was constructed where the numerator was the sum of all non-excluded hospitals’ scores and the denominator was the total number of hospitals, each calculated at either the national or individual state level.

The children’s asthma care national and state averages are calculated differently.

The average rate for all healthcare organizations in the nation that provide results for a measure. The average rate is calculated by dividing the total number of patients who had the recommended care provided for a measure by the total number of patients who met the inclusion and exclusion criteria for that measure in the nation for the timeframe being reported.

The average rate for all healthcare organizations in the state that provide results for a measure. The average rate is calculated by dividing the total number of patients who had the recommended care provided for a measure by the total number of patients who met the inclusion and exclusion criteria for that measure in the state for the timeframe being reported.

Sampling

Whether or not a hospital uses sampling is determined by rules established by The Joint Commission and CMS. The same sampling methodology is used by hospitals for both their non-Medicare cases and Medicare cases and is based on the number of discharges per topic each quarter. More detailed information is available at www.QualityNet.org.

Hospital Process of Care Measure Set

List of Current Measures

Heart Attack (Acute Myocardial Infarction or AMI)

• Aspirin at Arrival
• Aspirin at Discharge
• ACE Inhibitor or ARB for Left Ventricular Systolic Dysfunction
• Beta Blocker at Arrival
• Beta Blocker at Discharge
• Fibrinolytic Medication Within 30 Minutes Of Arrival
• Percutaneous Coronary Intervention (PCI) Received Within 90 Minutes of Hospital Arrival
• Smoking Cessation Advice/Counseling

Heart Failure

• Evaluation of Left Ventricular Systolic (LVS) Function
• ACE Inhibitor or ARB for Left Ventricular Systolic Dysfunction
• Discharge Instructions
• Smoking Cessation Advice/Counseling

Pneumonia

• Oxygenation Assessment
• Initial Antibiotic Timing
• Pneumococcal Vaccination
• Influenza Vaccination
• Blood Culture Performed in the Emergency Department Prior to Initial Antibiotic Received in Hospital
• Appropriate Initial Antibiotic Selection
• Smoking Cessation Advice/Counseling

Surgical Care Improvement Project

• Prophylactic Antibiotic Received Within 1 Hour Prior to Surgical Incision
• Prophylactic Antibiotics Discontinued Within 24 Hours After Surgery End Time
• Prophylactic Antibiotic Selection
• Surgery Patients with Recommended Venous Thromboembolism Prophylaxis Ordered
• Surgery Patients Who Received Appropriate Venous Thromboembolism Prophylaxis Within 24 Hours Prior to Surgery to 24 Hours After Surgery

Children's Asthma Care

Children receiving reliever medication (like albuterol) while hospitalized for asthma. Children receiving systemic corticosteroid medication (oral and IV medication that reduces inflammation and controls symptoms) while hospitalized for asthma.

Description of Measures

The definitive description of all measures reported on Hospital Compare, including their micro-specifications, is found at the QualityNet website. The information provided below and at www.cms.hhs.gov for each of the measures is intended to be illustrative, but is not a definitive listing of the micro-specifications. The complete measure specifications can be viewed on www.QualityNet.org.

Heart Attack

Every year, about one million people suffer a heart attack (acute myocardial infarction or AMI). AMI is among the leading causes of hospital admission for Medicare beneficiaries, age 65 and older.

Scientific evidence indicates that the following process of care measures represent the best practices for the treatment of AMI. Higher scores are better.

• Aspirin at arrival - Acute myocardial infarction (AMI) patients without aspirin contraindications who received aspirin within 24 hours before or after hospital arrival.
• Aspirin at discharge - AMI patients without aspirin contraindications who were prescribed aspirin at hospital discharge.
• ACE inhibitor or ARB for left ventricular systolic dysfunction - AMI patients with left ventricular systolic dysfunction (LVSD) and without angiotensin converting enzyme inhibitor (ACE inhibitor) contraindications or angiotensin receptor blocker (ARB) contraindications who are prescribed an ACE inhibitor or an ARB at hospital discharge.
• Beta Blocker at arrival - AMI patients without beta - blocker contraindications who received a beta-blocker within 24 hours after hospital arrival.
• Beta Blocker at discharge - AMI patients without beta-blocker contraindications who were prescribed a beta-blocker at hospital discharge.
• Fibrinolytic medication received within 30 minutes of hospital arrival - AMI patients receiving fibrinolytic therapy during the hospital stay and having a time from hospital arrival to fibrinolysis of 30 minutes or less
• PCI Received Within 90 Minutes Of Hospital Arrival - AMI patients receiving Percutaneous Coronary Intervention (PCI) during the hospital stay with a time from hospital arrival to PCI of 90 minutes or less.
• Smoking cessation advice/counseling - AMI patients with a history of smoking cigarettes, who are given smoking cessation advice or counseling during a hospital stay.

Heart Failure

Heart failure is the most common hospital admission diagnosis in patients age 65 or older, accounting for more than 700,000 hospitalizations among Medicare beneficiaries every year. It is associated with severe functional impairments and high rates of mortality and morbidity.

Substantial scientific evidence indicates that the following process of care measures represent the best practices for the treatment of heart failure. Higher scores are better.

• Evaluation of left ventricular systolic (LVS) function - Heart failure patients with documentation in the hospital record that an evaluation of the left ventricular systolic (LVS) function was performed before arrival, during hospitalization, or is planned for after discharge.
• ACE inhibitor or ARB for left ventricular systolic dysfunction - Heart failure patients with left ventricular systolic dysfunction (LVSD) and without angiotensin converting enzyme inhibitor (ACE inhibitor) contraindications or angiotensin receptor blocker (ARB) contraindications who are prescribed an ACE inhibitor or an ARB at hospital discharge.
• Discharge instructions - Heart failure patients discharged home with written instructions or educational material given to patient or care giver at discharge or during the hospital stay addressing all of the following: activity level, diet, discharge medications, follow-up appointment, weight monitoring, and what to do if symptoms worsen.
• Smoking cessation advice/counseling - Heart failure patients with a history of smoking cigarettes, who are given smoking cessation advice or counseling during a hospital stay.

Pneumonia

Community acquired pneumonia is a major contributor to illness and mortality in the United States, causing 4 million episodes of illness and nearly one million hospital admissions each year.

Scientific evidence indicates that the following process of care measures represent the best practices for the treatment of community-acquired pneumonia. Higher scores are better.

• Oxygenation Assessment - Pneumonia inpatients who receive an oxygenation assessment, arterial blood gas (ABG), or pulse oximetry within 24 hours of hospital arrival.
• Initial Antibiotic Timing - Pneumonia inpatients that receive within 6 hours after arrival at the hospital. Evidence shows better outcomes for administration times less than four hours.
• Pneumococcal Vaccination Status - Pneumonia inpatients age 65 and older who were screened for pneumococcal vaccine status and were administered the vaccine prior to discharge, if indicated.
• Influenza Vaccination Status - Pneumonia patients age 50 years and older, hospitalized during October, November, December, January, or February who were screened for influenza vaccine status and were vaccinated prior to discharge, if indicated.
• Blood Cultures Performed in the Emergency Department Prior to Initial Antibiotic Received in Hospital - Pneumonia patients whose initial emergency room blood culture specimen was collected prior to first hospital dose of antibiotics.
• Appropriate Initial Antibiotic Selection - Immunocompetent patients with pneumonia who receive an initial antibiotic regimen that is consistent with current guidelines.
• Smoking cessation advice/counseling - Pneumonia patients with a history of smoking cigarettes, who are given smoking cessation advice or counseling during a hospital stay.

Surgical Care Improvement Project

Hospitals can reduce the risk of wound infection after surgery by providing the right medicines at the right time on the day of surgery. Studies show a strong association of reduced incidence of post-operative infection with administration of antibiotics within the one hour prior to surgery. After the incision is closed, however, studies show that prolonged administration of prophylaxis with antibiotics may increase the risk of certain other infections at no additional benefit to the surgical patient.

Scientific evidence indicates that the following process of care measures represent the best practices for the prevention of infections after selected surgeries (colon surgery, hip and knee arthroplasty, abdominal and vaginal hysterectomy, cardiac surgery (including coronary artery bypass grafts (CABG)) and vascular surgery). Higher scores are better.

• Prophylactic Antibiotic Received Within 1 Hour Prior to Surgical Incision - Surgical patients who received prophylactic antibiotics within 1 hour prior to surgical incision.
• Prophylactic Antibiotics Discontinued Within 24 Hours After Surgery End Time - Surgical patients whose prophylactic antibiotics were discontinued within 24 hours after surgery end time.
• Prophylactic Antibiotic Selection - Surgical patients who received the recommended antibiotics for their particular type of surgery.
• Surgery Patients with Recommended Venous Thromboembolism Prophylaxis Ordered - Surgery patients with recommended venous thromboembolism (VTE) prophylaxis ordered anytime from hospital arrival to 48 hours after Surgery End Time.
• Surgery Patients Who Received Appropriate Venous Thromboembolism Prophylaxis Within 24 Hours Prior to Surgery to 24 Hours After Surgery - Surgery patients who received appropriate venous thromboembolism (VTE) prophylaxis within 24 Hours prior to Surgical Incision Time to 24 Hours after Surgery End Time.

Children's Asthma Care

Asthma is the most common chronic disease in children and a major cause of morbidity and increased health care expenditures nationally (Adams, et al., 2001). For children, asthma is one of the most frequent reasons for admission to hospitals (McCormick, et al., 1999). Other researchers noted that there are approximately 200,000 admissions for childhood asthma in the United States annually, representing more than $3 billion dollars in healthcare costs (Silber, et al., 2003). Under-treatment and/or inappropriate treatment of asthma are recognized as major contributors to asthma morbidity and mortality

• Use of Reliever Medication for Inpatient Asthma- Use of relievers in pediatric patients admitted for inpatient treatment of asthma.
• Use of Systemic Corticosteroid Medication for Inpatient Asthma- Use of systemic Corticosteroid Medication in pediatric patients admitted for inpatient treatment of asthma.

Calculation of 30-Day Risk-Adjusted Mortality Rates

The 30-day risk-adjusted mortality measures for heart attack, heart failure, and pneumonia are produced from Medicare claims and enrollment data using a sophisticated statistical model. The model predicts patient-level deaths for any cause within 30 days of hospital admission for heart attack or heart failure or pneumonia, whether the patients die while still in the hospital or die after discharge, and calculates a "risk-adjusted" hospital mortality rate that can be used to compare mortality across hospitals. Mortality measures for heart attack, heart failure and pneumonia based on this model have been endorsed by the National Quality Forum (NQF), the non-profit public-private partnership organization that endorses national healthcare performance measures.

For more detail on how the 30-day mortality rates are calculated, click on the subheadings below.

Data Collection Methods

Cases Included in the Model

All Medicare beneficiaries aged 65 or older who were enrolled in Original Medicare (traditional fee-for-service Medicare) for the entire 12 months prior to their hospital admission for heart attack or heart failure or pneumonia, and for whom complete administrative data for that 12-month period are available, are included in the model. The model identifies (1) all short-stay acute-care hospital discharges for heart attack or heart failure or pneumonia in the reference year based on a principal discharge diagnosis on the Medicare beneficiary’s inpatient claim, and (2) all deaths (for all causes) within 30 days of admission. Hospital stays that lasted one day or less are excluded, provided the patient was discharged alive and not against medical advice. (For the initial publication of the rates in June 2007, the reference year used for calculating mortality rates is July 2005 through June 2006. Subsequent updates to the rates are expected to use the same July/June reference year.)

Hospital mortality rates for heart attack are calculated based on all admissions for heart attack, even if an individual Medicare beneficiary was hospitalized more than once for this condition during the 12-month period. However, for purposes of calculating heart failure and pneumonia mortality rates, if a beneficiary had multiple admissions during the 12-month period, one admission is chosen randomly for inclusion in the model.

Use of a 30-Day Period to Assess Mortality

The model tracks deaths that occur within 30 days of a hospital admission, rather than inpatient mortality only, or mortality over some other post-discharge period. Thirty-day mortality was chosen over inpatient mortality because variability across hospitals in lengths of stay can make differences in inpatient mortality hard to interpret. For example, a heart attack patient hospitalized for 12 days may have a higher chance of dying during the hospital stay than a patient hospitalized for only 7 days, merely because the first patient’s outcome is tracked for 5 days longer than the second patient’s. Thirty-day mortality was chosen over longer windows (such as 90 days or one year), because mortality over longer periods may have less to do with the care received in the hospital and more to do with other complicating illnesses, patients’ own behavior, or the care they received after discharge.

Use of Administrative Claims Data

Administrative claims data, rather than medical records data, are used to predict 30-day mortality. These data are widely available for Original Medicare (traditional fee-for-service) beneficiaries, are relatively inexpensive to acquire, and are timely. Using administrative data makes it possible to calculate mortality without having to do chart reviews or requiring hospitals to report additional data. Research conducted when the measures were being developed demonstrated that the administrative claims-based models perform well in predicting mortality compared with models based on chart reviews.

Risk-Adjustment and Covariates Included in the Model

Risk-Adjustment

The model adjusts for differences in patients’ risks unrelated to their hospital care (risk-adjustment). The characteristics that Medicare patients bring with them when they arrive at a hospital with a heart attack or heart failure or pneumonia are not under the control of the hospital. However, some patient characteristics may make death more likely (increase the "risk" of death), no matter where the patient is treated or how good the care is. Moreover, some hospitals may treat people with a history of more severe disease. Therefore, when mortality rates are calculated for each hospital for a 12-month period, they are adjusted based on the unique mix of patients that hospital treated during that period. Factors included in the risk-adjustment model include age, gender, past medical history, and other diseases or conditions (comorbidities) that patients had when they arrived at the hospital that are known to increase their risk.

Past medical history and comorbidities are included in the model using CMS’s hierarchical condition categories (HCCs) and a history of certain procedures. Medicare patients are assigned to one or more HCCs based on diagnoses (ICD-9 codes) obtained from the patient’s discharge claim, and from the hospital inpatient, hospital outpatient, and physician Medicare claims submitted for the patient one year prior to the admission. Secondary diagnoses from the patient’s hospital discharge claim that might represent complications that occurred while the patient was in the hospital, rather than conditions that were present on admission, are not included in assigning the patient’s HCC. Research has shown that coding differences among providers affect HCCs only slightly. Diagnoses from unreliable sources (such as laboratory or other claims that were not based on face-to-face encounters) are not included when assigning the HCCs in the model.

To "risk-adjust" mortality rates for patient characteristics, the statistical model estimates the independent effects of age, gender, comorbidities, and a hospital-specific component of quality on mortality of patients within 30 days of hospital admission (the dependent variable). Using these estimates, the model calculates an adjusted mortality rate for each hospital that can be compared with those of other hospitals with different case mixes.

Covariates in 30-Day Mortality Risk-Adjustment Models

Statistical Methods Used to Calculate Mortality Rates

Hierarchical Regression Model

The statistical model for computing 30-day risk-adjusted mortality rate measures is a "hierarchical regression model." This type of model is based on the assumption that any heart attack or heart failure or pneumonia patients treated at a particular hospital will experience a level of quality of care that applies to all patients treated for the same condition in that hospital. In other words, the expected risk of death for two similar heart attack or heart failure or pneumonia patients treated in the same hospital would be more alike than the risk of death for the same two patients treated in two different hospitals. The likelihood that an individual patient will die is therefore a combination of (1) his or her individual risk characteristics (for example, gender, comorbidities, and past medical history) and 2) the hospital’s unique quality of care for all patients treated for that condition in that hospital. The model estimates the effects of both of these components on mortality.

Calculating Mortality Rates

Each hospital’s “30-day risk-adjusted mortality rate” (also called the “Risk Standardized Mortality Rate” or RSMR) is computed in several steps. First, the predicted 30-day mortality for a particular hospital obtained from the hierarchical regression model is divided by the expected mortality for that hospital, which is also obtained from the regression model. Predicted mortality is the rate of deaths from heart attack or heart failure or pneumonia that would be anticipated in the particular hospital during the 12-month period, given the patient case mix and the hospital’s unique quality of care effect on mortality. Expected mortality is the rate of deaths from heart attack or heart failure or pneumonia that would be expected if the same patients with the same characteristics had instead been treated at an “average” hospital, given the “average” hospital’s quality of care effect on mortality for patients with that condition. This ratio is then multiplied by the national unadjusted mortality rate for the condition for all hospitals to compute a “risk-adjusted mortality rate” for the hospital. So, the higher a hospital’s predicted 30-day mortality rate, relative to expected mortality for the hospital’s particular case mix of patients, the higher its adjusted mortality rate will be. Hospitals with better quality will have lower rates.

For example, suppose the model predicts that 10 percent of Hospital A’s heart attack patients would die within 30 days of admission in a given year, based on their ages, gender mix, and pre-existing health conditions, and based on the estimate of the hospital’s specific quality of care. Then, suppose that the expected rate of 30-day deaths for those same patients were higher – say, 15 percent – if they had instead been treated at an "average" U.S. hospital. If the actual mortality rate for the 12-month period for all heart attack patients in all hospitals in the U.S. is 12 percent, then the hospital’s risk-adjusted 30-day mortality rate would be 8 percent.

If, instead, 9 percent of these patients would be expected to have died if treated at the average hospital, then the hospital’s mortality rate would be 13.3 percent.

In the first case, the hospital performed better than the average hospital and had a relatively low risk-adjusted mortality rate (8 percent); in the second case it performed worse and had a relatively high rate (13.3 percent).

Hospitals with relatively low-risk patients whose predicted mortality rate is the same as the expected mortality rate for the average hospital for the same group of low-risk patients would have an adjusted mortality rate equal to the national rate (12 percent in this example). Similarly, hospitals with high-risk patients whose predicted mortality rate is the same as the expected mortality rate for the average hospital for the same group of high-risk patients would also have an adjusted mortality rate equal to the national rate of 12 percent. Thus, each hospital’s case mix should not affect the adjusted mortality rates used to compare hospitals.

Adjusting for Small Hospitals or a Small Number of Cases

The hierarchical regression model also adjusts mortality rates results for small hospitals or hospitals with few heart attack or heart failure or pneumonia cases in a given year. This reduces the chance that such hospitals’ performance will fluctuate wildly from year to year or that they will be wrongly classified as either a worse or better performer. For these hospitals, the model not only considers deaths among patients treated for the condition in the small sample size of cases, but pools together patients from all hospitals treated for the given condition, to make the result more reliable. In essence, the predicted mortality rate for a hospital with a small number of cases is moved toward the overall U.S. National mortality rate for all hospitals. The estimates of mortality for hospitals with few patients will rely considerably on the pooled data for all hospitals, making it less likely that small hospitals will fall into either of the outlier categories. This pooling affords a "borrowing of statistical strength" that provides more confidence in the results.

Significance Testing, Interval Estimates, and Comparing Rates Among Hospitals

Significance Testing and Interval Estimates

The model also calculates how precise the estimates of the adjusted mortality rate are, and determines upper and lower bounds (Interval Estimates) for each hospital’s risk-adjusted rate. Interval estimates, which are like confidence intervals, describe how much uncertainty there is around the rate—how much bigger or smaller the rate might really be. Larger hospitals typically have more precise estimates and smaller interval estimates, since more data are available to estimate mortality. The smaller the sample size, the greater the difference in mortality rates between a hospital and the national rate must be in order for that difference to be statistically meaningful.

Comparing Mortality Rates Among Hospitals

The risk-adjusted hospital rate with its interval estimate can be compared to the U.S. National crude mortality rate. If the interval estimate includes (overlaps with) the national crude mortality rate, the hospital’s performance is in the “no different than U.S. National rate” category. If the entire interval estimate is below the national crude mortality rate, then the hospital is performing “better than U.S. National rate.” If the entire interval estimate is above the national crude mortality rate, it is “worse than U.S. National rate.”

What is the purpose of the HCAHPS survey?

The CAHPS® Hospital Survey (Consumer Assessment of Healthcare Providers and Systems), also known as Hospital CAHPS® or HCAHPS, is a standardized survey instrument and data collection methodology for measuring patients' perspectives of hospital care. While many hospitals collect information on patient satisfaction, there is no national standard for collecting or publicly reporting this information that would enable valid comparisons to be made across all hospitals. In order to make apples to apples comparisons to support consumer choice, it is necessary to introduce a standard measurement approach. HCAHPS is a core set of questions that can be combined with customized, hospital-specific items to produce information that complements the data hospitals currently collect to support internal customer service and quality-related activities.

Three broad goals have shaped the HCAHPS survey. First, the survey is designed to produce comparable data on patients' perspectives of care that allows objective and meaningful comparisons among hospitals on topics that are important to consumers. Second, public reporting of the survey results is designed to create incentives for hospitals to improve quality of care. Third, public reporting will serve to enhance public accountability in health care by increasing transparency. With these goals in mind, the HCAHPS project has taken substantial steps to assure that the survey is credible, useful, and practical. This methodology and the information it generates will be made available to the public.

Hospitals implement HCAHPS under the auspices of the Hospital Quality Alliance (HQA), a private/public partnership that includes major hospital associations, government agencies, consumer groups, measurement and accrediting bodies, and other stakeholders that share a common interest in improving hospital quality. This invitation to participate includes hospitals that are sometimes called critical access hospitals. More information about the HCAHPS survey can be found at www.hcahpsonline.org.

Note: CAHPS® (Consumer Assessment of Healthcare Providers and Systems) is a registered trademark of the Agency for Healthcare Research and Quality, a U.S. Government agency.

What items are on the HCAHPS survey?

The HCAHPS survey is composed of 27 items: 18 substantive items that encompass critical aspects of the hospital experience (communication with doctors, communication with nurses, responsiveness of hospital staff, cleanliness and quietness of hospital environment, pain management, communication about medicines, discharge information, overall rating of hospital, and recommendation of hospital); four items to skip patients to appropriate questions; three items to adjust for the mix of patients across hospitals; and two items to support congressionally-mandated reports.

On average, it takes respondents about seven minutes to complete the HCAHPS survey items.

The actual wording of the HCAHPS questions and response categories, as well as the scripts for conducting the survey in the Telephone Only and Active Interactive Voice Response (IVR) modes, can be found under “Survey Instruments” on the HCAHPS On-line website, www.hcahpsonline.org.

Which modes of survey administration can be used for HCAHPS?

Because hospitals and survey vendors survey patients a number of ways, HCAHPS is available in four different survey modes: Mail Only, Telephone Only, Mail with Telephone follow-up (also known as Mixed mode), and Active Interactive Voice Response (IVR). Detailed information on the proper use of each mode of survey administration can be found in the Quality Assurance Guidelines Version 3.0, which is located at “Quality Assurance” at www.hcahpsonline.org.

CMS recognizes that patients’ responses to the survey may be affected by the mode of survey administration. For instance, respondents typically give somewhat more positive responses when surveyed by telephone, as compared to mail. Thus, choice of mode of survey administration could potentially affect comparisons of hospitals. CMS conducted a large-scale experiment to test for mode effects, and based on this research an adjustment has been built into the calculation of HCAHPS scores. This mode adjustment is used to remove the effect of survey mode on how patients respond to HCAHPS survey items.

The Mode Experiment was based on a nationwide random sample of short-term acute care hospitals. Participating hospitals contributed patient discharges from a four-month period: February, March, April, and May 2006. Within each hospital, an equal number of patients were randomly assigned to each of the four modes of survey administration. In total, 27,229 discharges from 45 hospitals were surveyed.

In general, patients randomized to the Telephone Only and active IVR provided more positive evaluations than those randomized to the Mail Only and Mixed modes. Mode effects varied little by hospital.

More information, as well as an overview of the results of the mode experiment, can be found under “Mode Adjustment” at www.hcahpsonline.org.

What must hospitals do in order to participate in HCAHPS?

CMS has developed detailed Rules of Participation and Minimum Survey Requirements for hospitals that either self-administer the survey or administer the survey for multiple hospital sites, and for survey vendors that conduct HCAHPS for client hospitals.

The HCAHPS Rules of Participation include the following activities and steps:

• Attend HCAHPS Training
• Follow the Quality Assurance Guidelines Version 3.0 and Policy Updates
• Attest to the accuracy of the organization’s data collection process
• Develop a HCAHPS Quality Assurance Plan
• Become a QualityNet Exchange Registered User for data submission
• Participate in oversight activities conducted by the HCAHPS Project Team

Hospitals and survey vendors administering the survey must also meet HCAHPS Minimum Survey Requirements with respect to survey experience, survey capacity, and quality control procedures. Details about these activities, steps and requirements can be found in the Quality Assurance Guidelines Version 3.0 under “Quality Assurance” at www.hcahpsonline.org.

Note: If a hospital, or its survey vendor, is found to be non-compliant with these rules or requirements, the hospital’s HCAHPS data may not be publicly reported.

Which patients are eligible to participate in HCAHPS?

The HCAHPS survey is broadly intended for patients of all payer types that meet the following criteria:

• 18 years or older at the time of admission
• At least one overnight stay in the hospital as an in-patient
• Non-psychiatric DRG/principal diagnosis at discharge
• Alive at the time of discharge

Patients who meet these criteria (except those that fall into an exclusion category, see below) should be included in the sample frame from which the survey sample is drawn.

A patient’s principal diagnosis at discharge is used to determine whether he or she falls into one of the three service line categories (maternity care, medical, or surgical) for HCAHPS eligibility. The Diagnosis-Related Group (DRG) is the preferred method for determining whether the service line is Maternity Care, Medical, or Surgical.

Pediatric patients (under 18 years old at admission) and psychiatric patients are ineligible because the current HCAHPS instrument is not designed to address the unique situation of pediatric patients and their families, or the behavioral health issues pertinent to psychiatric patients. Patients whose DRG/principal diagnosis is Maternity Care, Medical, or Surgical but who also have psychiatric comorbidities are eligible for the survey. Patients who did not have an overnight stay are ineligible because their experiences and interactions with the staff during the hospital visit may be limited. There are a few categories of otherwise eligible patients who, because of logistical difficulties in collecting data, are excluded from the sample frame before the random sample is selected. These are:

• Patients discharged to hospice care
• Court/Law enforcement patients (i.e., prisoners)
• Patients with a foreign home address (excluding U.S. territories—Virgin Islands, Puerto Rico, and Northern Mariana Islands)
• “No-Publicity” patients (see below)
• Patients who are excluded because of rules or regulations of the state in which the hospital is located

More information about patient eligibility and exclusions for the HCAHPS survey can be found in the Quality Assurance Guidelines Version 3.0 under “Quality Assurance” at www.hcahpsonline.org.

Note: A "No publicity patient" is a patient who requests at admission that the hospital: 1) not reveal that he or she is a patient; and/or 2) not survey him or her.

Note: Hospitals must document their use of all patient exclusions.

How are patients sampled for the HCAHPS survey?

The HCAHPS sampling protocol is designed to capture uniform information on hospital care from the patient’s perspective. HCAHPS scores are designed to reflect the care received by patients of all payer types, not just Medicare beneficiaries. Therefore, patients of all payer types are eligible for sampling.

The basic sampling procedure for HCAHPS is the drawing of a random sample of eligible discharges on a monthly basis. Smaller hospitals should survey all HCAHPS-eligible discharges.

Data are collected from patients throughout each month of the 12-month reporting period. Data are then aggregated, on a quarterly basis, to create a rolling 4-quarter data file for each hospital. The most recent four quarters of data are used for public reporting. To ensure comparability, hospitals may not switch type of sampling, mode of survey administration, or survey vendor within a calendar quarter. More information about the HCAHPS sampling protocol can be found in the Quality Assurance Guidelines Version 3.0 under “Quality Assurance” at www.hcahpsonline.org.

How is the sample drawn for the HCAHPS survey?

The basic sampling procedure for HCAHPS entails drawing a random sample of all eligible discharges from a hospital on a monthly basis. Sampling may be conducted either continuously throughout the month, or at the end of the month, as long as a random sample is generated from the entire month.

The target for the statistical precision of the publicly reported hospital scores is based on a reliability criterion. In brief, higher reliability means a higher ratio of “signal to noise” in the data. The reliability target for the HCAHPS global ratings and most composites is 0.8 or higher. Based on this target, hospitals must obtain at least 300 completed HCAHPS surveys (“completes”) over the entire 12-month reporting period.

The HCAHPS sample must be drawn according to this uninterrupted random sampling protocol. Hospitals/Survey vendors must sample from every month throughout the entire reporting period and not stop sampling or curtail ongoing interview activities once a certain number of completed surveys has been attained. All completed surveys should be submitted to the HCAHPS data warehouse. More information about the HCAHPS sampling protocol can be found in the Quality Assurance Guidelines Version 3.0 under “Quality Assurance” at www.hcahpsonline.org.

Note: Smaller hospitals that are unable to reach the target of 300 completes in a 12-month reporting period must survey ALL eligible discharges and attempt to obtain as many completes as possible.

When are patients surveyed?

Sampled patients are surveyed between 48 hours and six weeks after discharge, regardless of the mode of survey administration. Distributing surveys to patients while they are still in the hospital is not allowed.

Data collection for sampled patients ends no later than six weeks following the date the first survey is mailed (Mail Only and Mixed Modes) or the first telephone attempt (Telephone Only and IVR Modes). More information about the HCAHPS sampling protocol can be found in the Quality Assurance Guidelines Version 3.0 under “Quality Assurance” at www.hcahpsonline.org.

Will HCAHPS results be adjusted prior to public reporting?

CMS recognizes that patients’ responses to survey items may be affected by the mode of survey administration (e.g., respondents may give somewhat higher ratings on average when the survey is conducted by telephone as opposed to mail). Thus, the choice of mode of survey administration, as well as patient-mix and non-response tendencies could potentially affect cross-hospital comparisons.

To ensure that differences in HCAHPS results reflect differences in hospital quality only, HCAHPS survey results will be adjusted for: 1) patient-mix; 2) mode of data collection; and, 3) non-response bias. Only the adjusted results will be publicly reported and will be considered the official results. There will not be an adjustment for hospital size.

CMS conducted a large-scale mode experiment in Spring 2006 to test for mode, patient-mix and non-response effects and based on this developed adjustments for the calculation of HCAHPS results.

The adjustment model also addresses the impact of patient-mix across hospitals, which can systematically impact responses to the survey. Several questions on the survey, as well as items drawn from hospital administrative data, will be used for the patient-mix adjustment. Neither patient race or ethnicity will be used to adjust HCAHPS results; these items have been included on the survey to support congressionally-mandated reports. The adjustment model also addresses the effects of non-response bias.

While CMS will publicly report HCAHPS results for hospitals that obtain fewer than 100, the lower precision of these results will be noted in public reporting.

More information, as well as an overview of the results of the mode experiment, can be found under ”Mode Adjustment” at www.hcahpsonline.org.