On June 18, 2008, the Centers for Medicare & Medicaid Services (CMS) announced plans to add a five-star ranking system to its nursing home website, Nursing Home Compare (NHC), by December 2008.[1]  Under the proposed system, each nursing facility participating in Medicare or Medicaid, or both, will receive one to five stars, with a composite score based on information currently on the website.

CMS's rankings will combine three factors: federal survey data for three years (including complaints), some quality measures, and staffing data.  The Center for Medicare Advocacy is concerned that consumers may be misled by the rankings.  Self-reported data, which are two of the three sources of information for the rankings, will make nursing home quality appear to be better than it actually is.

CMS held an Open Door Forum on June 24 to explain the proposal and to hear from the public.[2]  It has also requested that members of the public submit comments by July 23 on nine specific questions, including the concept of a rating system, data sources (e.g., which of the 19 quality measures currently reported on Nursing Home Compare should be used), and whether resident satisfaction surveys should be included.[3]  

Although the Center for Medicare Advocacy supports efforts by CMS to provide more and better information to the public about nursing home quality on Nursing Home Compare, we question CMS's plan for several reasons.

Quality Measures are Self-reported, Unaudited, and Inaccurate

Nursing Home Compare currently reports 19 quality measures reflecting physical and clinical characteristics of residents, such as residents who are physically restrained and residents with moderate to severe pain.  The measures, derived from resident assessment information that facilities collect in order to develop residents' plans of care, are converted into percentile scores by CMS.  The measures were originally developed to help focus surveyors on potential problem areas as they conduct surveys.  As indicators of actual quality, however, the measures are suspect.[4] 

Pain is an especially suspect quality measure.  Nursing facilities report considerably lower rates of pain than independent researchers consider true.[5]  A CMS-sponsored project, Data Assessment and Verification (DAVE), which was designed to help facilities conduct more accurate resident assessments, found that facilities had the highest discrepancy rates in their assessments of medications and pain.[6]  CMS's own quality-based purchasing demonstration will not use either of the publicly-reported pain measures "because of concerns about differences across nursing homes in how they assess pain."[7]  Previous studies "found that the [Minimum Data Set] (MDS) underreports pain in cognitively impaired residents."[8]  A major change planned by CMS for the resident assessment instrument, the MDS 3.0, which is scheduled for implementation on October 1, 2009, is the determination of resident pain.  Resident interviews will replace staff observations; MDS 2.0 has "repeatedly shown to have poor correspondence with independent pain assessments."[9]

Last fall, four prominent nursing home researchers who developed a nursing home ranking system for Consumer Reports argued that rankings of performance measurement, while popular, are of questionable validity because:

• Nursing facilities can perform well in some areas, but poorly in others.  There is usually little correlation among the criteria.

• Nursing home residents are very varied and have different needs.  Different aspects of quality are important to different residents.

• It is difficult to attribute differences in resident outcomes to nursing home characteristics.

• Efforts to "risk adjust" various performance measures are problematic, either justifying poor care or failing to recognize differences in care.[10]

The researchers recommended a "less ambitious" approach to performance measurement.  Instead of reporting on all 15,000 nursing facilities nationwide, they recommended identifying the worst 10% of facilities in each state, "based solely on the review of each home's record over its three to four most recent annual certification/licensure surveys by the state agency."

Nurse Staffing Data are Similarly Self-reported, Unaudited, and Suspect

In 2005, CMS reported, "The limitations of staffing data on NHC derived from the Online Survey, Certification and Reporting (OSCAR) system that was not originally designed for this use, have been widely known for some time."[11]  CMS announced "a number of short-term interim steps for improving the current OSCAR system for reporting nursing home staffing," including "a set of exclusion rules for suspect data."  Using data sources that "are known to be more accurate than OSCAR," including "Medicaid cost reports, payroll data, and prior CMS staff time studies," CMS announced that it would compare facility-reported data to thresholds and temporarily exclude suspect staffing data.  That system remains in place today.

A recent study found that nursing homes over-report staffing levels compared with staffing reported on audited Medicaid cost reports.  Over-reporting of registered nurse coverage was associated with for-profit ownership of nursing homes.  The researchers recommended more careful scrutiny of staffing levels in for-profit facilities during the survey process and that improvements be made to the process of public reporting of staffing levels.[12]

The Nursing Home Industry's Recommendations Lack Merit

The three nursing home trade associations expressed concern about using survey data and want CMS to include resident satisfaction scores in any ranking system.[13]  While state surveys are not a perfect measure, resident satisfaction surveys are even more suspect.

State Surveys are independent evaluations of nursing facility performance.  Annual surveys are conducted by state survey agencies, usually the state's department of health, using protocols, procedures, and forms developed by CMS.  A consumer concern about surveys is the repeated finding by the Government Accountability Office, in a series of reports issued since 1998, that surveys understate deficiencies and cite deficiencies as less serious than they actually are.[14]  The survey component of CMS's proposed ranking system, while more objective and independent than quality measures and staffing information reported by nursing facilities, nevertheless provides a more positive statement about quality than justified.  Moreover, states are increasingly using their state enforcement systems, instead of the federal system, to sanction facilities for noncompliance with standards of care.  State enforcement actions do not appear on Nursing Home Compare.

Resident and family satisfaction surveys are a method of providing additional information.  At the Open Door Forum on June 24, Thomas E. Hamilton, Director of CMS's Survey and Certification Group, acknowledged the importance of determining resident satisfaction.  However, he noted that resident satisfaction surveys are currently used by nursing facilities for internal feedback and that there is a need to ensure that satisfaction surveys used in the public arena are objectively measured and consistent.

Recommendations

The Center for Medicare Advocacy recommends that CMS provide more and better information on Nursing Home Compare, including links to the actual survey forms and information about staff turnover.  The Center also recommends that CMS use payroll data to report staffing information.

Conclusion

The Center for Medicare Advocacy encourages the public to submit comments to CMS on the ranking system.

The Center supports good public information about nursing home quality.  Until the components of the ranking system more accurately reflect nursing home quality, ranking nursing homes on Nursing Home Compare may be premature. 

Finally, even with accurate and comprehensive public information, the Center supports a strong enforcement system to ensure that nursing homes meet the mandate of the Nursing Home Reform Law and provide each resident with care and services "to attain or maintain the highest practicable physical, mental, and psychosocial wellbeing."[15]