Report Cards • 349

interpret the results. In a similar vein, surveys of existing customers are susceptible to survivor bias, whereby those customers who did not like the product or service no longer use it and therefore are not surveyed. This is a problem for Amazon.com and similar ratings sites. Fans of specific authors and musicians tend to purchase their latest books and download their CDs and are also more likely to post rave reviews. A consumer who visits Amazon to learn about an unfamiliar author or musician will not see a representative set of reviews.

Customer demographics can also affect satisfaction ratings. For example, a study of patient satisfaction with mental health providers found that women and older patients tended to be more satisfied, even though the quality of care they received was objectively similar to the quality offered to younger, male patients.14 Satisfaction can also vary with the customer’s race, income, and education. As a result of these biases, firms that get good report card scores may not provide the best quality. Instead, they may simply be serving the “right customers”—those who tend to give higher scores as a matter of course. Not only does this add unwanted noise to report card rankings, it gives firms an incentive to judiciously choose their customers. In order to defeat these problems, the certifier should adjust the scores to account for differences in customer characteristics. Nowhere is the need to adjust quality ratings for customer characteristics more apparent than in health care, where report card scores that are not risk adjusted are worthless.

Risk Adjustment

Consider three surgeons, Doctors A, B, and C, performing the same hip replacement procedure on patients 1, 2, and 3. Patient 1 survives the surgery with no complications and six months later is able to do all normal daily activities such as climbing stairs and walking the dog. Patient 2 suffers a postoperative infection and, as a result, is never again able to climb stairs and walks with a pronounced limp. Patient 3’s surgery is uneventful, but the patient also never fully regains mobility. It might be tempting to conclude that Doctor A did a better job than Doctors B and C, but this would be premature. Many factors besides the surgeon’s skill determine outcomes. The anesthesiologist, nursing team, and physical therapist all play big roles, which is why it is important to consider the quality of the hospital and not just the individual surgeon.

Surgical outcomes also depend on the patient. Patient 1 may have been more persistent during weeks of painful rehabilitation. Patient 2 might have been older or frailer, while patient 3 lacked the financial resources to hire a caregiver after returning home from the hospital. Any of these factors may have contributed to patient 1’s superior outcome, yet none should reflect badly on Doctors B and C.

In order to properly evaluate the quality of a medical provider, it is essential to perform risk adjustment. Risk adjustment is a statistical process in which raw outcome measures, such as a surgeon’s average patient mortality rate, are adjusted for factors that are beyond the control of the seller.15 Health care provider report cards that do not perform some form of risk adjustment can be extremely misleading. Bear in mind that the best medical providers often get the toughest cases. If report cards are not risk adjusted, the best providers can end up at the bottom of the rankings. Example 10.5 explains the risk adjustments used in one of the first and best hospital quality report cards—the New York State cardiac surgery report card.

Certifiers should perform risk adjustment whenever measured quality depends on the characteristics of the customer. This includes all quality reports based on customer satisfaction as well as report cards for services such as health care and education.

350 • Chapter 10 • Information and Value Creation

The certifier can follow the steps in Example 10.5, substituting the outcomes and risk adjusters that are appropriate for the industry in question. Unfortunately, most certifiers outside of health care do not perform any kind of risk adjustment, making it difficult to confidently identify the best quality sellers. For example, education metrics such as test scores and graduation rates may say more about student demographics than they do about teaching quality. Or consider the widely cited on-time arrival and lost baggage statistics for airlines. Southwest Airlines routinely outperforms other major U.S airlines on these dimensions of performance. Although many industry analysts cite this as evidence of Southwest’s operational excellence, it more likely reflects Southwest’s “point-to-point” route system, which is far easier to operate than the other carriers’ “hub-and-spoke” systems. On any given origin/ destination pair, Southwest may perform no better than other carriers. To take another example, one of the most widely cited metrics in consumer goods—the Consumer Reports automobile and appliance repair frequencies—may reflect how customers use different brands as much as the reliability of those brands. BMW drivers may accelerate hard and brake harder, while Lexus drivers may pamper their cars. Resulting differences in repair records may have little to do with actual build quality.

Presenting Report Card Results

The presentation of report card scores can have a big effect on their impact. For many years, General Motors collected enormous quantities of data on the quality of the health plans available to its enrollees. Beginning in 1996, GM provided this information to its employees in the form of easy to understand “diamond” ratings, as shown in Figure 10.1. Plans could receive up to three diamonds in each of six categories, including “Preventive Care” and “Access to Care.” Employees responded by dropping from low scoring plans.16 GM believed that the simplicity of the presentation was crucial to the success of the report card.

Another important feature of the GM report card was that it captured dozens of quality metrics using just six scores. GM did this by creating composite scores—scores that summarize the information in multiple measures. Other report cards that present composite scores include U.S. News and World Reports’ rankings of universities and of hospitals as well as the World Health Organization’s rankings of national health systems.

The simplest way to create a composite score is to sum up or average individual scores. This of course requires all of the individual component scores to be measured on the same numeric scale. For example, when computing a composite ranking of universities, one might use the following scoring procedure:

•Acceptance Rate Score 5 100 2 Acceptance Rate

•Yield Score 5 Yield

•High School GPA Score 5 (Average HS GPA/4) 3 100

Composite score 5 (Acceptance Rate Score 1 Yield Score 1 High School GPA Score)/3 Using this formula, one can compute and compare composite scores for different

universities. Table 10.4 gives an example.

The composite scores in Table 10.4 weight each component score equally. It is often appropriate to emphasize some scores more than others by computing a weighted average score. A student’s grade point average is the weighted average of individual class grades, where the weights are the credit hours for each class. U.S. News’

Report Cards • 351

FIGURE 10.1

COMPARING YOUR 1997 GM MEDICAL OPTIONS

The following table shows the rating of the HMO option(s) available in eight selected quality measures. The ratings are based on historical data and therefore may not necessarily represent the quality of care you will receive in the future. GM does not endorse or recommend any particular medical plan option. The medical plan you elect is your personal decision.

For a more complete description of the eight selected quality measures, see the GM Medical Plan Guide.

HMO and PPO options are based on the plan service area. Eligibility is determined by zip code. You may not be eligible for any or all options listed. You may be eligible for other options if you live near a state line. See your enrollment information for your available options.

Michigan – Detroit

TABLE 10.4

Composite Report Scores for Universities