NORC--1998 Best HMOs Methodology

U.S. News & World Report's "Top HMOs" 1998 Methodologic detail
Dr. Craig Hill and Whitney Moore of NORC conducted the analysis and developed the methodology that resulted in the scores and rankings of HMO and POS plans nationwide. Below is a general, then a more specific, description of the methodology behind the rankings.

Overview
On behalf of U.S. News & World Report, we analyzed data collected by the National Committee for Quality Assurance (NCQA), a Washington, D.C., group that accredits managed-care plans. Our analysis concluded that differences in quality were largely explained by 28 variables grouped into five dimensions: Prevention, Adults' Access to Care, Member Satisfaction, Physicians' Credentials, and Children's Access to Care. Using these results, we calculated overall scores using values from these 28 variables and then ranked 271 plans in 45 states and the District of Columbia. (Eighteen plans could not be ranked because of insufficient data.)
      We then arithmetically transformed the overall score so that the highest-performing plan automatically earned a score of 100, the lowest a 0. (On this scale, score differences of less than 2.5 are not statistically significant.) Based on overall score, plans were awarded one to four stars: those scoring at least one standard deviation above the mean received four stars; between the mean and one SD earned three stars; from the mean to one SD below the mean got two stars; and, finally, those plans scoring more than one SD below the mean were given one star.

Step-by-step detail
Our conceptual framework has as its foundation extant literature in health services research and our own prior work which suggests that there are broad dimensions underlying the foundation of high quality care in health plans. Our analytic plan, then, involved identification of these dimensions and the grouping of variables within them via factor analysis, and sub-scoring of these dimensions guided by the factor analysis solution. A step-by-step run-through of NORC's methodology follows.

Preliminary analysis of missing data
Because not all HMOs submit data to NCQA and not all plans that do submit allow data to be reported publicly, several preliminary analyses were done to determine the utility of the data that were available and whether or not the data we did have would be effective in delineating differences between plans. First, we conducted a comparison of current data to last year's data to determine the number of plans per state and the number of covered lives per state. A review of these analyses allowed us to conclude that we did have enough data to proceed, though some states would have sparse coverage. The final data set used for this analysis consisted of 289 plans (202 HMOs and 87 POSs).
      In addition, we reviewed item non-response rates; that is, for each variable, we reviewed the number of plans which did not report a value and for which the variable was not applicable. This step resulted in many variables being dropped from further analysis.

Variable reduction
Next, we produced basic univariate statistics, including means, medians, standard deviations, minimums, and maximums, for all numeric variables, and frequencies were run for all character variables. During this step, we checked for out-of-range values in each variable and worked with NCQA to explain or correct any discrepancies found between the actual and expected values and ranges.
      In a drive for parsimony, we produced a correlation matrix and examined bivariate relationships between variables, identified variables which exhibited no correlation with others, and removed them, thereby reducing further the number of variables considered for the final model.

Model building/factor analysis
The factor analysis technique can help researchers detect clusters of variables that "load," or group, together in a large amount of data. These "factors," or groups of variables, define some underlying constructs and can, taken together, explain large portions of the variation in a dataset. In our application, once the final factor structure is determined, the variables that load onto each factor can be combined to form a scale and used in the calculation of a final score for the plans.
      Our final factor analysis model was built via several iterations of factor analysis runs. At each iteration, we reviewed the results and made decisions to remove variables based on the results. Our examination of two- and three-dimensional factor loading plots of the components led us to identify variables which appeared to load independently or to load equally (but weakly) on all factors; these were removed from subsequent runs.
      Our final and optimal model resulted in a factor analysis solution showing five tightly grouped and intuitively-appealing, logically-consistent components (all with eigenvalues of above 1). These five factors explain a total of 66.3% of the overall variance. The final factors, and our labels, are listed below, along with the percentage of the variance each explains.

1. Prevention (29.7% of the variance)
This factor is based on 11 quality measures including percentages of plan members' children who had all immunizations and who received the recommended number of well-child visits in the first fifteen months, from ages 3 to 6, and from ages 12 to 21; members who received recommended prenatal care in the first trimester, check-ups after delivery, breast cancer screening, cervical cancer screening, beta blocker treatment after a heart attack, retinal exams for diabetics, and follow-up after hospitalization for mental illness.

2. Access to Care-Adults (13.3% of the variance)
Includes percentages of adults aged 20-44, 45-64, and 65+ with access to preventive/ambulatory services from 1995-1997, and the percentage of primary care providers in the plan who were accepting new patients. The latter variable--percentage of primary care providers with open panels—loads negatively; thus, we inverted (100 - x) the values for this variable for inclusion in the final set of calculations. We interpret this result to suggest that highly-regarded providers are likely to have a closed panel, due to their "popularity."

3. Member Satisfaction (9.4% of the variance)
Includes percentages of patients who responded favorably to questions about overall satisfaction, receiving needed care, ability to obtain referrals to specialists, choice of physicians, and making appointments.

4. Physicians' Credentials (7.8% of the variance)
Includes percentages of primary care providers, physician specialists, pediatricians, OB/GYN providers, and geriatricians who are board certified in their specialty.

5. Access to Care-Children (6.1% of the variance)
Includes percentages of children aged 12-24 months, 25 months to 6 years, and 7-11 years with access to primary care providers in 1997.

Scoring of plans
After the factors were determined, calculations of the final scores and ratings could be made. Several steps went into the calculation of the final score.

Imputation
Data supplied to us from NCQA include several missing values, according to the reason for the missing data. We were provided with the following missing value codes:

‘NR'—Not Reported: plan chose not to supply information for a particular variable

‘NA'—Not Applicable: particular service or measure is not offered by the plan

‘EXC'—Excluded: information was provided by the plan, but the given value fell out of the logical range for that particular variable

      For the calculation of the final score, all missing values were imputed. Any variable with an ‘NR' or ‘EXC' was assigned a value equivalent to the minimum value in our data for that variable. Any variable with an ‘NA' was given a value equivalent to the mean value in our data for that variable.

Calculation of averages
Averages for each plan were calculated for all five factors. The average for each factor was computed by summing the values for each of the measures included in the factor and dividing by the total number of measures. For example, the average for the first factor was calculated by adding the values across the plan for each of the 11 measures included in the factor and dividing by 11. In some cases, a plan had missing values for one or more of the variables. If all variables were missing for a particular plan, an average was not calculated and they were given a value of ‘NA' or ‘NR', depending on the type of missing data in that factor (see above for distinction between ‘NA' and ‘NR'). If some but not all of the variables had missing values, the imputation described above provided values for use in the calculation of the averages. Thus, all plans that had at least one true value for a given factor were given an average for that factor.

Calculation of the final raw score
The final "U.S. News score" was calculated in several steps. First, we converted the values for all 28 variables from percentages (ranging from 0 to 100) to decimals (ranging from 0 to 1) by dividing their values by 100. (The "availability of primary care physicians—open panel" was reverse-coded because of its negative relationship with the other variables in the "Adults' access" factor).
      Second, each decimal variable in a factor was multiplied by the percentage of variance (also in decimal form) that factor describes. For example, each of the 11 variables in the "Prevention" factor were multiplied by .297, because that factor explains 29.7% of the variance in our data (see "Model Building/Factor analysis" for the variance percentages for each factor). Third, these values were summed within factors, resulting in a sum for each of the factors that is "weighted" by the percentage of variance explained by that factor. Fourth, the five factor-based sums were then added to compute a total sum for all five factors.

Transformed scores
Once this total raw sum was calculated, we calculated the minimum and range for the sum over all plans for which a final score was calculated (N=271). These were used to compute a final score on a 0-100 scale, using the formula: (x - min)/range.

Star system
Finally, the star rating was determined based on the mean and standard deviation of the final score. A four star rating was given to any plan scoring one or more standard deviations above the mean, which translates to a final score of 82-100. Three stars were given to any plan scoring between the mean and one standard deviation above the mean (a final score of 60-81). Two stars were given to a plan scoring between the mean and one standard deviation below the mean (a final score of 42-59). And one star was given to any plan scoring one standard deviation or more below the mean (a final score of 0-41).

      View the complete findings at the U.S. News & World Report's Web site

      Back to What's New