INTRODUCTION

TOP ABSTRACT INTRODUCTION METHODS RESULTS DISCUSSION REFERENCES

Hospitalization for pediatric asthma has been increasing in the United States over the past decade (1). Adverse outcomes, including hospitalization and emergency department (ED) visits, account for more than 40% of the direct medical costs of the disease (2). It is believed that optimal outpatient care might reduce adverse outcomes of pediatric asthma (3).

Clinical resources for managing populations with asthma, such as physician or case-manager time, are often limited. Optimal management of asthmatic populations will require directing resources toward the highest-risk patients. For pediatric asthma, risk prediction is particularly challenging. Less than 10% of children with asthma will be hospitalized during a given year (4). Previous studies have identified prior hospital and outpatient utilization, spirometry results, and medication utilization as predictors of future asthma hospitalization (5, 6). However, we are unaware of any studies that have attempted to validate prediction models by using computerized data for large populations.

The objectives of the present study were to: (1) develop and validate prediction models that use computerized data to identify children at high risk for asthma-related hospitalization or ED visits; and (2) evaluate the projected cost-effectiveness of a hypothetical asthma-management program with these models.

	METHODS

TOP ABSTRACT INTRODUCTION METHODS RESULTS DISCUSSION REFERENCES

Overview

The study was a retrospective cohort study of children at a regional health-maintenance organization (HMO). Information on health-services utilization and on subsequent asthma hospitalizations was collected from computerized data bases. Individuals were randomly assigned to a learning set and a test set of observations to enable split-sample model construction and validation. To identify variables that predicted hospitalization and ED visits, models were constructed with the learning set with two different approaches: proportional-hazards regression and classification trees. Decision analysis was used to evaluate the potential cost-effectiveness of prediction models.

Study Population

Northern California Kaiser Permanente is a group-model HMO that provides the full spectrum of primary to tertiary care to approximately 2.4-million members at 16 hospital-based medical centers and 15 freestanding medical clinics. All children aged 0 to 14 yr on April 1, 1994, from 15 medical clinics with computerized pharmacy-utilization data available for the study period, were eligible.

The study cohort of children with asthma-related utilization was identified within this clinic population. To be included, a child must have had one or more of the following during the 12 mo prior to the index date (April 1, 1994): (1) a hospitalization with a Ninth International Classification of Diseases (ICD-9) code for asthma (Codes 493.00 to 493.99); (2) an ED visit for asthma, identified by a set of previously validated text strings; and (3) an asthma-related medication record on the pharmacy information-management system. Asthma-related medications were inhaled and oral -agonists, cromolyn, inhaled corticosteroids, and methylxanthines. Children with only one -agonist prescription and no other asthma utilization were not included because these prescriptions were thought likely to represent bronchiolitis, a trial of -agonists for persistent cough, or very mild asthma. These criteria were chosen to achieve good sensitivity for identifying children with asthma, although it is likely that they also identified some children without definite asthma (7).

Only children with active health-plan membership as of the index date were included. Very young children (< 2 yr) and children with chronic conditions such as bronchopulmonary dysplasia were not excluded, because they have asthma morbidity at least equal to, if not greater than, that of other children. Although approximately 20% of children in this health plan are not insured for medications, these children were not excluded from the study group, because we wanted to develop and test the effectiveness of the asthma-prediction models under real circumstances.

Data Collection

For each patient, information on predictor variables was collected from computerized data bases. Predictor variables included sex, age, insurance type (commercial versus Medicaid) and copayment, and the presence or absence of an identified personal physician. Hospitalizations and ED visits for asthma during the 24 mo prior to the index date were identified. Using ICD-9 codes, asthma hospitalizations were selected as those with either: (1) a principal diagnosis of 493.00 to 493.99; or (2) any diagnosis of 493.00 to 493.99 plus a principal diagnosis of pneumonia, bronchitis, bronchiolitis, acute upper respiratory infection, pulmonary collapse, respiratory failure, pulmonary insufficiency, pneumothorax, bronchopulmonary dysplasia, or viral infection. ED visits for asthma, pneumonia, bronchitis, and other respiratory conditions were identified with a set of text strings validated in previous studies.

Outpatient clinic visits during the 12 mo prior to the index date were identified. No computerized information was available about diagnoses at outpatient visits for the period of the study. Information on utilization of the asthma-related medications listed above, plus oral corticosteroids, was collected from the pharmacy information-management system for the 12 mo prior to the index date. For -agonist inhalers, each inhaler was counted as one unit; for other medications, each dispense (whether an initial prescription or a refill) was counted as one unit. The number of different physicians prescribing asthma- related medications was used to measure the continuity of care; this was a modification of a previously described index (8).

Hospitalizations and ED visits for asthma were retrospectively identified during a 12-mo follow-up period (April 1, 1994 to March 31, 1995) from data bases that included all hospitalizations at Kaiser Permanente hospitals and those hospitalizations at non-Kaiser hospitals that were paid for by the health plan.

Prediction Models

A proportional-hazards regression was conducted with SAS software (SAS Institute, Inc., Cary, NC) to obtain estimates of the relative risk of hospitalization associated with the predictor variables (9). In the proportional-hazards models, follow-up time was censored when a patient either: (1) had a gap in health-plan membership that signified outmigration; or (2) experienced the adverse outcome being analyzed. Preliminary modeling was conducted to select appropriate predictors for inclusion in future models, to evaluate the linearity of the association between each medication variable and the outcome, to select appropriate cutoff points for medication variables, and to identify variables for inclusion in the final model.

Classification trees to identify predictors of hospitalization and ED visits were constructed using recursive partitioning (10). This method, also known as the classification and regression tree (CART) method, has the advantage of generating models that are simpler and clinically more intuitive than regression models. It is a nonparametric technique that splits subjects, on the basis of predictor variables, into groups with and without the outcome of interest. The method has the advantage of being free of distributional assumptions and relatively insensitive to outliers. It also identifies interactions automatically (i.e., it allows for different relationships between a predictor and the outcome of interest at different levels of other predictors).

Analyses were conducted with CART software (Salford Systems, San Diego, CA), which enabled us to specify variable costs of misclassification. This helped account for the fact that the cost of hospitalization was higher than the cost of the clinical intervention intended to prevent it. Misclassifying a patient destined for hospitalization as being at low risk could be assigned a higher relative cost than misclassifying a patient not destined for hospitalization as being at high risk. This produced models with higher sensitivity than those produced by using misclassification costs of 1:1.

Several sets of classification trees based on different assumed misclassification costs were generated, using the test sample to determine the optimum tree. We simplified these trees by first determining the sensitivity, specificity, and positive predictive values of each potential combination of subgroups identified as being at high-risk. Second, we selected combinations of high-risk subgroups that were judged to be clinically meaningful among sets of potential trees that had similar positive predictive values.

Model Validation

In our split-sample validation, the cohort was randomly divided in half. Models were built with one-half cohort (the learning set) and were validated with the other half (the test set). Receiver-operating-characteristic (ROC) curves were used to validate the proportional-hazards models (11). Each patient was assigned a risk score based on his or her regression coefficients from the proportional-hazards model. ROC curves for the learning and test sets were constructed on the basis of the sensitivity and specificity of decile cutoffs in predicting the outcome being studied. The area under the ROC curve is a measure of the performance of a model. The lower confidence limit for the area under an ROC curve should be greater than 0.50 for a model that performs significantly better than chance; a curve with an area of 0.50 denotes a test with a true-positive rate equal to the false-positive rate, which is no better than chance. The curves were constructed and analyzed according to standard parametric and nonparametric methods, as appropriate, using software provided by Robert Centor, M.D. (University of Alabama, Birmingham) (12, 13).

Cost-effectiveness Analyses

Decision analysis is a quantitative modeling method that has been used to project the costs and outcomes of interventions for asthma and many other health-care problems (14, 15). Using decision analysis, we conducted a cost-effectiveness analysis for a hypothetical population of children with asthma-related health-services utilization. The alternatives evaluated were: (1) disease management with a prediction model and a clinical intervention; and (2) disease management without using a prediction model. If disease management with a prediction model were implemented, the model would be used to classify patients as being at high risk or low risk. Those classified as high risk would receive the clinical intervention.

The probabilities of future asthma-related hospitalizations and ED visits were estimated from analysis of the study cohort. The sensitivity and specificity of the prediction model were based on our findings in the present study for the test set of observations. Average costs of hospital days and ED visits were estimated with Kaiser Permanente's Cost Management Information System, an accounting system that integrates financial and utilization data to produce patient-level and service costs.

Published data on the costs and effectiveness of various interventions for pediatric asthma are limited, but randomized trials of asthma-education programs provide a relevant example (16). To derive our baseline assumptions about the hypothetical clinical intervention, we chose a randomized trial of an asthma-education program that was conducted in a population similar to ours (17). This other study by Lewis and colleagues included only patients with asthma severe enough to require medication for at least 25% of days in an average month. The cost of asthma education was $125 per child in 1984, which we updated, using the medical component of the Consumer Price Index, to $260 in 1995 (18).

On the basis of Lewis's findings, we assumed that the asthma-education program would reduce hospital days by 0.29 d per patient in the following year among patients otherwise destined for an asthma-related hospitalization. We also assumed that an asthma-education program would reduce future ED visits by 1.0 visit per patient in the following year among patients otherwise destined for an asthma-related ED visit. There was no clear evidence about whether clinic visits and medication costs would increase or decrease as a result of intervention; we assumed that these costs would remain unchanged.

None of the asthma-intervention studies we reviewed evaluated effects of pediatric asthma on parents' loss of working time. However, a national observational study suggested that the costs of work loss from childhood asthma are almost twice the disease's direct medical costs (2). In our cost-effectiveness analysis, we assumed that for patients destined for hospitalization and/or ED visits, savings from preventions of work loss would be equal to direct medical savings. Average work-loss costs per child with asthma were estimated at $250 in 1985 dollars, or $330 in 1995 dollars (2, 19). For patients who were not destined for hospitalization or ED visits, savings from preventions of work loss were conservatively assumed to be 25% of average work-loss costs.

The summary outcome measures were hospital days, ED visits, and work loss prevented, the net cost to the health plan (dollars invested in intervention less dollars saved from hospital days and ED visits prevented), and the net cost to society (dollars invested in intervention less dollars saved from hospital days, ED visits, and work loss prevented). Because uncertainty exists about the costs and effectiveness of interventions, sensitivity analyses were conducted by varying the assumed costs and effectiveness of different hypothetical interventions over plausible ranges.

	RESULTS

TOP ABSTRACT INTRODUCTION METHODS RESULTS DISCUSSION REFERENCES

Study Population

There were 210,125 children aged 0 to 14 yr who were Kaiser Permanente members at the selected clinics during the identification period. Of these, 16,520 had asthma-related health-services utilization, including hospitalization, ED visits, and/or medication use, and were included in the study cohort.

Table 1 shows frequencies of the predictor variables among the study cohort. The median age of the cohort was 7 yr; 10% of the cohort were younger than 2 yr. More than 80% of patients had outpatient visit copayments of $5 or less; the same was true for prescription copayments. Three percent of patients had been hospitalized and 10% had made ED visits in the 12 mo before the start of the follow-up period. Medication usage records showed that almost half of the patients in the cohort had filled asthma prescriptions written by two or more different doctors in the 6 mo before follow-up began. Approximately one-fifth of patients had filled three or more prescriptions for -agonists, and one-fifth had obtained cromolyn. Twenty-six percent had had a prescription filled for oral corticosteroids, and 10% had received inhaled corticosteroids.

Among the 16,520 patients in the cohort, 1,948 (12%) had a gap in health-plan membership during the 12-mo follow-up period, and thus had their follow-up terminated because of outmigration. There were 14,963 person-yr of follow-up. Of the 16,520 patients, 265 (1.8%) were hospitalized once or more during follow-up. There were 325 hospitalizations, or 0.022 per person-yr of follow-up. Of the cohort, 927 patients (6.4%) made ED visits during follow-up. There were 1,200 ED visits, or 0.080 per person-yr of follow-up.

Hospitalization Models

Proportional-hazards model. Preliminary models showed that variables based on information from the 6 mo before the follow-up period performed as well as or better than variables based on the 12 or 24 mo before follow-up. Table 2 shows variables that were significantly associated with future hospitalization in the final model. Having had an oral steroid prescription filled and having been hospitalized during the 6 mo before follow-up were each associated with a nearly twofold risk of hospitalization during the follow-up year. Having a personal physician listed on the computer at the start of the study period was associated with a decreased risk of hospitalization.

Classification trees. From the sets of classification trees generated, we first selected the one with the best positive predictive value. This tree (Figure 1, Tree A) was a seven-terminal node tree produced at a relative misclassification cost of 5 (i.e., misclassifying a patient who actually had hospitalization as being at low risk was counted as five times worse than misclassifying a patient without hospitalization as being at high risk). This tree was complex and had a relatively low sensitivity of 9.7% on the test set. Its positive predictive value of 19% meant that approximately five patients would be identified as being at high risk for each patient actually hospitalized.

View larger version (21K): [in this window] [in a new window]   Figure 1.   Classification Tree A for identifying children with asthma at risk of hospitalization during the subsequent 12 mo. The tree was generated by recursive partitioning of utilization data for 8,260 patients aged 0 to 14 yr with asthma-related health-services utilization at Northern California Kaiser Permanente.

We believed that clinicians might prefer a simpler tree with higher sensitivity. Thus, we simplified Tree A to produce Tree B (Figure 2). For the test set, Tree B had a sensitivity of 32%, a specificity of 94%, and a positive predictive value of 7%, meaning that it misclassified 14 patients as being at high-risk for every patient correctly classified.

View larger version (21K): [in this window] [in a new window]   Figure 2.   Classification Trees B (for hospitalization) and C (for ED visits) for identifying children with asthma at high risk of these outcomes during the subsequent 12 mo. These trees were chosen from among sets of possible alternative classification trees on the basis of their simplicity, face validity, and high sensitivity. The trees were generated by recursive partitioning of utilization data for 8,260 patients aged 0 to 14 yr with asthma-related health-services utilization at Northern California Kaiser Permanente.

Validation of Hospitalization Models

For the proportional-hazards model, the area under the ROC curve (AUC) for the test set was 0.79 (SE = 0.02), a value not significantly different from the area of 0.75 (SE = 0.02) under the ROC curve for the learning set. As can be seen in Figure 3, the classification trees had similar sensitivity and specificity to the proportional-hazards model among the highest-risk (90th percentile and above) patients.

View larger version (17K): [in this window] [in a new window]   Figure 3.   Receiver-operating-characteristic (ROC) curve analysis for prediction models for hospitalization among children with asthma at Northern California Kaiser Permanente. The line with circles is the ROC curve for the proportional-hazards model applied to the test data set. It shows the sensitivity versus 1-specificity at varying cutoff points for risk percentiles, labeled as 10, 20, 30, and so forth. The dashed line is the ROC curve for the proportional-hazards model based on the learning data set. The triangles denote the sensitivity and 1-specificity of two separate classification trees, A and B, for the test data set.

ED-visit Models

Table 3 shows the variables significantly associated with future ED visits in the final proportional-hazards model. The number of ED visits made and the number of different physicians prescribing asthma medications during the 6 mo before the follow-up period were the variables associated with the highest relative risks of a future ED visit. Tree C in Figure 2 is the classification tree that predicted ED visits with the highest sensitivity. It used only two criteria: having an ED visit during the 6 mo prior to follow-up, and having three or more different physicians prescribe asthma medications during these same 6 mo.

ROC curve analysis showed that the proportional-hazards model performed similarly in the test set (AUC = 0.69, SE = 0.01) to its performance in the learning set of observations (AUC = 0.70, SE = 0.01) (p = NS). A plot of sensitivity versus 1-specificity also showed that classification tests performed similarly to the proportional-hazards model among patients with predicted risk in the 80th percentile or higher.

Projected Cost-effectiveness

For the cost-effectiveness analysis, we used selected combinations of the predictors identified in classification Trees B and C as criteria according to which patients would be identified as being at high risk for either hospitalization or ED visits and would have educational intervention. We chose these predictors because of their simplicity, face validity, and high sensitivity. As shown in Table 4, these criteria may be used to classify from 2% to 22% of patients as being at high risk, with a sensitivity of 9% to 53%. Our cost analysis found that the 1995 average cost of a hospital day for pediatric asthma was approximately $1,300; the average cost of an ED visit was $230.

The cost-effectiveness analysis was conducted for a hypothetical cohort of 1,000 asthma patients and was based on the assumptions described in the METHODS section. With use of the most inclusive prediction criteria (prior hospitalization, ED visit, six or more -agonist units, or three or more prescribing physicians in the 6 mo before follow-up) to select patients for intervention, 3 hospital days and 27 ED visits would be prevented, at a net cost to the health plan of $46,000 and a net cost to society of $21,000. The use of less inclusive prediction criteria would reduce benefits and result in lower net costs. The absence of any prediction model, with intervention given either to all patients or to a randomly selected subset, was used as a hypothetical basis for comparison. This would increase benefits somewhat, but would result in more than a fivefold increase in net costs to both the health plan and to society by comparison with use of the most inclusive prediction criteria.

Sensitivity analyses were conducted with the assumption that a disease-management program would be implemented with use of the most inclusive prediction criteria. The presumed effectiveness of the intervention in reducing future hospital days, ED visits, and work loss was varied from 50% to 200% of baseline assumptions. At 50% of baseline effectiveness, the disease-management program would result in a net cost to the health plan of $51,000 and a net cost to society of $31,000. At twice the baseline effectiveness, the disease-management program would result in a net cost to the health plan of $36,000 per 1,000 asthma patients, but would have approximately no cost to society.

When the cost of intervention was varied from $20 to $400, with the effectiveness assumptions held constant, the disease-management program would result in net savings for society if the intervention cost $160 or less. It would result in net savings for a health plan if the intervention cost $45 or less. A separate sensitivity analysis varied the reduction in average parent work loss among patients not destined for hospitalization or ED visits from 10% to 100%. If work loss were reduced by 60% or more, net savings to society would be achieved.

	DISCUSSION

TOP ABSTRACT INTRODUCTION METHODS RESULTS DISCUSSION REFERENCES

Major Findings

The study found that prediction models that use routinely available computerized utilization data can identify children at high risk of adverse asthma outcomes. Simple models based on classification trees identified patients with two- to 10-fold greater than average risks of hospitalizations and ED visits. As compared with a lack of risk stratification, the prediction criteria were projected to greatly reduce the costs of asthma management for a population, with a proportionately smaller reduction in benefit.

Comparisons with Other Studies

Our research in this study identified new as well as previously known predictors of hospitalization and ED visits for pediatric patients with asthma. Not having a personal physician listed on the computer, and having multiple physicians prescribing asthma medications, may both reflect lack of continuity of care, although seeing multiple physicians may also reflect greater severity of illness. From a population-based perspective, these are important new variables because patients who are not consistently seeing a single clinician may not be identified as being at high-risk during clinic visits with multiple providers in a large group practice.

No single variable that we studied had both high sensitivity and high positive predictive value for identifying patients destined for adverse outcomes of pediatric asthma. The association of previous hospitalization, ED visits, and medication utilization with future adverse outcomes is in accord with findings in previous studies (5, 6). The current study differed in that its purpose was not to develop an explanatory model, but to evaluate the validity of prediction models through the use of routinely available computerized data for a broad population. The prediction models developed in our study were valid in that they worked almost equally well in the split-sample test group as in the learning group.

The cost-effectiveness model presented here suggests that it will be challenging for health-care providers actually to reduce direct medical costs through the disease management of pediatric asthma. The relatively low positive predictive value of the models means that many patients who are not actually destined for hospitalization or ED visits will be targeted for intervention, and this reduces projected cost-effectiveness. However, most health-care interventions do not actually reduce costs (20). Pediatric asthma interventions should ideally be justified on the basis of their important nonfinancial benefits, including improved symptom control and self-efficacy (21).

Limitations

These results show that in choosing among possible prediction models, there is an inherent trade-off between sensitivity and positive predictive value. In the United States today, physicians and managed-care plans have growing access to computerized data. The variables used in this study are representative of the basic data available to many health-care providers, but administrative data seem likely to have limited predictive ability. A strategy of supplementing utilization data with survey data might permit the development of better prediction models. Variables that are not routinely computerized but that may be important predictors of outcomes include race/ethnicity, socioeconomic status, functional status, and work loss (16, 22). Adding other variables, including clinician judgment of disease severity, outpatient visit diagnoses (which Northern California Kaiser Permanente began to collect subsequent to this study), and spirometry results might also result in models with better performance (5, 6).

Our study population did not include children who had only one -agonist prescription with no other asthma medications and no prior diagnosis of asthma from a hospitalization or ED visit. To understand how this might have affected our results, we conducted a supplemental analysis that identified 13,574 children who met such criteria in 1995. The hospitalization rate among these children was 0.3%. In the classification-tree models, these children would always have fallen into the low-risk groups. Thus, including them would have slightly decreased the sensitivity but would not have affected the positive predictive value of the models.

Like all prediction models, the models developed in our study need to be validated in settings outside the one in which they were developed. The population we studied is diverse and representative of the general population (26). In this study, the proportions of children receiving inhaled corticosteroids (9%) and cromolyn (20%) were similar to those in a Boston HMO (11% and 16%, respectively) (6). However, the health-service utilization patterns of patients in other settings or with less comprehensive health insurance could differ from those in this study.

The cost-effectiveness model described here is hypothetical and is based on estimates from published studies. However, as better empirical data become available, the model can easily be adjusted to evaluate the cost-effectiveness of other prediction models and interventions, such as case management, referrals to specialists, and clinician education.

Policy Implications

Prediction models such as those developed in this study should be useful in clinical population management, particularly because they identify some high-risk patients (e.g., those who have no physician or who use multiple physicians) who may not yet have come to a clinician's attention. In Northern California Kaiser Permanente, each local pediatric group has an "asthma champion" who periodically receives a regionally produced list of high-risk children with asthma. Such lists are used in various ways: some physician groups have the patients' personal physicians review the lists for any needed changes in management, whereas others also conduct a routine intervention for all high-risk patients, such as a clinic evaluation, letter, or telephone call.

We conclude that prediction models that use computerized data can identify asthmatic children at high risk for future hospitalization and ED visits, and may draw attention to some high-risk patients who have not been found by clinicians. Models such as these may help resources to be focused on the highest-risk patients in population-based efforts to improve asthma management.