Discriminatory accuracy of a random effect in multilevel logistic regression
Multilevel logistic regression analyses are applied in a variety of different scenarios, not least for the study of geographic differences in health. Oftentimes these studies are intended as a basis for public health interventions and/or individual lifestyle recommendations. However, results from these multilevel analyses are not easily translated into measures that inform on the potential effectiveness of future interventions, or on the accuracy of individual health predictions, such as sensitivity, specificity and measures derived therefrom.
The project aim is to develop a method for translating multilevel analyses results into measures of discriminatory accuracy, such as the area under the receiver operating characteristics curve (AUC-ROC). The method will help elucidate the usefulness of random second level factors, such as geographical areas, schools, hospitals etc, as targets of public interventions and/or as predictors of individual health outcomes.
METHODS:
We examine the distributions of the random effects among cases and non-cases in different scenarios by means of theoretical investigation and simulation. The distributions are subsequently used to produce ROC curves and corresponding AUC-ROC measures. We also illustrate a mathematical relation between the AUC-ROC and ICC of the random effect under certain conditions. Results are applied to empirical data on hospital differences in mortality following breast cancer diagnoses (BCD) in Sweden 2005 – 2012.
RESULTS:
We found that the ICC in multilevel logistic regression can be directly translated into a ROC and AUC-ROC measure. In the case of mortality after a BCD in Sweden, the hospital level appeared to be a potential target for intervention as the AUC was 0.75 (ICC = 0.30). However, the predictive ability of treating hospital, as measured by the AUC, seemed to be connected to the distribution of women with prior breast cancer among hospitals. Correcting for this factor attenuated the predictive ability of the treating hospital (AUC =0.65).
CONCLUSIONS:
Random effect variance in multilevel logistic regression can easily be translated into measures of discriminatory. This provides a nice alternative to the median odds ratio and ICC used in social epidemiology when communicating the importance of the factor. Concerning 5-year mortality after BCD in Sweden, we initially found treating hospital to be potentially important in predicting individual outcomes. However, this was partly due to some hospitals having more women with recurrent cancers and severe disease than others. Therefore, hospitals may not be a good target for intervention when the aim is to improve 5-year mortality following BCD in Sweden.