Reliable and Computationally Efficient Maximum-Likelihood Estimation of “Proper” Binormal ROC Curves1

Rationale and Objectives

Estimation of ROC curves and their associated indices from experimental data can be problematic, especially in multireader, multicase (MRMC) observer studies. Wilcoxon estimates of area under the curve (AUC) can be strongly biased with categorical data, whereas the conventional binormal ROC curve-fitting model may produce unrealistic fits. The “proper” binormal model (PBM) was introduced by Metz and Pan to provide acceptable fits for both sturdy and problematic datasets, but other investigators found that its first software implementation was numerically unstable in some situations. Therefore, we created an entirely new algorithm to implement the PBM.

Materials and Methods

This paper describes in detail the new PBM curve-fitting algorithm, which was designed to perform successfully in all problematic situations encountered previously. Extensive testing was conducted also on a broad variety of simulated and real datasets. Windows, Linux, and Apple Macintosh OS X versions of the algorithm are available online at http://xray.bsd.uchicago.edu/krl/.

Results

Plots of fitted curves as well as summaries of AUC estimates and their standard errors are reported. The new algorithm never failed to converge and produced good fits for all of the several million datasets on which it was tested. For all but the most problematic datasets, the algorithm also produced very good estimates of AUC standard error. The AUC estimates compared well with Wilcoxon estimates for continuously distributed data and are expected to be superior for categorical data.

Conclusion

This implementation of the PBM is reliable in a wide variety of ROC curve-fitting tasks.

Key Words: Receiver operating characteristic (ROC) analysis, receiver operating characteristic (ROC) curves, proper binormal model, maximum likelihood estimation (MLE), multireader, multicase (MRMC) analysis