Improving the Robustness of Diagnostic Accuracy Results By Asking Study Readers to Further Distinguish Subjects Who Appear to be Without the Condition Of Interest

Published:August 05, 2021DOI:

      Rationale and Objectives

      In diagnostic accuracy studies, cases in which a reader does not see the condition of interest are often given the same score for ROC analysis (e.g. confidence score of 0%). However, many of these cases can be further distinguished and doing so may result in more robust ROC results.

      Materials and Methods

      We examined two recent, real-world studies which used different procedures to encourage readers to further distinguish subjects who appear to be without the condition of interest. For each study, we calculated the results under two conditions. In the “zeros distinguished” (ZD) condition, we incorporated the confidence scores collected to further distinguish the normal-looking subjects. In the “zeros not distinguished” (ZND) condition, we disregarded these scores and simply gave the unit of analysis a score of zero whenever the reader did not identify the condition of interest in that unit. We compared the two conditions on (1) coverage of the ROC space and (2) discrepancy between parametric and nonparametric results.


      Compared to the ZND condition, coverage of the ROC space was improved in the ZD condition for all ROC curves in both studies. In the first study, there was a significant reduction in the discrepancy between parametric and nonparametric results (median discrepancy in ZND vs ZD condition: 0.033 vs 0.011, p = 0.012). A similar reduction was not seen in the second study, though the discrepancies were very low in both conditions (0.003 vs 0.006, p = 0.313).


      Prompting readers to further distinguish cases in which they do not see the condition of interest may result in more robust ROC results, though further exploration of this topic is warranted.


      To read this article in full you will need to make a payment

      Purchase one-time access:

      Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online access
      One-time access price info
      • For academic or personal research use, select 'Academic and Personal'
      • For corporate R&D use, select 'Corporate R&D Professionals'


      Subscribe to Academic Radiology
      Already a print subscriber? Claim online access
      Already an online subscriber? Sign in
      Institutional Access: Sign in to ScienceDirect


        • Jiang Y
        • Metz CE.
        BI-RADS data should not be used to estimate ROC curves.
        Radiology. 2010; 256: 29-31
        • Zhou XH
        • Obuchowski NA
        • McClish DK.
        Statistical Methods in Diagnostic Medicine.
        2nd edition. John Wiley & Sons, Hoboken, NJ2011
        • Mallet S
        • Halligan S
        • Collins GS
        • et al.
        Exploration of analysis methods for diagnostic imaging tests: problems with ROC AUC and confidence scores in CT colonography.
        PLOS ONE. 2014; 9e107633
        • Gur D
        • Bandos AI
        • Rockette HE.
        Comparing areas under receiver operating characteristic curves: potential impact of the "Last" experimentally measured operating point.
        Radiology. 2008; 247: 12-15
        • Eng J.
        ROC analysis: web-based calculator for ROC curves.
        Johns Hopkins University, Baltimore2021 (Accessed January 22)
        • Obuchowski NA.
        Nonparametric analysis of clustered ROC curve data.
        Biometrics. 1997; 53: 567-578
      1. R Core Team (2020). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. URL