Rationale and Objectives
In diagnostic accuracy studies, cases in which a reader does not see the condition
of interest are often given the same score for ROC analysis (e.g. confidence score
of 0%). However, many of these cases can be further distinguished and doing so may
result in more robust ROC results.
Materials and Methods
We examined two recent, real-world studies which used different procedures to encourage
readers to further distinguish subjects who appear to be without the condition of
interest. For each study, we calculated the results under two conditions. In the “zeros
distinguished” (ZD) condition, we incorporated the confidence scores collected to
further distinguish the normal-looking subjects. In the “zeros not distinguished”
(ZND) condition, we disregarded these scores and simply gave the unit of analysis
a score of zero whenever the reader did not identify the condition of interest in
that unit. We compared the two conditions on (1) coverage of the ROC space and (2)
discrepancy between parametric and nonparametric results.
Results
Compared to the ZND condition, coverage of the ROC space was improved in the ZD condition
for all ROC curves in both studies. In the first study, there was a significant reduction
in the discrepancy between parametric and nonparametric results (median discrepancy
in ZND vs ZD condition: 0.033 vs 0.011, p = 0.012). A similar reduction was not seen in the second study, though the discrepancies
were very low in both conditions (0.003 vs 0.006, p = 0.313).
Conclusion
Prompting readers to further distinguish cases in which they do not see the condition
of interest may result in more robust ROC results, though further exploration of this
topic is warranted.
Keywords
To read this article in full you will need to make a payment
Purchase one-time access:
Academic & Personal: 24 hour online accessCorporate R&D Professionals: 24 hour online accessOne-time access price info
- For academic or personal research use, select 'Academic and Personal'
- For corporate R&D use, select 'Corporate R&D Professionals'
Subscribe:
Subscribe to Academic RadiologyAlready a print subscriber? Claim online access
Already an online subscriber? Sign in
Register: Create an account
Institutional Access: Sign in to ScienceDirect
REFERENCES
- BI-RADS data should not be used to estimate ROC curves.Radiology. 2010; 256: 29-31https://doi.org/10.1148/radiol.10091394
- Statistical Methods in Diagnostic Medicine.2nd edition. John Wiley & Sons, Hoboken, NJ2011
- Exploration of analysis methods for diagnostic imaging tests: problems with ROC AUC and confidence scores in CT colonography.PLOS ONE. 2014; 9e107633https://doi.org/10.1371/journal.pone.0107633
- Comparing areas under receiver operating characteristic curves: potential impact of the "Last" experimentally measured operating point.Radiology. 2008; 247: 12-15https://doi.org/10.1148/radiol.2471071321
- ROC analysis: web-based calculator for ROC curves.Johns Hopkins University, Baltimore2021 (Accessed January 22)
- Nonparametric analysis of clustered ROC curve data.Biometrics. 1997; 53: 567-578https://doi.org/10.2307/2533958
R Core Team (2020). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. URL https://www.R-project.org/.
Article Info
Publication History
Published online: August 05, 2021
Accepted:
June 18,
2021
Received in revised form:
June 16,
2021
Received:
March 16,
2021
Identification
Copyright
© 2021 The Association of University Radiologists. Published by Elsevier Inc. All rights reserved.
