Effects of incorrect computer-aided detection (CAD) output on human decision-making in mammography

Rationale and objectives

To investigate the effects of incorrect computer output on the reliability of the decisions of human users. This work followed an independent UK clinical trial that evaluated the impact of computer-aided detection (CAD) in breast screening. The aim was to use data from this trial to feed into probabilistic models (similar to those used in “reliability engineering”) which would detect and assess possible ways of improving the human–CAD interaction. Some analyses required extra data; therefore, two supplementary studies were conducted. Study 1 was designed to elucidate the effects of computer failure on human performance. Study 2 was conducted to clarify unexpected findings from Study 1.

Materials and methods

In Study 1, 20 film readers viewed 60 sets of mammograms (30 of which contained cancer) and provided “recall/no recall” decisions for each case. Computer output for each case was available to the participants. The test set was designed to contain an unusually large proportion (50%) of cancers for which CAD had generated incorrect output. In Study 2, 19 different readers viewed the same set of cases in similar conditions except that computer output was not available.

Results

The average sensitivity of readers in Study 1 (with CAD) was significantly lower than the average sensitivity of readers in Study 2 (without CAD). The difference was most marked for cancers for which CAD failed to provide correct prompting.

Conclusion

Possible automation bias effects in CAD use deserve further study because they may degrade human decision-making for some categories of cases under certain conditions. This possibility should be taken into account in the assessment and design of CAD tools.

Key words:  Computer-aided detection (CAD) , mammography , evaluation , reliability