Advertisement

Chest X-ray Interpretation by Radiographers Is Not Inferior to Radiologists

A Multireader, Multicase Comparison Using JAFROC (Jack-knife Alternative Free-response Receiver Operating Characteristics) Analysis
Open AccessPublished:April 30, 2018DOI:https://doi.org/10.1016/j.acra.2018.03.026

      Highlights

      • Chest X-ray interpretation by reporting radiographers is noninferior to radiologists.
      • Chest X-ray reporting by reporting radiographers can increase diagnostic capacity.
      • Maximizing the use of reporting radiographers could streamline patient pathways.

      Rationale and Objectives

      Chest X-rays (CXR) are one of the most frequently requested imaging examinations and are fundamental to many patient pathways. The aim of this study was to investigate the diagnostic accuracy of CXR interpretation by reporting radiographers (technologists).

      Methods

      A cohort of consultant radiologists (n = 10) and reporting radiographers (technologists; n = 11) interpreted a bank (n = 106) of adult CXRs that contained a range of pathologies. Jack-knife alternate free-response receiver operating characteristic (JAFROC) methodology was used to determine the performance of the observers (JAFROC v4.2). A noninferiority approach was used, with a predefined margin of clinical insignificance of 10% of average consultant radiologist diagnostic accuracy.

      Results

      The diagnostic accuracy of the reporting radiographers (figure of merit = 0.828, 95% confidence interval 0.808–0.847) was noninferior to the consultant radiologists (figure of merit = 0.788, 95% confidence interval 0.766–0.811), P < .0001.

      Conclusions

      With appropriate postgraduate education, reporting radiographers are able to interpret CXRs at a level comparable to consultant radiologists.

      Key Words

      Introduction

      Clinical imaging plays a central role in health care and is viewed by clinicians as an essential tool to support them in diagnostic and therapeutic decision-making. New and emerging technologies, coupled with a growing and aging population and increasing treatment options, have seen the demand for clinical imaging outstrip capacity within the United Kingdom (UK) (
      • Royal College of Radiologists
      Clinical radiology UK workforce report 2011.
      ,
      • Royal College of Radiologists
      Patients waiting too long for test results.
      ,
      • Royal College of Radiologists
      Clinical radiology UK workforce census 2015 report.
      ). The situation is not unique to the UK, with service delivery challenges due to sustained increases in imaging echoed worldwide (
      • Canadian Institute for Health Information
      Health care in Canada: a focus on wait times.
      ,
      • Queensland Department of Health
      Inquiry into Gold Coast X-ray reporting.
      ). Training radiographers, the UK equivalent of radiation technologists, to undertake clinical reporting has addressed this situation to some extent (
      • Royal College of Radiologists
      Clinical radiology UK workforce census 2015 report.
      ,
      • Woznitza N.
      • Piper K.
      • Rowe S.
      • et al.
      Optimizing patient care in radiology through team-working: a case study from the United Kingdom.
      ,
      • Snaith B.
      • Milner R.C.
      • Harris M.A.
      Beyond image interpretation: capturing the impact of radiographer advanced practice through activity diaries.
      ,
      • Beardmore C.
      • Woznitza N.
      • Goodman S.
      The radiography workforce current challenges and changing needs.
      ).
      Radiographer reporting, particularly skeletal reporting, has become widespread across the UK (
      • Royal College of Radiologists
      Clinical radiology UK workforce census 2015 report.
      ,
      • Snaith B.
      • Hardy M.
      • Lewis E.F.
      Radiographer reporting in the UK: a longitudinal analysis.
      ), and in many departments, it provides a significant contribution to reporting capacity (
      • Woznitza N.
      • Piper K.
      • Rowe S.
      • et al.
      Optimizing patient care in radiology through team-working: a case study from the United Kingdom.
      ,
      • Snaith B.
      • Milner R.C.
      • Harris M.A.
      Beyond image interpretation: capturing the impact of radiographer advanced practice through activity diaries.
      ). Scandinavia has recently introduced skeletal reporting radiographers (
      • Buskov L.
      • Abild A.
      • Christensen A.
      • et al.
      Radiographers and trainee radiologists reporting accident radiographs: a comparative plain film-reading performance study.
      ). Chest x-ray (CXR) interpretation by radiographers is also not a new concept, with early work conducted as part of the lung cancer screening programs exploring this in the 1970s (
      • Sheft D.J.
      • Jones M.D.
      • Brown R.F.
      • et al.
      Screening of chest roentgenograms by advanced roentgen technologists.
      ,
      • Flehinger B.J.
      • Melamed M.R.
      • Heelan R.T.
      • et al.
      Accuracy of chest film screening by technologists in the New York early lung cancer detection program.
      ). Promising results have been reported in two studies that have explored the accuracy of radiographer reporting of CXRs: one in an academic setting (
      • Piper K.
      • Cox S.
      • Paterson A.
      • et al.
      Chest reporting by radiographers: findings of an accredited postgraduate programme.
      ) and the other in clinical practice (
      • Woznitza N.
      • Piper K.
      • Burke S.
      • et al.
      Adult chest radiograph reporting by radiographers: preliminary data from an in-house audit programme.
      ). Regions with socioeconomic challenges and limited radiologist resource could benefit from adopting radiographer reporting, for example, chest radiography in tuberculosis detection (
      • Ben Shimol S.
      • Dagan R.
      • Givon-Lavi N.
      • et al.
      Evaluation of the World Health Organization criteria for chest radiographs for pneumonia diagnosis in children.
      ,
      • Ostensen H.
      Diagnostic imaging: what is it? When and how to use it where resources are limited?.
      ). No study appears evident in the literature that directly compared the performance of consultant radiologists and reporting radiographers when interpreting CXRs.
      Jack-knife alternate free-response receiver operating characteristic curve (JAFROC) methodology is able to compare CXRs when each is almost unique in terms of number, position, and type of lesions (
      • Chakraborty D.P.
      Recent advances in observer performance methodology: Jackknife free-response ROC (JAFROC).
      ). In line with recent guidance on radiographer reporting, reporting radiographers must perform at a level comparable to consultant radiologists (
      • College of Radiographers
      Preliminary clinical evaluation and clinical reporting by radiographers: policy and practice guidance.
      ,
      • Royal College of Radiologists
      • Society and College of Radiographers
      Team working in clinical imaging.
      ). This study therefore compared the diagnostic accuracy of reporting radiographers and consultant radiologists when interpreting a bank of adult CXRs.

      Methods

      Ethical approval was obtained from City Road and Hampstead Research Ethics Committee before commencement of the study.

      Study Design

      A prospective quasi-experimental assessment of diagnostic accuracy for multiple readers and multiple cases was employed. A noninferiority approach was used to compare radiologists and radiographers.

      Setting

      The study was conducted in a district general hospital with academic links.

      Case Selection

      A consecutive series of adult CXRs (single posterior-anterior or anteroposterior projections, including mobile examinations) performed for clinical reasons at Homerton University Hospital between April 1, 2011, and March 30, 2012, were selected. Lateral CXRs are not routinely performed at the study site. Inclusion criteria for the study were adult patients (>16 years), with a CXR performed when the patient was referred by a hospital-based clinician (emergency department, inpatient including intensive care unit, outpatient); CXRs referred from general practice (primary care) were excluded. The image bank was stratified for a normal to abnormal ratio of 1:1 and for a disease category (infection:cardiac:malignancy:other, ratio of 3:3:1:3) according to the interpretation of two independent expert thoracic radiologists. Detailed information on the construction of the reference standard for diagnosis has been published (
      • Woznitza N.
      • Piper K.
      • Burke S.
      • et al.
      Agreement between expert thoracic radiologists and the chest radiograph reports provided by consultant radiologists and reporting radiographers in clinical practice: review of a single clinical site.
      ). A balanced design, with an equal proportion of normal and abnormal cases, was selected, as it is the most efficient for diagnostic accuracy studies. Such a design permits the detection of small differences in observer performance relative to the number of cases and observers (
      • Obuchowski N.A.
      Sample size tables for receiver operating characteristic studies.
      ). The proportion of cases within each disease category was based on audit data of the most frequent discharge diagnoses associated with a CXR performed at the study site that correlated with the national dataset (
      • The Health and Social Care Information Centre
      Hospital episode statistics for England.
      ). A proportion (n = 18, 17%) of subtle normal CXRs, that is, cases that were initially incorrectly interpreted as abnormal in clinical practice (false-positive clinical practice n = 3; 17%) and difficult abnormal cases (abnormalities missed at initial clinical report) were included for each disease category (false negative n = 11, 61%). These were drawn from the monthly radiology department discrepancy meeting and teaching files. The disease category for subtle normal and abnormal cases was n = 7, 39% infection; n = 4, 22% cardiac; n = 3, 17% malignancy; n = 1, 5% other. Inclusion of subtle cases ensured that a sufficient range of true-positive and false-positive responses were generated by the participants and ensured that the data were appropriate for analysis (
      • Chakraborty D.
      The FROC, AFROC and DROC variants of the ROC analysis.
      ,
      • Chakraborty D.
      Recent developments in FROC methodology.
      ).

      Participants

      Appropriately qualified professionals interpreted the images. Consultant radiologists were staff that held a consultant post within the National Health Service and specialist registration in clinical radiology with the General Medical Council. Reporting radiographers were consultant or advanced practitioner radiographers that held registration with the Health and Care Professions Council and Masters level qualification (MSc or postgraduate diploma or postgraduate certificate) in adult CXR interpretation accredited by the College of Radiographers (UK). Reporting radiographer participants had completed accredited education from different higher education institutes. An example of the course content and structure of postgraduate CXR interpretation training has been published previously (
      • Piper K.
      • Cox S.
      • Paterson A.
      • et al.
      Chest reporting by radiographers: findings of an accredited postgraduate programme.
      ). Participants (expert thoracic consultant radiologists, consultant radiologists [nonthoracic radiologist], reporting radiographers, and arbiters) were voluntarily recruited through convenience and snowball sampling (where participants identify other potential recruits) (
      • Sadler G.R.
      • Lee H.C.
      • Lim R.S.
      • et al.
      Recruitment of hard-to-reach population subgroups via adaptations of the snowball sampling strategy.
      ). To allow for participant attrition, 11 participants from each professional group (radiologist or radiographer) were recruited. Number of years of practice and annual volume of CXR interpreted were collected from each participant.

      Reference Standard Diagnosis

      The CXRs were independently interpreted by two expert thoracic consultant radiologists, blinded to the clinical report provided at the time of examination (
      • Woznitza N.
      • Piper K.
      • Burke S.
      • et al.
      Agreement between expert thoracic radiologists and the chest radiograph reports provided by consultant radiologists and reporting radiographers in clinical practice: review of a single clinical site.
      ). The expert thoracic consultant radiologists had access to all pertinent imaging including previous CXRs, follow-up x-rays, and cross-sectional imaging (eg, computed tomography [CT]) where available. They were provided with the patient demographics (age, gender), referral source, and the clinical history provided by the referring clinician at the time of the request. For this study, the acceptance radius was predefined zonal criteria, chosen to reflect the system used in clinical practice (
      • de Lacey G.
      • Morley S.
      • Berman L.
      Chest x-ray: a survival guide.
      ).
      Histology was traditionally used as the reference standard diagnosis for CXR diagnostic accuracy studies, especially for studies that have investigated observer performance in lung nodule detection. Increasingly, CT is used as the reference standard diagnosis, and the combination of independent expert review with follow-up imaging including CT allowed for inclusion of a range of pathologies.

      Test Methods

      All image interpretation occurred under normal viewing conditions. Digital Imaging and Communications in Medicine (DICOM) images were viewed on paired high-resolution Picture Archive and Communication System (PACS) workstations in observer controlled lighting conditions free from distractions and background noise.
      A bank (n = 106) of CXRs was given to the participant reporting radiographers (n = 11) and consultant radiologists (n = 10), blinded to the proportion of normal and abnormal cases. One radiologist withdrew from the study before data collection. Each participant provided independent interpretations for each case, blinded to both the clinical report and the reference standard diagnosis. Report data for each case were entered into a proforma, and participants were required to localize and characterize all abnormalities and assign a confidence score (1—uncertain to 4—definitely abnormal) to form a mark-rating pair for all identified lesions, for example, “A 2 cm right upper zone nodule is suspicious for a malignant lesion (3).” Participants were not able to modify or edit reports or data once provided and did not discuss any aspect of the study with other participants. The participants had access to previous CXRs but not any other imaging investigation. They were provided with patient demographics (age, gender), referral source (emergency department, outpatient, inpatient), and the clinical history provided by the referring clinician at the time of the request. A list of incidental findings to be considered normal and examples of abnormal findings to be considered as significant was available at the time of interpretation (Table 1). The list of abnormal findings was not intended to be exhaustive but rather a guide for the observers. Localization of any abnormalities was performed using predefined zonal criteria and the same acceptance radius (maximum distance from localization to lesion true-positive rating) as the reference standard diagnosis (
      • Haygood T.M.
      • Ryan J.
      • Brennan P.C.
      • et al.
      On the choice of acceptance radius in free-response observer performance studies.
      ). A diagram was included with the proforma (Appendix S1). In particular, “hilar” was related specifically to hilar structures and “perihilar” to the lung parenchyma immediately adjacent to the hilum.
      TABLE 1Reporting Guidance Adapted From Robinson (1999)
      • Robinson P.
      • Wilson D.
      • Coral A.
      • et al.
      Variation between experienced observers in the interpretation of accident and emergency radiographs.
      Findings To Be Considered NormalFindings To Be Considered Abnormal
      • Congenital/anatomical variants
      • Small calcified foci
      • Old fractures
      • Previous surgery
      • Hiatus hernia
      • Pleural fluid
      • Pneumothorax
      • Consolidation
      • Noncalcified nodules
      • Cardiac enlargement
      • Mediastinal widening
      • Recent fracture
      • Foreign body
      All abnormal cases included in the study were reviewed by a professor of respiratory medicine and a consultant thoracic radiologist with a comprehensive case note summary and all imaging (previous and follow up) available to assign weights (from 0 to 1, with 0 being noncontributory [or normal/normal variant] to the diagnosis and 1 being the most [sole] significant lesion pointing to the diagnosis) to each abnormality (lesion) on abnormal cases. Independent thoracic radiologists, acting as the reference standard, were required to identify, localize, and characterize all abnormalities. Weights were assigned to each lesion identified in the reference standard diagnosis. When multiple abnormalities were contained on a single case according to their impact on patient management (for a total of 1.0 for each image), based on a consensus decision between the professor of medicine and an expert thoracic radiologist. Normal cases did not require weights to be assigned and therefore did not require additional review.

      Analysis

      Sensitivity was calculated at lesion level (number of lesion localizations [LLs]) and specificity determined at a case level; the number of true-negative cases identified, in line with previous work (
      • Littlefair S.
      • Mello-Thoms C.
      • Reed W.
      • et al.
      Increasing prevalence expectation in thoracic radiology leads to overcall.
      ). Sensitivity and specificity were determined for each practitioner and for both professional groups (radiologists and radiographers), and 95% confidence intervals were calculated.
      Analysis of the data was performed using the jack-knife method and following best practice guidance (
      • Haygood T.M.
      • Ryan J.
      • Brennan P.C.
      • et al.
      On the choice of acceptance radius in free-response observer performance studies.
      ,
      • Chakraborty D.P.
      How to conduct a free-response study 2005.
      ,
      • Chakraborty D.P.
      A status report on free-response analysis.
      ). Mark-rating pair data were extracted for each case and compared to the reference standard diagnosis independently by two arbiters for true-positive (LL) and false-positive (non–lesion localization [NLL]) decisions. Pseudo-values were calculated using JAFROC software (Chakraborty, version 4.2, August 2014) (
      • Chakraborty D.P.
      JAFROC [program]. 4.2 version.
      ) and analyzed using SPSS (version 21; IBM, Armonk, NY).
      A free-response paradigm was used to permit the inclusion of CXRs with more than one area of pathology and using the indices of LL fraction (number of lesions noted against total lesions for each CXR) and NLL fraction (number of false positives divided by total number of false positives reported by the cohort). Each lesion had been assigned a weight according to its clinical impact.
      Alternate free-response receiver operating characteristic studies require participants to locate abnormalities on the images and to assign a relative confidence score to each abnormality (mark-rating pair), accurately reflecting diagnostic decisions made in clinical practice. The reference standard diagnosis was applied to all mark-rating pairs for each practitioner using the acceptance radius to determine LLs (true positive) and NLLs (false positive).
      JAFROC software (version 4.2, http://www.devchakraborty.com/index.php) (
      • Chakraborty D.P.
      JAFROC [program]. 4.2 version.
      ) was used to calculate the figure of merit (FoM) for individual participants and for the average performance of the cohort of consultant radiologists and the cohort of reporting radiographers. Further, 95% confidence intervals were calculated assuming a normal distribution. To simulate clinical practice as closely as possible, images with more than one abnormality had each lesion weighted for clinical importance. The primary analysis used in this study to determine the diagnostic accuracy of consultant radiologist and reporting radiographer CXR reporting was weighted JAFROC FoM. Unweighted FoM was calculated for comparison.
      A noninferiority approach was used, that is, the study was designed to investigate if the reporting radiographers' accuracy was no worse than the consultant radiologists. Robinson et al. estimated variability of 11%–19% between consultant radiologists when interpreting CXRs (
      • Robinson P.
      • Wilson D.
      • Coral A.
      • et al.
      Variation between experienced observers in the interpretation of accident and emergency radiographs.
      ). Guidance from relevant professional bodies (joint College of Radiographers and Royal College of Radiologists) reinforces the need for equivalence (
      • Royal College of Radiologists
      • Society and College of Radiographers
      Team working in clinical imaging.
      ). A theoretical approach defining acceptable variation between observers by values outside 2 standard deviations would suggest that 10% is acceptable for noninferiority. Similarly, for skewed distributions, 95% confidence intervals are acceptable, so again 10% would be considered outliers. An interobserver variability of less than 10% in CXR interpretation was considered to be acceptable in clinical practice. On these grounds, a 10% difference in diagnostic accuracy was used as the predefined margin of noninferiority in the study.
      Using the noninferiority approach, the null hypothesis that the FoM for the consultant radiologists exceeded the predefined clinical significance level (10% of average consultant radiologist) would be rejected if the P value was less than .05 using a one-tailed test, in line with CONSORT recommendations (
      • Tourassi G.
      Receiver operating characteristic analysis: basic concepts and practical applications.
      ). To detect an effect size (difference between consultant radiologists and reporting radiographers) of 0.08 in the FoM, with 10 observers in each group and a normal/abnormal CXR ratio of 1:1, 105 cases were required for the test bank, accepting a 20% chance of a type II error (incorrect acceptance of null hypothesis) and a 5% possibility of a type I error (incorrect rejection of null hypothesis) (
      • Chakraborty D.P.
      Recent advances in observer performance methodology: Jackknife free-response ROC (JAFROC).
      ,
      • Obuchowski N.A.
      Sample size tables for receiver operating characteristic studies.
      ). This powered the study to detect a small difference between highly accurate observers, if one existed. A sample size of 106 was used to allow a 1:1 ratio of normal to abnormal cases. One-way analysis of variance was used to compare within and between practitioner group variation.

      Results

      The image bank contained 106 cases, with an equal proportion of normal and abnormal cases (53 each). The number of lesions included in the 53 abnormal cases ranged from 1 to 6, with an average of 2.28 lesions. A total of 121 lesions were included in the image bank.
      Ten consultant radiologists and 11 reporting radiographers consented to participate in the study and completed the image bank. Participant demographics are presented in Table 2, with data from one practitioner missing. The 21 participants should therefore have generated a total of 2226 CXR reports during the data collection. A small number of cases (13/2226) were not reported by radiologists (5/1060) and radiographers (8/1166). There was no pattern to the cases missed, and these were attributed to participant oversight. A total of 2213 reports were available for analysis; 1055 (48%) by consultant radiologist and 1158 (52%) by reporting radiographer.
      TABLE 2Participant Experience and Volume of Chest X-rays Reported Annually
      Experience (y)Consultant RadiologistsReporting Radiographers
      VolumeVolume
      <5,0005,001–9,999≥10,000<5,0005,001–9,999≥10,000
      0–52N/AN/A141
      6–94N/AN/A121
      ≥1022N/AN/AN/AN/A
      N/A, not applicable (data from one reporting radiographer missing).
      The overall sensitivity and specificity of the consultant radiologists and reporting radiographers are presented in Figure 1 and were broadly comparable. The JAFROC performance, in terms of FoM values, of the consultant radiologists and reporting radiographers are presented in Table 3, Figure 2, Figure 3. The diagnostic accuracy of the cohort of reporting radiographers was noninferior to that of the consultant radiologists for both unweighted JAFROC (t = 11.826, P < .0001) and weighted JAFROC (t = 12.654, P < .0001) analyses. Between-group variance (mean square 0.865) was significantly higher than within-group variance (mean square 0.126; F = 6.85, P = .009).
      Figure 1
      Figure 1Sensitivity and specificity of consultant radiologists and reporting radiographers (with 95% confidence intervals).
      TABLE 3Diagnostic Accuracy Figure of Merit Values of Consultant Radiologists and Reporting Radiographers
      Reporting PractitionerNumber of CasesFigure of Merit (95% CI)
      UnweightedWeighted
      Consultant radiologist10550.788 (0.766–0.811)0.786 (0.764–0.808)
      Reporting radiographer11580.828 (0.808–0.847)0.830 (0.811–0.849)
      CI, confidence interval.
      Figure 2
      Figure 2Unweighted JAFROC (jack-knife alternate free-response receiver operating characteristic) curves for consultant radiologists and reporting radiographers. CR, consultant radiologist; RR, reporting radiographer.
      Figure 3
      Figure 3Weighted JAFROC (jack-knife alternate free-response receiver operating characteristic) curves for consultant radiologists and reporting radiographers. CR, consultant radiologist; RR, reporting radiographer.
      The diagnostic accuracy of consultant radiologists and reporting radiographers, stratified by both experience and number of CXRs interpreted annually, is presented in Table 4. There was no apparent difference in diagnostic accuracy between the consultant radiologists and reporting radiographers for different current annual workload and experience (Table 4). The small number of participants in each subset prevented further statistical analysis.
      TABLE 4Diagnostic Accuracy of Practitioners According to Experience and Volume of Chest X-rays Interpreted
      Consultant RadiologistsReporting Radiographers
      VolumeVolume
      Experience (y)<5,0005,001–9,999≥10,000<5,0005,001–9,999≥10,000
      0–50.809N/AN/A0.8390.8390.803
      6–90.760N/AN/A0.8240.8440.822
      ≥100.7870.813N/AN/AN/AN/A
      N/A, not applicable.

      Discussion

      Both unweighted and weighted JAFROC FoM values showed no significant difference between radiographers and radiologists.

      Strengths and Limitations

      Sources of potential bias (image bank case selection, reference standard, observer selection, and measurements) were identified and minimized for each element of the diagnostic accuracy study. The cases were selected for the range of pathologies and included subtle radiography details. The reference standard was constructed according to these criteria. With the exception of a small number of cases that were not interpreted by some practitioners due to oversight (consultant radiologists n = 5, <1%; reporting radiographers n = 8, <1%), all reporting practitioners interpreted an identical image bank. Observers were selected opportunistically, but their years of experience and reporting histories were carefully documented (Table 4). Measurements were standardized but then also weighted for their clinical significance.
      The use of two independent, expert consultant thoracic radiologists produced a valid reference standard diagnosis for all cases included in the study, and was applied to all participant observer interpretations independently by two arbiters. The reference standard diagnosis used in the diagnostic accuracy study was the consensus findings of two independent expert thoracic radiologists who had access to follow-up imaging. Histology is frequently used as the reference standard diagnosis. However, where a biopsy is not taken and radiological follow-up is important, CT may be used as the reference standard (
      • Baldwin D.
      Pulmonary nodules again? The 2015 British Thoracic Society guidelines on the investigation and management of pulmonary nodules.
      ).
      Marked variation between observers can exist when interpreting CXRs and there is limited current evidence that examines the accuracy of reporting radiographer CXR interpretation. The majority of previous studies have concentrated on a single task, for example, lung nodule detection (
      • Donovan T.
      • Litchfield D.
      Looking for cancer: expertise related differences in searching and decision making.
      ,
      • Litchfield D.
      • Ball L.J.
      • Donovan T.
      • et al.
      Viewing another person's eye movements improves identification of pulmonary nodules in chest x-ray inspection.
      ,
      • Manning D.
      • Barker-Mill S.C.
      • Donovan T.
      • et al.
      Time-dependent observer errors in pulmonary nodule detection.
      ,
      • Manning D.
      • Ethell S.
      • Donovan T.
      • et al.
      How do radiologists do it? The influence of experience and training on searching for chest nodules.
      ) or lung cancer screening (
      • Sheft D.J.
      • Jones M.D.
      • Brown R.F.
      • et al.
      Screening of chest roentgenograms by advanced roentgen technologists.
      ,
      • Flehinger B.J.
      • Melamed M.R.
      • Heelan R.T.
      • et al.
      Accuracy of chest film screening by technologists in the New York early lung cancer detection program.
      ), unlike the current study that included a range of pathologies. Other work has assessed the performance of radiographer abnormality detection (
      • Sonnex E.P.
      • Tasker A.D.
      • Coulden R.A.
      The role of preliminary interpretation of chest radiographs by radiographers in the management of acute medical problems within a cardiothoracic centre.
      ) or preliminary clinical evaluation (
      • Buissink C.
      • Thompson J.D.
      • Voet M.
      • et al.
      The influence of experience and training in a group of novice observers: a jackknife alternative free-response receiver operating characteristic analysis.
      ,
      • Ekpo E.U.
      • Egbe N.O.
      • Akpan B.E.
      Radiographers' performance in chest X-ray interpretation: the Nigerian experience.
      ) rather than clinical reporting by qualified and practicing reporting radiographers. The only other previous study that has examined the diagnostic accuracy of CXR reporting by reporting radiographer was in an academic setting (
      • Piper K.
      • Cox S.
      • Paterson A.
      • et al.
      Chest reporting by radiographers: findings of an accredited postgraduate programme.
      ), but no direct comparison was made with the performance of consultant radiologists.
      The main limitation was that the work occurred in a controlled setting rather than clinical practice (
      • Irwig L.
      • Bossuyt P.
      • Glasziou P.
      • et al.
      Designing studies to ensure that estimates of test accuracy will travel.
      ,
      • Hardy M.
      • Flintham K.
      • Snaith B.
      • et al.
      The impact of image test bank construction on radiographic interpretation outcomes: a comparison study.
      ,
      • Brealey S.
      • Scally A.J.
      Methodological approaches to evaluating the practice of radiographers' interpretation of images: a review.
      ). This approach had the advantage of increasing the number of participants and all reporting practitioners interpreting the same cases, with a robust reference standard diagnosis. The inclusion of images with subtle changes and those that had been viewed at a discrepancy meeting ensured that any difference in skilled interpretation of images would have been exaggerated, but is therefore a more stringent test of noninferiority.

      Comparison with Literature

      There is a considerable body of evidence that examine the diagnostic accuracy of CXR interpretation in a controlled (image bank) setting. A summary of studies that have compared chest radiograph diagnostic accuracy of reporting radiographers and consultant radiologists using a free response methodology are presented in Table 5. The majority of observers in these studies are consultant radiologists, with their performance compared to new technologies including digital subtraction (
      • Kashani H.
      • Varon C.A.
      • Paul N.S.
      • et al.
      Diagnostic performance of a prototype dual-energy chest imaging system ROC analysis.
      ,
      • Schalekamp S.
      • Karssemeijer N.
      • Cats A.M.
      • et al.
      The effect of supplementary bone-suppressed chest radiographs on the assessment of a variety of common pulmonary abnormalities: results of an observer study.
      ) and tomosynthesis (
      • Yamada Y.
      • Jinzaki M.
      • Hasegawa I.
      • et al.
      Fast scanning tomosynthesis for the detection of pulmonary nodules: diagnostic performance compared with chest radiography, using multidetector-row computed tomography as the reference.
      ,
      • Zachrisson S.
      • Vikgren J.
      • Svalkvist A.
      • et al.
      Effect of clinical experience of chest tomosynthesis on detection of pulmonary nodules.
      ). To facilitate comparison with the literature and the current study, the FoMs from the control reading, that is, under normal reporting conditions without a new technique for example, have been used.
      TABLE 5Summary of Summary of Studies That Have Used Alternate Free-response Receiver Operating Characteristic (AFROC) or Jack-knife Alternate Free-response Receiver Operating Characteristic (JAFROC) Methodology for Assessment of Chest X-ray Diagnostic Accuracy
      StudyNumber of ParticipantsPractitioner CharacteristicsNumber of Chest X-raysNormal:Abnormal RatioSimulated or Natural NodulesNature of Intervention/ComparisonObserver Performance Area Under Curve (AFROC) and FoM (JAFROC) Control Intervention
      Current study2110 CRs1061:1Natural—range of pathologiesDirect comparison of CRs and RRsCR mean FoM = 0.786
      11 RRsRR mean FoM = 0.830
      Manning et al.
      • Manning D.
      • Barker-Mill S.C.
      • Donovan T.
      • et al.
      Time-dependent observer errors in pulmonary nodule detection.
      218 CRs1201:2?naturalEye trackingAFROC (expert)
      5 RRs before/after 6 mo trainingRR after AUC = 0.82; CR AUC = 0.80
      8 UG radiographers (naïve)
      Donovan and Litchfield
      • Donovan T.
      • Litchfield D.
      Looking for cancer: expertise related differences in searching and decision making.
      40Naïve (nonmedical)301:124 natural, 4 simulatedEye tracking study, comparison with observer experienceNaïve mean FoM = 0.41
      UG radiographersFirst UG mean FoM = 0.60
      Experts (CR and RR)Third UG mean FoM = 0.71
      Experts mean FoM = 0.72
      AUC, area under the curve; CR, consultant radiologist; FoM, figure of merit; RR, reporting radiographer; UG, undergraduate.
      The FoMs of consultant radiologists and reporting radiographers in this study compare favorably to those reported in the literature (
      • Littlefair S.
      • Mello-Thoms C.
      • Reed W.
      • et al.
      Increasing prevalence expectation in thoracic radiology leads to overcall.
      ,
      • Donovan T.
      • Litchfield D.
      Looking for cancer: expertise related differences in searching and decision making.
      ,
      • de Hoop B.
      • De Boo D.W.
      • Gietema H.A.
      • et al.
      Computer-aided detection of lung cancer on chest radiographs: effect on observer performance.
      ,
      • Kasai S.
      • Li F.
      • Shiraishi J.
      • et al.
      Usefulness of computer-aided diagnosis schemes for vertebral fractures and lung nodules on chest radiographs.
      ,
      • Kohli A.
      • Robinson J.W.
      • Ryan J.
      • et al.
      Reader characteristics linked to detection of pulmonary nodules on radiographs: ROC vs. JAFROC analyses of performance.
      ,
      • Schalekamp S.
      • van Ginneken B.
      • Heggelman B.
      • et al.
      New methods for using computer-aided detection information for the detection of lung nodules on chest radiographs.
      ,
      • Yano Y.
      • Yabuuchi H.
      • Tanaka N.
      • et al.
      Detectability of simulated pulmonary nodules on chest radiographs: comparison between irradiation side sampling indirect flat-panel detector and computed radiography.
      ). Previous studies had a wide range of study designs and, in general, smaller participant numbers (3–4), with the exception of Brennan et al. (n = 15 general radiologists) (
      • Brennan P.C.
      • Ryan J.
      • Evanoff M.
      • et al.
      The impact of acoustic noise found within clinical departments on radiology performance.
      ), limiting generalizability, whereas the current study had a greater number of observers. Extrapolation of the diagnostic accuracy to the wider population of reporting radiographers is, therefore, likely better. Approximately 40 CXRs were included in these studies compared to 106 in this study. A greater number of cases, with a broad spectrum of pathologies, enable the results of the current study to be generalized to a wide population of patients.
      Previous work that used the JAFROC methodology to assess reporting radiographers in CXR interpretation concentrated on a single task, namely lung nodule detection, rather than a range of pathologies commonly encountered in clinical practice. Manning et al. found that the five radiographers after postgraduate training (FoM = 0.82) performed much the same as the eight consultant radiologists (FoM = 0.80) (
      • Manning D.
      • Ethell S.
      • Donovan T.
      • et al.
      How do radiologists do it? The influence of experience and training on searching for chest nodules.
      ). The number of radiologist (n = 8) observers and the number of cases (n = 120) were also similar to the current study, although fewer radiographer (n = 5) observers were used. Donovan and Litchfield included two reporting radiographers in their study (
      • Donovan T.
      • Litchfield D.
      Looking for cancer: expertise related differences in searching and decision making.
      ). Only the summary figure of merit was reported (FoM = 0.72, standard error 0.06) for the expert group and they did not differentiate reporting radiographers (n = 2) from consultant radiologists (n = 8) (
      • Donovan T.
      • Litchfield D.
      Looking for cancer: expertise related differences in searching and decision making.
      ). This was possibly due to the small numbers of observers.

      Conclusions

      Chest radiographs are a complex imaging investigation and are central to many patient pathways. The current study shows that reporting radiographers and radiologists demonstrate similar levels of diagnostic reporting accuracy for CXR interpretation. The performance of the reporting radiographers in the current study also compares well to previous work, which measured the diagnostic accuracy of consultant radiologists. That both the number of cases and the number of observers in the current study are comparable to the total sum of the previous literature is a strength of the current study and suggests that the results are generalizable to a wider population of trained reporting radiographers. Reporting radiographers could contribute as part of a sustainable strategy to meet additional clinical demand and diagnostic capacity requirements.

      Supplementary Data

      The following is the supplementary data to this article:

      References

        • Royal College of Radiologists
        Clinical radiology UK workforce report 2011.
        Royal College of Radiologists, London2012
        • Royal College of Radiologists
        Patients waiting too long for test results.
        Royal College of Radiologists, London2014
        • Royal College of Radiologists
        Clinical radiology UK workforce census 2015 report.
        Royal College of Radiologists, London2016
        • Canadian Institute for Health Information
        Health care in Canada: a focus on wait times.
        Canadian Institute for Health Information, Ottawa, Canada2012
        • Queensland Department of Health
        Inquiry into Gold Coast X-ray reporting.
        (Queensland, Australia)2014
        • Woznitza N.
        • Piper K.
        • Rowe S.
        • et al.
        Optimizing patient care in radiology through team-working: a case study from the United Kingdom.
        Radiography. 2014; 20: 258-263
        • Snaith B.
        • Milner R.C.
        • Harris M.A.
        Beyond image interpretation: capturing the impact of radiographer advanced practice through activity diaries.
        Radiography. 2016; 22: e233-e238
        • Beardmore C.
        • Woznitza N.
        • Goodman S.
        The radiography workforce current challenges and changing needs.
        College of Radiographers, London2016
        • Snaith B.
        • Hardy M.
        • Lewis E.F.
        Radiographer reporting in the UK: a longitudinal analysis.
        Radiography. 2015; 21: 119-123
        • Buskov L.
        • Abild A.
        • Christensen A.
        • et al.
        Radiographers and trainee radiologists reporting accident radiographs: a comparative plain film-reading performance study.
        Clin Radiol. 2013; 68: 55-58
        • Sheft D.J.
        • Jones M.D.
        • Brown R.F.
        • et al.
        Screening of chest roentgenograms by advanced roentgen technologists.
        Radiology. 1970; 94: 427-429
        • Flehinger B.J.
        • Melamed M.R.
        • Heelan R.T.
        • et al.
        Accuracy of chest film screening by technologists in the New York early lung cancer detection program.
        AJR Am J Roentgenol. 1978; 131: 593-597
        • Piper K.
        • Cox S.
        • Paterson A.
        • et al.
        Chest reporting by radiographers: findings of an accredited postgraduate programme.
        Radiography. 2014; 20: 94-99
        • Woznitza N.
        • Piper K.
        • Burke S.
        • et al.
        Adult chest radiograph reporting by radiographers: preliminary data from an in-house audit programme.
        Radiography. 2014; 20: 223-229
        • Ben Shimol S.
        • Dagan R.
        • Givon-Lavi N.
        • et al.
        Evaluation of the World Health Organization criteria for chest radiographs for pneumonia diagnosis in children.
        Eur J Pediatr. 2012; 171: 369-374
        • Ostensen H.
        Diagnostic imaging: what is it? When and how to use it where resources are limited?.
        World Health Organization, Geneva2001
        • Chakraborty D.P.
        Recent advances in observer performance methodology: Jackknife free-response ROC (JAFROC).
        Radiat Prot Dosimetry. 2005; 114: 26-31
        • College of Radiographers
        Preliminary clinical evaluation and clinical reporting by radiographers: policy and practice guidance.
        College of Radiographers, London2013
        • Royal College of Radiologists
        • Society and College of Radiographers
        Team working in clinical imaging.
        Royal College of Radiologists and the Society and College of Radiographers, London2012
        • Woznitza N.
        • Piper K.
        • Burke S.
        • et al.
        Agreement between expert thoracic radiologists and the chest radiograph reports provided by consultant radiologists and reporting radiographers in clinical practice: review of a single clinical site.
        Radiography. 2018; (In press)https://doi.org/10.1016/j.radi.2018.01.009
        • Obuchowski N.A.
        Sample size tables for receiver operating characteristic studies.
        AJR Am J Roentgenol. 2000; 175: 603-608
        • The Health and Social Care Information Centre
        Hospital episode statistics for England.
        (Inpatient statistics, 2011-12)2012
        • Chakraborty D.
        The FROC, AFROC and DROC variants of the ROC analysis.
        in: Van Metter R. Beutel J. Knundel H. Handbook of medical imaging. SPIE Press, Bellingham, WA2000: 771-796
        • Chakraborty D.
        Recent developments in FROC methodology.
        in: Samei E. Krupinski E.A. The handbook of medical image perception and techniques. Cambridge University Press, New York, USA2010: 216-239
        • Sadler G.R.
        • Lee H.C.
        • Lim R.S.
        • et al.
        Recruitment of hard-to-reach population subgroups via adaptations of the snowball sampling strategy.
        Nurs Health Sci. 2010; 12: 369-374
        • de Lacey G.
        • Morley S.
        • Berman L.
        Chest x-ray: a survival guide.
        Elsevier, Spain2008
        • Haygood T.M.
        • Ryan J.
        • Brennan P.C.
        • et al.
        On the choice of acceptance radius in free-response observer performance studies.
        Br J Radiol. 2013; 86 (42313554)
        • Littlefair S.
        • Mello-Thoms C.
        • Reed W.
        • et al.
        Increasing prevalence expectation in thoracic radiology leads to overcall.
        Acad Radiol. 2016; 23: 284-289
        • Chakraborty D.P.
        How to conduct a free-response study 2005.
        (Available at:) (Accessed June 26, 2012)
        • Chakraborty D.P.
        A status report on free-response analysis.
        Radiat Prot Dosimetry. 2010; 139: 20-25
        • Chakraborty D.P.
        JAFROC [program]. 4.2 version.
        (Pennsylvania, USA)2014
        • Robinson P.
        • Wilson D.
        • Coral A.
        • et al.
        Variation between experienced observers in the interpretation of accident and emergency radiographs.
        Br J Radiol. 1999; 72: 323-330
        • Tourassi G.
        Receiver operating characteristic analysis: basic concepts and practical applications.
        in: Samei E. Krupinski E.A. The handbook of medical image perception and techniques. Cambridge University Press, New York, USA2010: 187-203
        • Baldwin D.
        Pulmonary nodules again? The 2015 British Thoracic Society guidelines on the investigation and management of pulmonary nodules.
        Clin Radiol. 2016; 71: 18-22
        • Donovan T.
        • Litchfield D.
        Looking for cancer: expertise related differences in searching and decision making.
        Appl Cogn Psychol. 2013; 27: 43-49
        • Litchfield D.
        • Ball L.J.
        • Donovan T.
        • et al.
        Viewing another person's eye movements improves identification of pulmonary nodules in chest x-ray inspection.
        J Exp Psychol Appl. 2010; 16: 251-262
        • Manning D.
        • Barker-Mill S.C.
        • Donovan T.
        • et al.
        Time-dependent observer errors in pulmonary nodule detection.
        Br J Radiol. 2006; 79: 342-346
        • Manning D.
        • Ethell S.
        • Donovan T.
        • et al.
        How do radiologists do it? The influence of experience and training on searching for chest nodules.
        Radiography. 2006; 12: 134-142
        • Sonnex E.P.
        • Tasker A.D.
        • Coulden R.A.
        The role of preliminary interpretation of chest radiographs by radiographers in the management of acute medical problems within a cardiothoracic centre.
        Br J Radiol. 2001; 74: 230-233
        • Buissink C.
        • Thompson J.D.
        • Voet M.
        • et al.
        The influence of experience and training in a group of novice observers: a jackknife alternative free-response receiver operating characteristic analysis.
        Radiography. 2014; 20: 300-305
        • Ekpo E.U.
        • Egbe N.O.
        • Akpan B.E.
        Radiographers' performance in chest X-ray interpretation: the Nigerian experience.
        Br J Radiol. 2015; 88 (20150023)
        • Irwig L.
        • Bossuyt P.
        • Glasziou P.
        • et al.
        Designing studies to ensure that estimates of test accuracy will travel.
        in: Knottnerus J.A. Buntinx F. The evidence base of clinical diagnosis: theory and methods of diagnostic research. Blackwell Publishing, Singapore2009: 96-117
        • Hardy M.
        • Flintham K.
        • Snaith B.
        • et al.
        The impact of image test bank construction on radiographic interpretation outcomes: a comparison study.
        Radiography. 2016; 22: 166-170
        • Brealey S.
        • Scally A.J.
        Methodological approaches to evaluating the practice of radiographers' interpretation of images: a review.
        Radiography. 2008; 14: e46-e54
        • Kashani H.
        • Varon C.A.
        • Paul N.S.
        • et al.
        Diagnostic performance of a prototype dual-energy chest imaging system ROC analysis.
        Acad Radiol. 2010; 17: 298-308
        • Schalekamp S.
        • Karssemeijer N.
        • Cats A.M.
        • et al.
        The effect of supplementary bone-suppressed chest radiographs on the assessment of a variety of common pulmonary abnormalities: results of an observer study.
        J Thorac Imaging. 2016; 31: 119-125
        • Yamada Y.
        • Jinzaki M.
        • Hasegawa I.
        • et al.
        Fast scanning tomosynthesis for the detection of pulmonary nodules: diagnostic performance compared with chest radiography, using multidetector-row computed tomography as the reference.
        Invest Radiol. 2011; 46: 471-477
        • Zachrisson S.
        • Vikgren J.
        • Svalkvist A.
        • et al.
        Effect of clinical experience of chest tomosynthesis on detection of pulmonary nodules.
        Acta Radiol. 2009; 50: 884-891
        • de Hoop B.
        • De Boo D.W.
        • Gietema H.A.
        • et al.
        Computer-aided detection of lung cancer on chest radiographs: effect on observer performance.
        Radiology. 2010; 257: 532-540
        • Kasai S.
        • Li F.
        • Shiraishi J.
        • et al.
        Usefulness of computer-aided diagnosis schemes for vertebral fractures and lung nodules on chest radiographs.
        AJR Am J Roentgenol. 2008; 191: 260-265
        • Kohli A.
        • Robinson J.W.
        • Ryan J.
        • et al.
        Reader characteristics linked to detection of pulmonary nodules on radiographs: ROC vs. JAFROC analyses of performance.
        Proc SPIE. 2011; 7966 (79660K-79660K-79668)
        • Schalekamp S.
        • van Ginneken B.
        • Heggelman B.
        • et al.
        New methods for using computer-aided detection information for the detection of lung nodules on chest radiographs.
        Br J Radiol. 2014; 87 (20140015)
        • Yano Y.
        • Yabuuchi H.
        • Tanaka N.
        • et al.
        Detectability of simulated pulmonary nodules on chest radiographs: comparison between irradiation side sampling indirect flat-panel detector and computed radiography.
        Eur J Radiol. 2013; 82: 2050-2054
        • Brennan P.C.
        • Ryan J.
        • Evanoff M.
        • et al.
        The impact of acoustic noise found within clinical departments on radiology performance.
        Acad Radiol. 2008; 15: 472-476