Validation of a New Reference Standard for the Diagnosis of Vasospasm

Rationale and Objectives

The purpose of our study is to perform an internal validation of a new reference standard for vasospasm diagnosis in aneurysmal subarachnoid hemorrhage (A-SAH) patients.

Materials and Methods

A retrospective study was performed on A-SAH patients between January 2002 and May 2009. All patients were applied to this new reference standard using a multistage hierarchical approach incorporating clinical and imaging criteria. An internal validation method was performed in two phases to compare the new reference standard with digital subtraction angiography (DSA) and to assess accuracy. In Phase I, the diagnostic outcomes from DSA at the primary level were compared with the secondary/tertiary levels in the reference standard. In Phase II, the new reference standard was compared with chart diagnosis. Accuracy test characteristics, agreement rates, kappa values, and bias indices were calculated.

Results

In Phase I (n = 85), there was 87% agreement rate, 0.674 kappa, and 0.12 bias index. However, there was 100% agreement in patients diagnosed with vasospasm by DSA. Sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) were 100%, 61%, 83%, and 100% respectively. In Phase II (n = 137), there was 91% agreement rate, 0.824 kappa, and 0.04 bias index. Sensitivity, specificity, PPV, and NPV were 88%, 95%, 96%, and 87%, respectively.

Conclusion

Performing validation methods for a new reference standard is an evolving and ongoing process because limitations and bias in the reference standard are identified. Based on the results of this internal validation, a modification in the new reference standard is made at the primary level, resulting in improvement in its accuracy and classification of A-SAH patients.

Key Words: Validation, reference standard, vasospasm, aneurysmal subarachnoid hemorrhage, DSA