Receiver-operating characteristic curves and likelihood ratios: improvements over traditional methods for the evaluation and application of veterinary clinical pathology tests

 

Vet Clin Pathol. March 2006;35(1):8-17. 45 Refs

Ian A Gardner, Matthias Greiner

 

I.                    Introduction

a.       For both continuous and ordinal measurements, a threshold or cutoff value is required to categorize a test result as positive (abnormal) or negative (normal)

b.      Cutoff values are required in test evaluation studies fro calculation of sensitivity (Se) and specificity (Sp)

c.       The main criticism leveled against the use of cutoff values that give equal weight to all results on the same side of the cutoff is that it fails to adequately capture the magnitude of the test result and associated probability of disease or nondisease

                                                               i.      These concerns raise the fundamental question as to whether cutoff values are essential to the test evaluator and clinician who makes the decisions and, if they are not, what alternatives are available

d.      The paucity of veterinary clinical pathology articles that use these techniques most likely reflects a lack of familiarity with methods and lack of awareness of how readily these calculations can be done with many computer programs

II.                 What are ROC curves and when should they be used?

a.       ROC methodology was developed to differentiate between signal and noise in the context of problems with different radar devices

b.      In the medical sciences, ROC analysis has been most widely applied in medical imaging, psychometric research, and clinical chemistry

c.       We use the term plot to denote a graph of the empirical data and the term curve to denote a smoothed ROC function where the curve parameters are estimated from empirical data

                                                               i.      ROC plots (and curves) provide a graphic representation of all possible true positive (Se) and false positive (1-Sp) fractions for an ordinal or continuous test

                                                             ii.      ROC plots can be used, therefore, as universal tools for test comparison even when the tests are quite different in their cutoff values and in their units and ranges of measurement

III.               Why use ROC analysis?

a.       Because there is an inverse relationship between Se and Sp as the cutoff value is changed, optimizing Se (Sp) will result in decreased Sp (Se)

b.      ROC analysis provides a cutoff-independent method of summarizing the diagnostic accuracy of a test over all possible operating points

c.       ROC analysis is especially useful for comparison of the accuracy of two tests for the same disease because there is no need to select cutoff values to compare Se and Sp

d.      There may be some situations where presentation of the ROC curve and AUC comparison over all operating points could be misleading

e.       Geometrically, the point of the ROC plot with the greatest distance in the northwest direction from the diagonal line represents the maximum combined value of Se and Sp that can be achieved with the test

f.        The advantage of TG-ROC is that one can read the measures of accuracy from the graph as a function of cutoff value, and the program can be used to derive two cutoff values that define an intermediate or suspicious category of test results

IV.              Plotting ROC curves in practice

a.       ROC curves can be generated using spreadsheet programs such as Excel

V.                 Area under the ROC curve and its interpretation

a.       The area under the ROC curve provides a global summary statistic of test accuracy and for useful tests, AUC ranges between 0.5 and 1

b.      A perfect test has an AUC of 1 and a useless test has an area of 0.5

c.       Two main interpretations of the AUC

                                                               i.      It can be interpreted as the probability that test results for a randomly selected pair of diseased and nondiseased animals are correctly ordered

1.      The probability of a correct answer in this experiment is independent of prevalence because it is fixed at 50% by design

                                                             ii.      The AUC can be interpreted as the average Se, averaged across the entire range of false-positive fractions between 0 and 1

VI.              Estimation of the ROC curve and its associate area

a.       Methods of estimation can be classified into 1 of 3 broad categories:

                                                               i.      Empirical methods

                                                             ii.      Distribution-free parametric methods

                                                            iii.      Distribution-modeling methods

VII.            Use of ROC curves to compare the accuracy of multiple tests

a.       For dichotomous test results, the Se and Sp of pairs of test can be compared using Pearson’s chi-square tests or McNemar’s test for correlated data

b.      Testing of the same set of samples in a paired design usually is preferable because confounding by variables such as age, stage of disease, or disease severity will be prevented

c.       For comparison of the accuracy of ordinal or continuous tests, the optimal approach is comparison of the test-specific ROC curves and estimation of the difference in the AUC of these curves

d.      Two ROC curves can have almost identical areas and even intersect at some points, but have quite different shapes

e.       A procedure for comparing two ROC curves based on a permutation has been developed and is useful for paired-sample designs and for comparing the shape of ROC curves with identical AUC’s

VIII.         Sample size considerations for ROC studies

a.       Sample size is usually considered to be of secondary importance to the representativeness of samples used in an ROC analysis

                                                               i.      However, the number of samples tested from diseased and nondiseased animals affects the precision of estimation of the AUC of a single test and the ability to detect differences between the AUC’s of two tests, regardless of whether the tests are performed on the same or independent samples

IX.              ROC analysis without a gold standard

a.       For many animal diseases, a gold standard is not available, is prohibitively expensive to acquire, or is not feasible or ethically possible to obtain

                                                               i.      In the absence of a gold standard, the problem is not statistically identifiable, unless additional information can be provided about the accuracy of one ore more of the tests

X.                 Likelihood ratios

a.       Usually, the aim of testing is to increase the post-test probability of disease above a threshold of certainty that warrants an intervention, or to reduce the post-test probability below a threshold that warrants exclusion of the given disease as a diagnostic possibility

b.      The magnitude of the calculated post-test probability of disease (D+) is dependent on the pretest (prior) probability of disease (P) and the Se and Sp of the tests

c.       The pretest odds is the ratio of the number of diseased to nondiseased animals and is calculated as P/(1-P)

d.      The LR represents a combination of Se and Sp values into a single estimate for a positive (LR+) and a negative (LR-) test result

e.       LR+ (LR-) expresses the odds that a + (-) result is evident in an individual with the disease of interest (compared to without)

XI.              Clinical application of likelihood ratios

a.       If a test outcome is continuous, the probability of observing a unique value (x) theoretically will be 0 in both diseased and nondiseased animals

XII.            Selection of a pretest probability for post-test probability calculations

a.       Pretest probabilities for a given disease can vary between 0 and 100% and are uniquely determined based on prior experience of clinicians and local conditions such as the type of practice (primary or referral) and geographic location

b.      A recent study found large variation in the estimated pretest probability of disease for doctors presented with identical patient information

c.       One disadvantage of interval likelihood ratios is that unless the sample size is large, there is loss of precision when many interval are used

XIII.         Advantages of likelihood ratios and misuse in the medical literature

a.       They summarize the information in both Se and Sp into a single value

b.      Do not vary with prevalence per se

c.       Most important, they can be calculated on an interval (continuous data) or a category basis (ordinal data) rather than at a single threshold

d.      If the pretest probability of a disease is known, likelihood ratios allow for direct calculation of post-test probabilities based on a modification of Bayes theorem

e.       When test are used sequentially, the final post-test probability can be readily obtained by multiplying the post-test odds derived using the first test result in the sequence by the likelihood ratio for the second test

f.        Common misuses of likelihood ratios in the medical literature include:

                                                               i.      Reporting of likelihood ratios without CIs to represent the inherent uncertainty associated with limited numbers of test results in each category

                                                             ii.      Selection of a cutoff point based on LR+ alone

                                                            iii.      Not considering the information content in the LR-

                                                           iv.      The greatest problem in the literature often is the lack of differentiation of likelihood ratios for cutoff points vs. ranges of values

XIV.         Relationship between likelihood ratios and ROC curves

a.       Likelihood ratios can be calculated from the slope of the ROC curve

b.      There are three possible slope definitions depending on whether test results are dichotomous, continuous and categorized into intervals, or continuous but not categorized into intervals

XV.           Conclusions

a.       Test evaluation studies must be well designed following recommended guidelines to avoid bias in estimation of ROC curves and likelihood ratios

b.      We recommend reporting of ROC curves and AUC (and 95% CI) as standard practice in test evaluation and comparison studies involving ordinal and continuous tests

c.       The main advantage of these approaches is that they provide a cutoff independent method of test accuracy assessment

d.      The website www.anaesthetist.com/mnm/stats/roc contains ROC applets for readers interested in getting started in ROC curves