Thomas P. Karnowski et al.

Table 14.2. Performance results generated for two data sets. The top row shows the expected performance of the NL archive using a hold-one-out method. For comparison, another independent data set was used in the bottom three rows, with variable confidence (σ) and image quality values. From Ref. [67] — S = sensitivity and A = accuracy.

independent data, we have assembled an entirely independent test population as described earlier, represented by the results in the bottom three rows of Table 14.2. For this test population, we have 98 records as shown. We initially specified the quality metric to only accept images of value >0.5, resulting in sensitivity and accuracy of 67% and 77%, respectively. Next, we set the quality threshold back to 0 and set a confidence level of 3σ, resulting in a slightly higher sensitivity, and accuracy of 78% and 88%. Finally, we set the image quality threshold to accept >0.5 and 3σ confidence to result in the best sensitivity and accuracy of 82% and 89%.

These results show tests of our system using a separate, completely independent set of data collected under unrelated conditions. Despite the disparity in the two data sets, we have shown a level of robustness resulting in sensitivity to disease discrimination of 82% and overall accuracy of 89%.

14.2.7. Patient Demographics and Statistical Outcomes

Since its initial deployment at the Church Health Center in Memphis, TN, in February 2009, and more recently in Internal Medicine Clinics at the University of North Carolina, the ocular telehealth network has provided diagnostic reports on 1,373 eyes from 669 patients. According to the patient demographics, the population ranged in age from 20 to 91, with a mean age of 55.4 years. The patient population was predominantly female (64.28%)

444

Automating the Diagnosis, Stratification, and Management of DR

and the vast majority of the patients evaluated had Type II diabetes (93.42%). Type I diabetes comprised 3.58% of patients, and 2.86% of patients were unrecorded. Ethnicity profiles showed that 59.8% of patients were African American, 31.4% Caucasian, and approximately 3.8% were Hispanic or unrecorded.

14.2.8. Disease State Assessment

An assessment of the disease images is summarized in Table 14.3. A total of 1,036 eyes, comprising 75.46% of all patient eyes examined, had no evidence of DR. These patients did not require further evaluation by an eyecare specialist for diabetic eye disease and will be managed in the primary care clinic with follow-up retinal photography in 12 months. The incidence

Table 14.3. Epidemiology of DR in a population of diabetic patients examined for retinal lesions using the ocular telehealth network in Memphis, TN and Chapel Hill, NC.

445

Thomas P. Karnowski et al.

of any DR in the population was 17.47% and is consistent with the known epidemiology of DR in the region. Of the patients with DR, 62.50% (10.92% of total eyes examined) had minimal or mild DR without fluid leakage and did not require a formal eye examination. These patients will be rescreened in the primary care clinic in 6–12 months (12 months for patients with only rare microaneurysms). Thus, 86.38% of all diabetic eyes screened to date will continue to be followed for DR in the primary care setting. A total of 6.55% of all eyes had DR severe enough to warrant referral and treatment. An additional 84 eyes (6.12%) from 68 patients (10.16%) had retinal findings, exclusive of DR, that warranted a formal ophthalmic evaluation. These findings included macular degeneration, vascular occlusions, optic disc findings suggestive of glaucoma, and other problems or diseases.

14.2.9. Image QA

The QA module computes the image quality value and issues a “good” or “bad” evaluation result by thresholding. The QA not only provides immediate feedback of the image quality to the photographers after image acquisition, but also provides a quantitative indication of statistical changes for monitoring the image quality. The overall image quality steadily improved since initial deployment (Fig. 14.10). More recently, significant variance in overall image quality was noted, as a new clinic came on board from UNC Chapel Hill. This fluctuation of the statistical image quality can help to identify the training needs for the photographers when new people are enrolled into the system.

14.2.10.Physician Oversight Based on Quality and Confidence Levels

We conclude this chapter with a review of the concept of physician oversight. Throughout this work, we have explained that we employ automatic processing and seek to establish confidence levels that can be used to invoke physician oversight in a truly automated system. Physician oversight is invoked in three main areas. First, the quality level must exceed a basic threshold before it is deemed acceptable for submission to the network. Second, once an image is received, another, higher quality level threshold must

446

Automating the Diagnosis, Stratification, and Management of DR

be satisfied to deem the image of sufficient quality for automatic screening. We proved the above in Ref. [50], in which we showed that ON estimation improved based on the quality of the image. We showed that diagnosis improved when the quality of the image improved.67 Third, although not in “chronological” processing order, the confidence of the ON detection can be judged by the complementary method, which is more accurately a “degree of agreement.” Images that fail this metric may still be evaluated but will most likely be passed to the reviewing physician. Last, the use of Poisson statistics allows us to attach a level of confidence to automatic diagnosis. Improved diagnosis accuracy is achieved at the cost of fewer automatic screenings, but again the role of the oversight physician can be to improve the system performance.

One can imagine other thresholds of interest; for example, since we assume the images are macula centered, images with estimates of the ON location or macula location that stray too far from a mean position may be flagged for manual review. Another possibility that remains for exploration is the goodness of a fit to the parabolic model of the vascular tree. Our network employs some issues of practice that are rarely (if ever) invoked. Since our system is under development, any software bugs in the process automation would invoke physician oversight immediately. (This immediate alert is of course not an issue in our current system, since a physician reviews all images.) Any nonlinear computational issues that are iterative by nature are also set with timing thresholds so that if convergence is not reached they will be set to automatic physician review. These issues are also not expected to be common, but they are essential to ensuring robust operation and maximum patient care.

Finally, any future progress in the use of computer-aided and automated telemedical applications for the diagnosis of DR and other retinal diseases must also consider the “Telehealth Practice Recommendations for Diabetic Retinopathy” proposed by the Ocular Telehealth Special Interest Group (SIG) of the American Telemedicine Association.71 The SIG recognizes four categories of validation for telehealth for DR relative to the ETDRS 35-mm slide reference standards.

Category 1 validation can separate patients into those with none or very mild nonproliferative DR (ETDRS level 20 or below), and those with greater than ETDRS level 20 (stratifying DR by yes/no criteria for more than

447