Medical images form a core portion of all the information a clinician utilizes to render diagnostic and treatment decisions while a patient is under his/her care. As such, medical imaging is a major feature of modern medical care. An important requirement in using medical images is to understand what an image indicates; there is therefore a need to perceive (i.e. interpret) medical images and an associated need to have physicians subspecialized in medical image interpretation. The goal of this chapter is to provide a broad picture of the importance of medical image perception from a general healthcare perspective.
Medical imaging has been primarily ascribed to the subspecialty of radiology, with about a billion radiological imaging exams performed worldwide every year. The images include many types of examinations – single projection X-rays used in musculoskeletal, chest, and mammography applications; dynamic X-ray exams such as fluoroscopy, three-dimensional computed tomography (CT), and magnetic resonance (MR) exams; nuclear medicine emission images; and ultrasound. With the advent of digital imaging and multi-detector CT, the type and number of radiology examinations have been changing as well. The range of image types is also expanding rapidly with new modalities such as tomosynthesis and molecular imaging, which is being investigated for numerous applications, from identifying lesion margins during surgical removal to identifying cancer cells in the blood. Imaging technologies are extremely varied. Medical images can be grayscale or color, high-resolution or low-resolution, hardcopy or softcopy, uncompressed or compressed (lossy or lossless), acquired with everything from sophisticated dedicated imaging devices to off-the-shelf digital cameras.
While imaging is the central technology behind the subspecialty of radiology, during the past several years, imaging has also expanded beyond radiology to embrace other subspecialties including cardiology, radiation oncology, pathology, and ophthalmology, to name a few. Study of pathological specimens used to be limited to glass slide specimen “images” rendered by the microscope for the pathologist to view. With the advent of digital slide scanners in recent years, however, virtual slides are becoming more prevalent not only in telepathology applications but in everyday reading (Weinstein, 2001). In many medical schools and pathology residency programs, students are no longer required to purchase a microscope and a box of glass specimen slides. Students now learn from a CD with directories of virtual slides to view as softcopy images. Ophthalmology has relied on images for years (mainly as 35 mm film prints or slides) for evaluating such conditions as diabetic retinopathy. With the advent of digital images and high-performance color displays, screening raters are increasingly using softcopy images. Telemedicine has opened up an entirely new area in which medical images are being acquired, transferred, and stored to diagnose and treat patients (Krupinski, 2002). Specialties such as teledermatology, teleophthalmology, telewound/burn care, and telepodiatry are all using images on a regular basis for store-and-forward telemedicine applications. Real-time applications such as telepsychiatry, teleneurology and telerheumatology similarly rely on video images for diagnostic and treatment decisions.
One way to demonstrate the pervasiveness of medical imaging is to examine the amount of money spent each year on healthcare and then portion out the amount devoted to medical imaging (Beam, 2006). Relying on 2004 data from the Centers for Medicare and Medicaid Services (CMS), approximately 16% of the gross domestic product (GDP), or $1.6 trillion, is allotted to national healthcare expenditures (http://www.cms.hhs.gov/home/rsds.asp). Medicare expenditures represent 17% of national healthcare expenditures, of which Part B (43%) accounts for the non-facility or physician-related expenditures. Approximately 8% of Part B (or nearly $10 billion) constitutes physician-based imaging procedures. Imaging also accounts for over 40% of all hospital procedures reported in the discharge report according to the Agency for Healthcare Research and Quality (AHRQ) (http://www.ahrq.gov/data/hcup/). Based on Medicaid Part B spending, one can conservatively assume that imaging procedures comprise only 8% of non-Medicaid Part B health spending. Therefore, medical imaging in the USA is estimated to amount to $56 billion ($10 billion/17%/43%), or 0.5% of GDP.
With the pervasiveness of imaging in modern medicine, there has been significant attention and interest in the technology of imaging operations, ranging from hardware features to software functionalities. What is less appreciated is the perceptual act underlying the interpretation of these images (Manning, 2005). In order to impact patient care, an image must be perceived and interpreted (i.e. understood in the context of patient care) (Figure 1.1). If one assumes each of the one billion imaging examinations performed worldwide annually involves an average of four individual images per exam, one could compute that on the average, 120 medical image perception events take place every second! This astounding frequency speaks further of the pervasiveness of medical image perception in healthcare enterprise.
| Image not available in HTML version |
| Image not available in HTML version |
The need for interpretation of medical images comes from the fact that medical images are not self-explanatory. In the popular culture, “a picture is worth a thousand words,” a phrase that reflects the power and utility of images. Ironically however, the interpretation of a medical image involves summarizing a multi-dimensional image into a few words because medical images by themselves do not deliver the certainty that they promise (Figure 1.2). This uncertainty, which necessitates interpretation, stems from the nature of medical imaging. Imaging is ultimately a visual discipline, impacted by psychophysical processes involved in the interpretation of images. For example, medical images can contain anatomical structures that can camouflage a feature of clinical interest that is not prevalent (in the case of screening). This uncertainty impacts the psychology of interpretation. Added to this complexity are notable variations from case to case and a multiplicity of compounding abnormalities and related factors that the interpreter needs to be mindful of.
There are clearly a significant number of images being viewed and interpreted by clinicians today in a variety of clinical specialties. As such, diagnostic accuracy cannot be defined independently of the interpretation, and any limitations or suboptimality in terms of how the images are used can significantly influence the diagnostic and therapeutic clinical decisions that they enable. Given a one-to-one link between an image and its interpretation, imaging technology alone can offer little in terms of patient care if the image is misinterpreted. The complexities of image interpretation can lead to interpretation errors and clinicians do make mistakes in the interpretation of image data (Berlin, 2005, 2007). Estimates in radiology alone suggest that in some areas there may be up to a 30% miss rate and an equally high false-positive rate. Errors can also occur in the recognition of an abnormality (e.g. whether a lesion is benign or malignant). Such errors can have a significant impact on patient care due to delays or misdiagnoses. What is less well appreciated is the prominent contribution of the inherent limitations of human perception to these errors. Image perception is the most prominent yet least appreciated source of error in diagnostic imaging. The prominence of imaging reading errors in malpractice litigation is an example of this ignorance.
The likelihood of error in the interpretation of images emphasizes the need to understand how the clinician interacts with the information in an image during interpretation. Such an understanding enables us to determine how we can further improve decision-making. That brings us to the science of medical image perception. Error is one reason to study medical image perception.
First and foremost, it is important to understand the nature and causes of interpretation error. For that objective, one needs to distinguish between visual errors (estimated to amount to about 55% of the errors) because the clinician does an incomplete search of the image data (Giger, 1988); and cognitive errors (45%), where an abnormality is recognized but the clinician makes a decision-making error in calling the case negative (Kundel, 1978). Visual errors are further subdivided into errors where the clinician fails to look at the territory of the lesion (30%) (Kundel, 1975, 1978), and those when he/she does not fixate on the territory for an adequate amount of time to extract the lesion's relevant features (25%) (Carmody, 1980).
Contributing to interpretation errors are a host of psychophysical processes. Camouflaging of the abnormality by normal body features (so called anatomical noise) is one of the main contributors to interpretation error. Masking of subtle lesions by normal anatomical structure is estimated to affect lesion detection threshold by an order of magnitude (Samei, 1997). The visual search process, necessitated by the limited angular extent of the high-fidelity foveal vision of the human eye, is another important contribution to image interpretation. Preceded by a global impression or gist, a visual search of an image involves moving the eye around the image scene to closely examine the image details (Nodine, 1987). Studies on visual search have highlighted the prominent role of peripheral vision during the interpretation, where there is an interplay between foveal and peripheral vision as the observer scans the scene (Kundel, 1975). As a result, there are characteristic dwell times associated with correct and incorrect decisions that are influenced by the task and idiosyncratic observer search patterns (Kundel, 1989). Satisfaction of search – once an abnormal pattern is recognized, it takes additional
Image quality is yet another topic of interest. While intuitively recognized, image quality has been more elusive than image interpretation to characterize in such a way that it would directly relate to diagnostic accuracy (or its converse, diagnostic error). In that regard, it is important to understand how best to assess image quality and its impact on perception in order to optimize it and minimize error (Krupinski, 2008). Studies have focused on the impact of image acquisition, imaging hardware, image processing, image display, and reading environment on image quality and diagnostic accuracy.
Ergonomic aspects of interpreting medical images also play a role in the perception process. There is a need to understand the impact of ergonomic and presentation factors to minimize error (Krupinski, 2007), including determination of the causes of fatigue and how they can be minimized, the contribution of fatigue to error, the environmental distractions, the impact of the viewing interface, especially with softcopy images, and the impact of the color tint of the image.
Though we hope and aim for consistent and correct clinical decisions with every case, that aim is hard to achieve. The likelihood of two clinicians rendering two different interpretations of the same image is unsettlingly high and the expertise of the clinician plays an important role in this problem. Medical expertise is the ability to efficiently use contextual medical knowledge to make accurate and consistent diagnoses. Medical imaging expertise further involves perceptual and cognitive analysis of image features and manifests itself in a rich structured knowledge of normalcy and “perturbations” from the normal, an efficient hypothesis-driven search strategy, and an ability to generalize visual findings to idealized patterns. Achieving such expertise requires talent further honed by motivated effortful study, preferably supervised, and dedicated work, where accuracy is roughly proportional to the logarithm of the number of cases read annually (Nodine, 2000). Topics of interest in this line of investigation include the impact of the clinician's experience, age, and visual acuity on accuracy, toward better training and utilization of medical imaging clinicians.
Considering the impact of image perception on diagnostic accuracy, it is often necessary to test various imaging technologies and methods in terms of the associated impact on image perception. Such studies require the use of experienced clinicians, which is an expensive undertaking. Thus, there is a great need for accurate computational programs that can model visual perception and predict human performance. A host of such perceptual models have been developed, including the ideal human observer model, non-prewhitening models, channelized models, and visual discrimination models (Abbey, 2000). These models naturally require a reasonably accurate understanding of the image interpretation process. As our knowledge of the process is limited, so is the accuracy of these models. As such, their use often requires certain assumptions, verifications of their accuracy and relevance in pilot experiments, and certain calibrations, e.g. adding internal noise to make the model predictions fit human results. Nonetheless, these models have demonstrated valuable, though limited, utility in many applications, and their advancement continues to shed light on the image interpretation process.
By and large, image interpretation is currently a human task. However, increasingly, artificial intelligence tools are being used to aid in interpretation or to replace the radiologist altogether. The most common technology currently used is computer aided diagnosis (CAD), computer algorithms that examine the image content for certain abnormal features of clinical interest and then prompt the clinician for a closer examination of those features (Doi, 2007). CAD is becoming an important tool for interpreting medical images, considering the exponential growth of imaging and the shortage of specialized expertise. There is currently a need to understand the impact of CAD on diagnosis by investigating issues such as how best to integrate the human and the machine in such a way that the strength of both can be fully utilized towards improved diagnosis. For example, an experienced clinician might ignore the CAD prompts or be distracted by them if the system indicates too many false-positives. On the other hand, an inexperienced clinician might overly depend on CAD, initiating unnecessary follow-up procedures or dismissing an abnormality that might not have been picked up by the CAD algorithm. Such patterns might also change over time as a clinician gets used to a system, and such “getting used to” might not necessarily lead to improved diagnosis or efficiency. Thus, there is a need to understand the impact of CAD on the clinician's psychology, expertise, efficiency, and specialization paradigms.
Fundamental to the discussion above is the need to measure diagnostic accuracy itself (Metz, 2006; Wagner, 2007). There are a number of measures of performance such as fraction correct, sensitivity, and specificity. However, such simple measures do not adequately reflect accuracy as they can be dependent on disease prevalence or the criteria applied by the clinician, e.g. a clinician who calls all cases abnormal will have a perfect sensitivity but poor specificity, and vice versa. Seeking an overall performance measure independent of disease prevalence and criterion, receiver operating characteristic (ROC) analysis has emerged as the current gold standard for measuring diagnostic accuracy. However, ROC analysis has a number of limitations, including being limited primarily to single tasks, non-binary confidence ratings, and location-independent decisions. In recent years, a number of advances of the ROC methodology have been developed, a welcome expansion which has shown continued progress.
Medical image perception is a mature science that continues to be advanced by expert scientists. When over-specialization causes specialized “territories” to be left to the experts, one may ask why a clinician who interprets medical images needs to care about medical image perception. Needless to say, no one expects a clinician to also be a medical perception scientist. However, some knowledge of perception issues and concerns can provide
1. Patient care-related: Understanding perceptual issues could help a clinician to improve his/her performance. Knowledge of key perceptual factors such as satisfaction of search, the relevance of prolonged dwell time, search strategies, and psychological impacts of CAD can affect the way he/she interprets medical images. Awareness of these issues enforces a greater care about the way the images are created, a greater appreciation for image quality and its effect on accuracy and efficiency, an appreciation for the influence of fatigue and the proper ergonomic design of the working environment, and higher confidence in the use of new display technologies.
2. Science-related: Being better informed about key perceptual factors enables a more proper design of projects involving medical images, develops an ability to better answer perceptual questions that inevitably arise in the review of imaging-related papers and grant applications, and increases proficiency in the reviewing of such papers and grants.
3. Teaching and learning-related: Knowledge of perceptual factors can help clinicians better communicate their expertise to trainees and help clinicians hone their perceptual skills.
4. Consumer-related: Understanding the importance of perceptual factors enables a clinician to be a better shopper of medical image-related products and services. For example, he/she will be more mindful of the image quality performance of acquisition and display devices, and the importance of the graphical user interface of picture archiving and communication system workstations.
5. Profession-related: Awareness of image perception issues enables a clinician to better educate patients, other medical professionals, and the public about the statistical nature of medical image interpretation, and to play a more effective role in related malpractice litigations.
As outlined above, medical image perception is a frequent clinical task and a notable component of modern medicine. With perceptual error as one of the major sources of medical decision errors, our knowledge of perceptual issues gives us resources to minimize these errors and to educate future medical imaging clinicians and scientists. This book aims to provide a comprehensive reflection of medical perception concepts and issues within a single volume. Chapters in this text deal with a variety of perceptual issues in detail.
The first part of the book offers chapters by four prominent scientists, reflecting on historical developments of the field and its theoretical foundations. This part includes some reflections of the late Robert Wagner, the legendary perception scientist whose work and impact has been paramount in shaping the field as we know it today. The second part of the book includes six chapters discussing the science of medical image perception. Main topics include visual and cognitive factors, satisfaction of search, and the role of expertise. This part concludes with the perceptual relevance of image quality and reflections on the limitations of the human visual system. Part three focuses on perception metrology, with chapters on the logistics of designing perception experiments, and ROC methodology and its variants. This part ends with discussion of perceptual observer models and their implementation. Part four focuses on decision support and CAD, with topics ranging from the design of CAD studies to perceptual factors associated with the use of CAD in interpreting chest, breast, and volumetric images.
The last major part of the book offers six additional chapters about specific optimization considerations from a perceptual standpoint. Applications include radiography, CT, mammography, image processing, and image display. This part further offers a perspective on ergonomic design of workplaces for radiologists. The book ends with an epilogue outlining future possible directions for medical image perception science.
Abbey, C.K., Bochud, F.O. (2000). Modeling visual detection tasks in correlated image noise with linear model observers. In Beutel, J., Van Metter, R., Kundel, H. (eds). Handbook of Medical Imaging, Vol. 1: Physics and Psychophysics. Bellingham, WA: SPIE Press, pp. 655–682.
Beam, C.A., Krupinski, E.A., Kundel, H.L., Sickles, E.A., Wagner, R.F. (2006). The place of medical image perception in 21st-century health care. JACR, 3, 409–412.
Berbaum, K.S., Franken E.A., Dorfman, D.D., et al. (1989). Satisfaction of search in diagnostic radiology. Invest Radiol, 25, 133–140.
Berlin, L. (2005). Errors of omission. AJR, 185, 1416–1421.
Berlin, L. (2007). Accuracy of diagnostic procedures: has it improved over the past five decades? AJR, 188, 1173–1178.
Carmody, D.P., Nodine, C.F., Kundel, H.L. (1980). An analysis of perceptual and cognitive factors in radiographic interpretation. Perception, 9, 339–344.
Doi, K. (2007). Computer-aided diagnosis in medical imaging: historical review, current status and future potential. Comput Med Imag & Graphics, 31, 198–211.
Giger, M.S., Doi, K., MacMahon, H. (1988). Image feature analysis and computer-aided diagnosis in digital radiography. 3. Automated detection of nodules in peripheral lung fields. Med Phys, 15, 158–166.
Krupinski, E.A., Jiang Y. (2008). Evaluation of medical imaging systems. Med Phys, 35, 645–659.
Krupinski, E.A., Kallergi, M. (2007). Choosing a radiology workstation: technical and clinical considerations. Radiol, 242, 671–682.
Krupinski, E.A., Nypaver, M., Poropatich, R., et al. (2002). Clinical applications in telemedicine/telehealth. Telemed J e-Health, 8, 13–34.
Kundel, H.L. (1975). Peripheral vision, structured noise and film reader error. Radiol, 114, 269–273.
Kundel, H.L., Nodine, C.F., Carmody, D. (1978). Visual scanning, pattern recognition and decision-making in pulmonary nodule detection. Invest Radiol, 13, 175–181.
Kundel, H.L., Nodine, C.F., Krupinski, E.A. (1989). Searching for lung nodules: visual dwell indicates locations of false-positive and false-negative decisions. Invest Radiol, 24, 472–478.
Manning, D.J., Gale, A., Krupinski, E.A. (2005). Perception research in medical imaging. Br J Radiol, 78, 683–685.
Metz, C.E. (2006). Receiver operating characteristic analysis: a tool for the quantitative evaluation of observer performance and imaging systems. JACR, 3, 413–422.
Nodine, C.F., Kundel, H.L. (1987). Using eye movements to study visual search and to improve tumor detection. RadioGraphics, 7, 1241–1250.
Nodine, C.F., Mello-Thoms, C. (2000). The nature of expertise in radiology. In Beutel, J., Van Metter, R., Kundel, H. (eds). Handbook of Medical Imaging, Vol. 1: Physics and Psychophysics. Bellingham, WA: SPIE Press, pp. 859–894.
Samei, E., Flynn, M.J., Kearfott, K.J. (1997). Patient dose and detectability of subtle lung nodules in digital chest radiographs. Health Physics, 72(6S).
Smith, M.J. (1967). Error and Variation in Diagnostic Radiology. Springfield, IL: Charles C. Thomas.
Tuddenham, W.J. (1962). Visual search, image organization, and reader error in Roentgen diagnosis: studies of psychophysiology of Roentgen image perception. Radiol, 78, 694–704.
Tuddenham, W.J. (1963). Problems of perception in chest roentgenology: facts and fallacies. Radiol Clin North Am, 1, 227–289.
Wagner, R.F., Metz, C.E., Campbell, G. (2007). Assessment of medical imaging systems and computer aids: a tutorial review. Acad Radiol, 14, 723–748.
Weinstein, R.S., Descour, M.R., Liang, C., et al. (2001). Telepathology overview: from concept to implementation. Human Path, 32, 1283–1299.