Abstract: A model of visual saliency is often used to highlight interesting or perceptually significant features in an image. If a specific task is imposed upon the viewer, then the image features that disambiguate task-related objects from non-task-related locations should be incorporated into the saliency determination as top-down information. For this study, viewers were given the task of locating potentially cancerous lesions in synthetically-generated medical images. An ensemble of saliency maps was created to model the target versus error features that attract attention. For MRI images, lesions are most reliably modeled by luminance features and errors are mostly modeled by color features, depending upon the type of error (search, recognition, or decision). Other imaging modalities showed similar differences between the target and error features that contribute to top-down saliency. This study provides evidence that image-derived saliency is task-dependent and may be used to predict target or error locations in complex images.