Background. The SpyGlass Direct Visualization System (Boston Scientific, Natick, MA) is routinely used during single operator choledochoscopy (SOC) to identify biliary lesions or strictures with a diagnostic accuracy up to 88%. The objective of this study was to determine the interobserver agreement (IOA) of modified scoring criteria for diagnosing biliary lesions/strictures. Methods. 27 SPY SOC video clips were reviewed and scored by 9 interventional endoscopists based on published criteria that included the presence and severity of surface structure, vasculature visualization, lesions, and findings. Results. Overall IOA was "slight" for all variables. The K statistics are as follows: surface (K = 0.12, SE = 0.02); vessels (K = 0.14, SE = 0.02); lesions (K = 0.11, SE = 0.02); findings (K = 0.08, SE = 0.03); and final diagnosis (K = 0.08, SE = 0.02). The IOA for "findings" and "final diagnosis" was also only "slight." The final diagnosis was malignant (11), benign (11), and indeterminate (5). Conclusion. IOA using the modified criteria of SOC images was slight to almost poor. The average accuracy was less than 50%. These findings reaffirm that imaging criteria for benign and malignant biliary pathology need to be formally established and validated.