Content-based image retrieval (CBIR) has often been proposed to assist medical decision making in complement to textual information search. However, applications of this novel technology have rarely reached the end users. The study presented in this paper describes the design and setup for performing pilot user tests in order to assess a medical information retrieval system that supports CBIR with the goal of having more detailed tests with an updated system. Five individuals with medical education participated in the study at the University hospitals of Geneva. They were recorded and observed while interacting with the system, and then provided feedback on the usability of the system. Participants seemed to understand the concept and practical uses of the new tools, and needed 10-15 minutes to feel confident with the system. The results of this pilot study will be used for improving the system functionalities as well as an input for designing a new iteration of larger-scale user tests among radiologists.