The need for annotating the continuously increasing volume of medical image data is recognized from medical experts for a variety of purposes, regardless if this is medical practice, research or education. The rich information content latent in medical images can be made explicit and formal with the use of well-defined ontologies. Evolution of the Semantic Web now offers a unique opportunity of a web-based, service-oriented approach. Remote access to FMA and ICD-10 reference ontologies provides the ontological annotation framework. The proposed system utilizes this infrastructure to provide a customizable and robust annotation procedure. It also provides an intelligent search mechanism indicating the advantages of semantic over keyword search. The common representation layer discussed facilitates interoperability between institutions and systems, while semantic content enables inference and knowledge integration.