Accurate patient population stratification is a key requirement for a personalized medicine and more precise biomarkers are expected to be obtained by better exploiting the available data. We introduce a novel computational framework that exploits both the information from gene expression data and histopathology images for constructing a tissue-based biomarker, which can be used for identifying a high-risk patient population. Its utility is demonstrated in the context of colorectal cancer data and we show that the resulting biomarker can be used as a proxy for a prognostic gene expression signature. These results are important for both the computational discovery of new biomarkers and clinical practice, as they demonstrate a possible approach for multimodal biomedical data mining and since the new tissue-based biomarker could easily be implemented in the routine pathology practice.