Information visualization (infovis) is a powerful tool for exploring rich datasets. Within humanistic research, rich qualitative data and domain culture make traditional infovis approaches appear reductive and disconnected, leading to low adoption. In this paper, we use a multi-step approach to scrutinize the relationship between infovis and the humanities and suggest new directions for it. We first look into infovis from the humanistic perspective by exploring the humanistic literature around infovis. We validate and expand those findings though a co-design workshop with humanist and infovis experts. Then, we translate our findings into guidelines for designers and conduct a design critique exercise to explore their effect on the perception of humanist researchers. Based on these steps, we introduce Layers of Meaning, a framework to reduce the semantic distance between humanist researchers and visualizations of their research material, by grounding infovis tools in time and space, physicality, terminology, nuance, and provenance.