The dry landscape garden at Ryoanji Temple in Kyoto, Japan, a UNESCO world heritage site, intrigues hundreds of thousands of visitors every year with its abstract, sparse and seemingly random composition of rocks and moss on an otherwise empty rectangle of raked gravel. Here we apply a model of shape analysis in early visual processing to show that the 'empty' space of the garden is implicitly structured and critically aligned with the temple's architecture. We propose that this invisible design creates the visual appeal of the garden and was probably intended as an inherent feature of the composition.