We present a quantitative measure of physical complexity, based on the amount of information required to build a given physical structure through self-assembly. Our procedure can be adapted to any given geometry, and thus, to any given type of physical structure that can be divided into building blocks. We illustrate our approach using self-assembling polyominoes, and demonstrate the breadth of its potential applications by quantifying the physical complexity of molecules and protein complexes. This measure is particularly well suited for the detection of symmetry and modularity in the underlying structure, and allows for a quantitative definition of structural modularity. Furthermore we use our approach to show that symmetric and modular structures are favored in biological self-assembly, for example in protein complexes. Lastly, we also introduce the notions of joint, mutual and conditional complexity, which provide a useful quantitative measure of the difference between physical structures.