The analysis of room impulse responses to localise reflecting surfaces and estimate room geometry is applicable in numerous aspects of acoustics, including source localisation, acoustic simulation, spatial audio, audio forensics, and room acoustic treatment. Geometry inference is an acoustic analysis problem where information about reflections extracted from impulse responses are used to localise reflective boundaries present in an environment, and thus estimate the geometry of the room. This problem, however, becomes more complex when considering non-convex rooms, as room shape cannot be constrained to a subset of possible convex polygons. This paper presents a geometry inference method for localising reflective boundaries and inferring the room's geometry for convex and non-convex room shapes. The method is tested using simulated room impulse responses for six scenarios, and real-world room impulse responses measured in a cuboid-shaped room, using a spherical microphone array containing multiple spatially distributed channels capable of capturing both time- and direction-of-arrival. Results show that the general shape of the rooms is inferred for each case, with a higher degree of accuracy for convex shaped rooms. However, inaccuracies generally arise as a result of the complexity of the room being inferred, or inaccurate estimation of time- and direction-of-arrival of reflections.