Textureless objects, repetitive patterns and limited computational resources pose significant challenges to man-made structure reconstruction from images, because feature-points-based reconstruction methods usually fail due to the lack of distinct texture or ambiguous point matches. Meanwhile multi-view stereo approaches also suffer from high computational complexity. In this paper, we present a new framework to reconstruct 3D surfaces for buildings from multi-view images by leveraging another fundamental geometric primitive: line segments. To this end, we first propose a new multi-resolution line segment detector to extract 2D line segments from each image. Then, we construct a 3D line cloud by introducing an improved Line3D++ algorithm to match 2D line segments from different images. Finally, we reconstruct a complete and manifold surface mesh from 3D line segments by formulating a Bayesian probabilistic modeling problem, which accurately generates a set of underlying planes. This output model is simple and has low performance requirements for hardware devices. Experimental results demonstrate the validity of the proposed approach and its ability to generate abstract and compact surface meshes from the 3D line cloud with low computational costs.