A novel structure from motion (SfM) pipeline that combines point features and line features together is presented in this paper. Line segments usually are more difficult to handle than points in SfM due to the instability of their end points and the property of being unconstrained by epipolar geometry. Our proposed method first implements a very robust and efficient line segment matching by applying homography and optical flow tracking. Global homography and multiple homography schemes are combined to get a better effect. By virtue of sampling lines, line matching is converted to point matching. A unified global SfM pipeline is then available for both lines and points. In the end, the proposed method is evaluated through various datasets. The result shows that our line-based SfM can generate accurate 3D models with lines and give meaningful representation for a man-made environment.