Abstract: Image-based building reconstruction is a hot topic in computer vision and computer graphics. However, few studies have been conducted on reconstruction from a single image because it is an ill-conditioned problem and is very difficult to resolve. In the current paper, we present an efficient method by matching contours between an image and the projection of three-dimensional (3D) models. Our method simplifies the reconstruction process and avoids camera calibrating and human interaction. Given a building photo captured in daily life as input, we first segment the photo into different regions and then automatically extract the contour of the building. Finally, we obtain the 3D building model by matching the contour of the building with the projection contour of the 3D models from different views stored in the model database. Experiments show that our method is efficient and effective, especially for building photos containing shelters.