Abstract: In this paper, we reverse engineer facade design from single rectified image of existing building facade by the use of similarity and hierarchy features of man-made objects. The inferred design is encoded into parametric grammar rules, named as ArchSys, which draw a compact and semantically meaningful characterization of the building structure and can be considered to support the design of other architectures. Combining with Gradient-based Mutual Information measure, we propose a rough-fine template-based similarity detection method to extract the structure patterns in a hierarchical way, which reduces computation time while increases robustness of the whole system. Our approach can be applied to various architectural typologies to detect not only symmetrical features but also similar patterns in one facade image. A feedback loop is built to refine the facade structure analysis and rule sets' parameters. Experimental results illustrate that our method is of robustness and general applications.