Abstract: In augmented reality, it is essential that the rendered virtual objects are embedded harmonically into the view of the background scenes and their appearance should reflect the changing lighting condition of the real scene to ensure illumination consistency. In this paper, we propose a novel method to solve for the sunlight and skylight basis images of static outdoor scenes from a time-lapse image sequence. It is proved that the resulted basis images encapsulate the geometry and material reflectivity of the scene, correspond to the global illumination effects of the outdoor scene under a unit intensity of the sunlight and skylight. Our method is fully automatic. Unlike previous methods, it gets rid of the constraints that the reflectance of all objects in scenes should be ideal diffuse, or the weather condition should be overcast or sunshine. During decomposition, we first detect shadowed pixels by analyzing the time-lapse curve of each pixel through k-means clustering, the basis images of sunlight and skylight are then solved by an iterative procedure with the decomposition equation. The basis images are further optimized by exploiting their constraints and priors. Experimental results demonstrate the effectiveness and flexibility of the proposed method. Our method can also be applied in image understanding and compressing.