Abstract: In augmented reality, consistent illumination is an important aspect of integrating a virtual object into a video of a real scene. In this paper, we propose a novel image-based framework to estimate online the dynamically changing illumination parameters of outdoor video sequences captured by a fixed or pure rotation camera. Unlike previous approaches, which either require knowledge of the scene geometry or significant storage to preserve time- and view-dependent basis images or statistical parameters, our approach requires a very simple interaction at the initialization stage: a few brushes are used to select areas with a specified normal surface, and these areas are used to calculate the sunlight parameters. Additional areas with the same normal parameters will be detected automatically, and an optimization procedure is applied to ensure the robustness and precision of our estimation. Our new approach adapts to videos of the same scene with varied sun positions and camera panning. The experimental results demonstrate the effectiveness and flexibility of our proposed approach.