Abstract: In this paper we present a novel approach for capturing the environment matte of a scene. We impose no restrictions on material properties of the objects in the captured scene and exploit scene characteristics (e.g. material properties and self-shadowing) to minimize recording time and to bound the error. Using a CRT monitor, wavelet patterns are emitted onto the scene in order of importance to efficiently construct the environment matte. This order of importance is obtained by means of a feedback loop that takes advantage of the knowledge learned from previously recorded photographs. Once the recording process is finished, new backdrops can be efficiently placed behind the scene.