Abstract: We present a system for generating real-time 3D reconstructions of the user and other real objects for visualization and interaction within an immersive virtual environment (IVE). For example, when parts of the user's body is in his field of view, our system allows him to see a visually faithful corresponding graphical representation of himself, an avatar. In addition, a user can grab real objects, and then see and interact with those objects in the IVE. We do not use additional tracking sensors or prior object knowledge, nor do we generate dense 3D representations of objects using computer vision techniques. Instead our system bypasses an explicit 3D modeling stage, using a set of outside-looking- in cameras and a novel visual hull technique that leverages the tremendous recent advances in graphics hardware. We accelerate the visual hull computation by using projected textures to rapidly determine which volume samples lie within the visual hull. The samples are combined to form the object reconstruction from a viewpoint. Our system produces results at interactive rates, and because it harnesses ever-improving graphics hardware, the rates and quality should continue to improve. We further examine real-time generated models as active participants in simulations (with lighting) in IVEs, and give results using synthetic and real data.