Abstract: With the continued technology innovation in object sensing and human motion tracking, the traditional two-dimensional video-based tele-conference systems are projected to evolve into the three-dimensional immersive and augmented reality (AR) based on which one can communicate with the teleported remote other as if present, moving and interacting naturally in the same location. One technical hurdle to this vision is the need to resolve the environmental differences and the resulting teleported avatar motion anomaly between the remote and local sites. This paper presents a novel method to first establish a spatial and object-level match between the remote and local sites and adapts the position and motion of the teleported avatar into the local AR space according to the matched information. This results in a natural looking and spatially correct rendering of the remote user in the local augmented space and a significantly improved tele-conference experience and communication performance. Copyright 2015 John Wiley & Sons, Ltd.