Abstract: We propose a novel framework in the context of structure and motion for representing and analyzing three-dimensional motions particularly for human heads and faces. They are captured via a stereo camera system and a scene graph is constructed that contains low and high-level vision information. It represents and describes the observed scene of each frame. By creating graphs of successive frames it is possible to match, track and segment main important features and objects as a structure of each scene and reconstruct these features into the three dimensional space. The cue-processor extracts feature information like 2D- and 3D-position, velocity, age, neighborhood, condition, or relationship among features that are stored in the vertices and weights of the graph to improve the estimation and detection of the features and/or objects in the continuous frames. The structure and change of the graph leads to a robust determination and analysis of changes in the scene and to segment and determine these changes even for temporal and partial occluded objects over a long image sequence.