Abstract: In this paper we present a purely vision based implementation of markerless hand detection and tracking system which effectively detects and tracks the hand positions irrespective of hand orientation. A shape based detection algorithm using a new line approximation technique followed by an adaptive minimum distance classifier based tracking technique is implemented. The technique is very generic and can be practically used in all types of immersive and semi-immersive environments like simulations, three-dimensional (3D) games, visually assisted medical surgery and other human-computer interactive applications. The proposed technique overcomes certain limitations of glove based, handheld object based and marker based techniques for a truly immersive experience. The technique also overcomes the main limitation of previous vision based hand tracking systems by providing much larger degrees of freedom essential for effective virtual reality systems. The novelty of the technique proposed is that it separates the detection and tracking parts and therefore allows tracking of multiple hands in a 3D space for real-time interaction.