Abstract: Skin color is a strong cue in vision-based human tracking. Skin detection has been widely used in various applications, such as face and hand tracking, people detection in the video databases. In this paper, we propose and develop an effective hand tracking method based on a simple color classification. This method includes two major procedures: training and tracking. In the training procedure, the user specifies a region on a hand to obtain the training data. Based on the skin-color distribution, the training data will be classified into several color clusters using randomized list data structure. In the hand tracking procedure, the hand will be segmented in real-time from the background using the randomized lists that have been trained in the training procedure. The proposed method has two advantages: (1) It is fast because the image segmentation algorithm is automatically performed on a small region surrounding the hand; and (2) It is robust under different lighting conditions because the lighting factor is not employed in our effective color classification. Several experiments have been conducted to validate the performance of the proposed method. This proposed method has good potential in many real applications, such as virtual reality or augmented reality systems.