Abstract: Most of current background subtraction algorithms have issues of ghost and foreground aperture when they process the crowded video sequences in the outdoor scenes. In this paper we present a novel method based on the pixel state to solve the issues. Every pixel in a video steam is assumed to own two different states --- active or inactive. Via the pixel state, we divide the whole observing time into many short units. Meanwhile, a new concept, confidence, is proposed to measure the significance of each cluster. By observing small units of time, our method automatically selects the clusters with the highest confidence as the background model. The experimental results show our method not only provides the accurate motion detection of crowded video sequences, but also handles the light change and performs in real time.