Abstract: Pervasive games have become a popular field of investigation in recent years, in which natural human computer interaction (HCI) plays a key role. In this paper, a vision-based approach for human hand motion gesture recognition is proposed for natural HCI in pervasive games. A LED light pen is used to indicate the user's hand position, while a web-camera is used to capture the hand motion. A rule-based approach is used to design a set of hand gestures which are classified into two categories: linear gestures and arc-shaped gestures. A determinate finite state automaton is developed to segment the captured hand motion trajectories. The proposed interaction method has been applied to the traditional game Tetris on PC with the hand-held LED light pen being used to drive the game instead of traditional key strokes. Experimental results show that the vision-based interactions are natural and effective.