Abstract: With the explosive growth of video resource, efficient techniques for generating video summarization are appealing for facilitating understanding and presenting video content. Traditional video summarizations were usually given through extracting key frames based on the features of frames in video sequence. However, in many cases, the given key frames don't meet the key frames reside in the mind of users. In this paper, we propose an innovative approach based on relevance feedback to select the key frames of a video sequence for video summarization considering users' subjective visual preference. A two-step strategy to select the key frames is given. 1) we evaluate the preference of user, and a Bayesian method is used to update the probability in condition of all the responses; 2) we take the interaction of the frames into consideration and select a proper frame set for video summarization. We verified that the relevance in different people's mind is not totally irrelevant. A relevance distance based on the characteristics of the video frames and the trend of users' decision making is proposed for more accurate likelihood definition. Experiments showed that our approach could provide satisfied summarizations in acceptable iteration in most cases and demonstrated the efficiency of the interactive and feedback process.