Abstract: In this paper we represent the object with multiple attentional blocks which reflect some findings of selective visual attention in human perception. The attentional blocks are extracted using a branch-and-bound search method on the saliency map, and meanwhile the weight of each block is determined. Independent particle filter tracking is applied to each attentional block and the tracking results of all the blocks are then combined in a linear weighting scheme to get the location of the entire target object. The attentional blocks are propagated to the object location found in each new frame and the state of the most likely particle in each block is also updated with the new propagated position. In addition, to avoid error accumulation caused by the appearance variations, the object template and the positions of the attentional blocks are adaptively updated while tracking. Experimental results show that the proposed algorithm is able to efficiently track salient objects and is better accounted for partial occlusions and large variations in appearance.