Abstract: This paper presents an approach for video event inference from dozens of actions performed by multiple players. First, we constructed an And-Or graph to describe the different configurations of the event category such as shooting in soccer matches. We considered both temporal relations and role relations for the graph and encode them as vector parameters for each pair of graph nodes. Then, we developed an inference algorithm by using bottom-up and top-down processes. We found the proposals for each node during the bottom-up step by considering three terms of energies and refined the proposals during the top-down step by measuring the action-labeling similarity and the temporal misplacement penalty. The optimal proposal of the inferring event and its score are obtained as the result. In the experiments, we tested the inference performance of the approach for the shooting events on real soccer match videos. By our approach, we can infer different kinds of shooting events in one scenario and interpret them play-by-play in a flexible way.