Video surveillance is recently one of the most active research topics in computer vision. It has a wide spectrum of promising public safety and security applications. As the number of cameras exceeds the capability of human operators to monitor them, the traditional passive video surveillance is proving ineffective. Hence, converting to intelligent visual surveillance is inevitable. Intelligent visual surveillance aims to detect, recognize and track certain objects from image sequences automatically, and more generally to understand and describe object behaviors. Many researchers have contributed to the field of automated video surveillance through detection, classification, and tracking algorithms. Despite recent progress in computer vision and other related areas, there are still major technical challenges to be overcome before reliable automated video surveillance can be realized. Recently, the problem of analyzing behavior in videos has been the focus of several researchers’ efforts. It aims to analyze and interpret individual behaviors and interactions between different objects found in the scene. Hence, obtaining a description of what is happening in a monitored area, and then taking appropriate action based on that interpretation. In this paper, we give a survey of behavior analysis work in video surveillance and compare the performance of the state-of-the-art algorithms on different datasets. Moreover, useful datasets are analyzed in order to provide help for initiating research projects.