With the large number of surveillance cameras now in operation, both in public placesand in commercial centers, significant research efforts have been invested in attempts to automate surveillance video analysis. The goal of visual surveillance is not only to put cameras in the place of human eyes, but also to accomplish the entire surveillance task as automatically as possible. Recently, the problem of analyzing behavior in videos has been the focus of several researchers’ efforts. They concentrate on developing intelligent visual surveillance systems to replace traditional passive video surveillance systems which can only store surveillance videos but are not able to identify or describe interesting activities. In this paper, we give a survey of behavior analysis work in video surveillance and compare the performance of the state-of-the-art algorithms on different datasets.Furthermore, useful datasets are analyzed in order to provide help for initiating research projects.