Our paper deals with active multiview object recognition focusing on the directional support of sequential multiple shots. Since inertial sensors are easily available nowadays, we propose the use of them to estimate the orientation change of the camera and thus to estimate the probability of relative poses. With the help of relative orientation change, we can compute transition probabilities between possible poses and can use a hidden Markov model to evaluate state (pose) sequences and can thus increase the recognition rate. Furthermore, we can plan our next viewing position to minimize the risk of misclassification, resulting in higher overall recognition rates. Besides giving the theoretical details, we use two datasets to illustrate the performance of our model through several tests including occlusion, blur, Gaussian noise, and to compare to a solution with a long short-term memory network.