An end-to-end generative framework for video segmentation and recognition

H. Kuehne, J. Gall and T. Serre

Proc. of IEEE Winter Conference on Applications of Computer Vision (WACV), 2016

Abstract

We describe an end-to-end generative approach for the segmentation and recognition of human activities. In this approach, a visual representation based on reduced Fisher Vectors is combined with a structured temporal model for recognition. We show that the statistical properties of Fisher Vectors make them an especially suitable front-end for generative models such as Gaussian mixtures. The system is evaluated for both the recognition of complex activities as well as their parsing into action units. Using a variety of video datasets ranging from human cooking activities to animal behaviors, our experiments demonstrate that the resulting architecture outperforms state-of-the-art approaches for larger datasets, i.e. when sufficient amount of data is available for training structured generative models.

Code

Current version of the system is available on GitHub: https://github.com/hildekuehne/HTK_actionRecognition

If you use this code and/or data in your project, please cite:

@InProceedings{Kuehne16end,
author = {Hilde Kuehne and Juergen Gall and Thomas Serre},
title = {An end-to-end generative framework for video segmentation and recognition},
booktitle = {Proc. IEEE Winter Applications of Computer Vision Conference (WACV 16)},
year = {2016},
month = {Mar},
address = {Lake Placid},
}

Data

The evaluation was done on the following datasets: ADL, Olympic, ToyAssembly, CMU-MMAC, MPIICooking, 50Salads, Breakfast and CRIM13. We provide the reduced FV representation and the segmentation used (.xml and .txt). We also provide precomputed dense trajectories (using the code availlable here) or improved dense trajectories (using the code availlable here) on request, as long as they are not included at the dataset website. Please contact me (kuehne @ iai . uni-bonn . de) .