Temporal Order-Preserving Dynamic Quantization for Human Action Recognition

Temporal Order-Preserving Dynamic Quantization for Human Action Recognition from Multimodal Sensor Streams Jun Ye Kai Li Guo-Jun Qi Kien A. Hua University of Central Florida Outline Background Problem, existing methods, challenges

Our algorithm Dynamic Temporal Quantization Multimodal Feature Fusion Performance study MSR-Action3D UTKinect-Action MSR-ActionPairs Conclusions 2 Background Depth sensors becomes affordable and popular New human-computer interaction

Gesture recognition Speech recognition Application domain Video games, education, business, healthcare 3 Problem and Challenges Key problem: modeling the temporal dynamics of 3D human action/gestures Existing methods

Histogram-based methods do not preserve order (bagof-3d-words [5, 21], HOJ3D [16], HON4D [9] ) Temporal modeling suffer from video misalignment (motion template [7, 20], temporal pyramid [9, 14]) Challenge: temporal misalignment due to Temporal translation Execution rate variation 4 Objective Modeling the temporal patterns of 3D actions according to the transition of sub-actions satisfying 1. Frames with similar postures are clustered together (sub-action constraint) 2. Temporal order of the sequence must be preserved (order-preserving)

Dynamic Temporal Quantization Algorithm 5 Dynamic Temporal Quantization Quantization: videos X1,X2, Xn of varied length n quantized vector V1,V2,Vm of fixed length m. Optimal frame assignment a Objective function: Optimal quantization can be obtained by jointly optimizing a and V 6 Dynamic Temporal Quantization (contd)

Nontrivial to jointly solve the frame assignment a Initialization: uniform partition Aggregation step: given fixed assignment a, vj is computed by the aggregation Assignment step: fixed the quantized vector V, update the assignment a by DTW Iterate until convergence. 7 Hierarchical representation Multilayers of the Dynamic Quantization

Top layers: global temporal patterns Bottom layers: local temporal patterns Concatenate all layers 8 Multimodel Feature Fusion Multimodal features:

joint coordinate pairwise angle joint offset [21] histogram of velocity components (HVC) Supervised learning for all quantized vectors Multiclass SVM Fusion by regression (softmax) 9 Experiments Experiments on three public 3D human action datasets MSR-Action3D

UTKinect-Action MSR-ActionPairs 10 Experiment: dynamic quantization VS deterministic Dynamic quantization quantization outperforms deterministic MSR-Action3D dataset quantization. Accuracy Feature Dynamic quantization Deterministic quantization position

81.61% 76.24% angle 73.95% 71.65% offset 73.95% 68.20% velocity 80.84% 72.80% fused

90.42% 83.15% Similar performances can be observed in the other two datasets. 11 Experiment: hierarchical representation MSR-Action3D dataset with the joint coordinate feature Layers 1 2 3 4 5 Accuracy

66.28% 67.82% 71.26% 81.61% 77.39% More layers generally produce higher accuracy though need to take care of the overfitting. 12 Experiment: Comparison with state-of-the-art results Method Accuracy Actionlet Ensemble [14] HON4D [9]

DCSF [15] Lie Group [13] Super Normal Vector [18] Proposed method 88.2% 88.89% 89.3% 89.48% 93.09% 90.42% MSR-Action3D dataset Method Accuracy Actionlet Ensemble [14] HON4D [9] HON4D + Ddisc [9] Super Normal Vector [18] Proposed method

82.22% 93.33% 96.67% 98.89% 93.71% MSR-ActionPairs dataset Method Accuracy Histogram of 3D joints [17] Combined features with random forest [21] Lie Group [13] Proposed method UTKinect-Action dataset (100% accuracy) 13 90.92% 91.9% 97.08%

100% Conclusions A novel algorithm for 3D human action sequence recognition from the perspective of dynamic temporal quantization. Extensive experiments on three public datasets demonstrate the effectiveness of the proposed technique for temporal modeling. 14 Thank you. Questions? 15