A multimodality method for human activity recognition based on Multi-Stage Temporal Convolutional Network

Computer Engineering

: Skeleton data is among the most widely used for Human Activity Recognition (HAR) due to its advantages. However, it is difficult to meet the classification requirements using a single modality. Although some recent studies use more than one type of vision data, such as combining skeleton data with RGB or Depth and get some improvements, combinations being hopefull of data from ambient and wearable sensors are still rare. In this paper, we proposed a method for Human activity recognition based on Multi-Stage Temporal Convolutional Network (MS-TCNs) and a fusion of skeleton and acceleration acquired from Kinect sensors and inertial sensors, respectively. We firstly use Multi-Stage Temporal Convolutional Networks to model time series data of skeleton and acceleration. Feature vectors being outputs of MS-TCNs are then combined and passed through two fully connected layers to give labels. The experimental results demonstrated the expected effects of combining inertial sensor data and derived data of the visual data in the HAR problem. The proposed method is evaluated on a benchmark datasets for action recognition (UTD-MHAD dataset). The proposed method outperforms the state-of-the-art ones. That the proposed method reached 95.2 % with recognition rate provides the prospects of our proposed framework.

This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.