Edge-assisted Activity Recognition Using Skeletal Representation and Deep Learning for Video Surveillance

Faculty Development Competitive Research Grants Program 2022-2024

This project brings together the research area of computer vision, deep learning, and edge computing for interdisciplinary collaboration. While in recent years, there has been a great deal of interest from scientists and researchers in these areas, there has been little work on the design and development of an intelligent surveillance system that integrates cutting-edge technologies for efficient activity recognition. More specifically, the contributions of this project can be stated as (1) A HAR framework is developed by fully exploiting the benefits of human skeleton information (i.e., privacy protection and latency reduction) in IoT environments, (2) A comprehensive study of feature extraction schemes is provided to investigate the effectiveness and ability of different feature types in capturing dynamics of body joints in both spatial and temporal domains, (3) Depending on the type of skeletal features, we design various lightweight DL models for inferring human activities on resourced-constrained devices in two directions: CNNs for learning fixed-size video representation and deep sequential networks (RNN and Transformer) for learning dynamics of variable-length sequences, and (4) We propose a collaborative computation offloading algorithm to achieve the load balancing among the edge servers and to meet latency and accuracy requirements of an edge HAR system. Hence, the project has both high scientific and commercial potential due to the high demand for large-scale surveillance applications for industry, healthcare, risk monitoring, and automated assisted living systems.