CatNet: Class Incremental 3D ConvNets for Lifelong Egocentric Gesture Recognition
Citation:
Wang, Z., She, Q., Chalasani, T., Smolic, A., "CatNet: Class Incremental 3D ConvNets for Lifelong Egocentric Gesture Recognition," 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Seattle, WA, USA, 2020, pp. 935-944Download Item:
Abstract:
Egocentric gestures are the most natural form of communication for humans to interact with wearable devices such as VR/AR helmets and glasses. A major issue in such scenarios for real-world applications is that may easily become necessary to add new gestures to the system e.g., a proper VR system should allow users to customize gestures incrementally. Traditional deep learning methods require storing all previous class samples in the system and training the model again from scratch by incorporating previous samples and new samples, which costs humongous memory and significantly increases computation over time. In this work, we demonstrate a lifelong 3D convolutional framework - c(C)la(a)ss increment(t)al net(Net)works (CatNet), which considers temporal information in videos and enables life-long learning for egocentric gesture video recognition by learning the feature representation of an exemplar set selected from previous class samples. Importantly, we propose a two-stream CatNet, which deploys RGB and depth modalities to train two separate networks. We evaluate Cat- Nets on a publicly available dataset - EgoGesture dataset, and show that CatNets can learn many classes incrementally over a long period of time. Results also demonstrate that the two-stream architecture achieves the best performance on both joint training and class incremental training compared to 3 other one-stream architectures. The codes and pre-trained models used in this work are provided at https://github.com/villawang/CatNet.
Sponsor
Grant Number
Science Foundation Ireland (SFI)
15/RP/2776
Author's Homepage:
http://people.tcd.ie/smolica
Author: Smolic, Aljosa
Sponsor:
Science Foundation Ireland (SFI)Other Titles:
Conference on Computer Vision and Pattern Recognition 2020 (CVPR 2020), 2020.Type of material:
Conference PaperCollections
Availability:
Full text availableKeywords:
Videos, Task analysis, Three-dimensional displays, Computer architecture, Training, Spatiotemporal phenomena, Gesture recognitionSubject (TCD):
Creative Technologies , Digital Engagement , Data Analysis , Information technology in education , Multimedia & CreativityDOI:
10.1109/CVPRW50498.2020.00123Metadata
Show full item recordLicences: