2023 · 31 citations · 56 references
ComputingAutonomous SystemsForecastingMachine LearningReal-time SystemsAction RecognitionAction RepresentationReal-time ForecastingTime Series PredictionDeep Learning For ForecastingForecasting AccuracyLatency MetricsLong-term HorizonOnline Action RecognitionTemporal Convolutional NetworksTransformer ModelsTransformer-based Prediction Models
We present RAFTformer, a realtime action forecasting transformer for latency-aware realworld action forecasting. RAFTformer is a two-stage fully transformer based architecture comprising of a video transformer backbone that operates on high resolution, short-range clips, and a head transformer encoder that temporally aggregates information from multiple short-range clips to span a long-term horizon. Additionally, we propose a novel self-supervised shuffled causal masking scheme as a model level augmentation to improve forecasting fidelity. Finally, we also propose a novel realtime evaluation setting for action fore-casting that directly couples model inference latency to overall forecasting performance and brings forth a hith-erto overlooked trade-off between latency and action fore-casting performance. Our parsimonious network design fa-cilitates RAFTformer inference latency to be 9x smaller than prior works at the same forecasting accuracy. Owing to its two-staged design, RAFTformer uses 94% less training compute and 90% lesser training parameters to outperform prior state-of-the-art baselines by 4.9 points on EGTEA Gaze+ and by 1.4 points on EPIC-Kitchens-100 validation set, as measured by Top-5 recall (T5R) in the offline setting. In the realtime setting, RAFTformer outperforms prior works by an even greater margin of upto 4.4 T5R points on the EPIC-Kitchens-100 dataset. Project Webpage: https://karttikeya.github.io/publication/RAFTformer/.
56
Sepp Hochreiter, Jürgen Schmidhuber · Neural Computation · 1997 · 98.1K citations
Focal Loss for Dense Object Detection
Tsung-Yi Lin, Priya Goyal, Ross Girshick et al. · 2017 · 25.7K citations
SlowFast Networks for Video Recognition
Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik et al. · 2019 · 3.6K citations
Ze Liu, Ning Jia, Yue Cao et al. · 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) · 2022 · 1.9K citations