2023 · 10 citations · 15 references
ComputingComputer ScienceRobotics EngineeringComputer VisionMachine LearningRoboticsAction RecognitionDeep LearningMotion AnalysisMotion EstimationMotion SensorsMultimodal InteractionGroundingJoint Representation LearningMotion Representation LearningMultimodal Machine LearningSensor Data Processing
We present IMU2CLIP, a novel pre-training approach to align Inertial Measurement Unit (IMU) motion sensor recordings with text and video, by projecting them into the joint representation space of Contrastive Language-Image Pre-training (CLIP). The proposed approach allows IMU2CLIP to translate human motions (as measured by IMU sensors) into their corresponding textual descriptions and videos – while preserving the transitivity across these modalities. We introduce several new IMU-based Wearable AI applications such as motion-based media search, or an LM-based multimodal reasoning with motion sensor data – all using text as the grounding platform. In addition, we show that IMU2CLIP significantly improves downstream performances when fine-tuned for each application, demonstrating its universal usage as a new pre-trained resource. Our code and models will be released publicly.
15
Transformers: State-of-the-Art Natural Language Processing
Thomas Wolf, Lysandre Debut, Victor Sanh et al. · 2020 · 8K citations · Full text
A Simple Framework for Contrastive Learning of Visual Representations
Ting Chen, Simon Kornblith, Mohammad Norouzi et al. · arXiv (Cornell University) · 2020 · 7.3K citations · Full text
GPT-3: Its Nature, Scope, Limits, and Consequences
Luciano Floridi, Massimo Chiriatti · Minds and Machines · 2020 · 2.2K citations · Full text