EgoCaptioner targets detailed, timestamped egocentric video narration
EgoCaptioner combines a video-level overview with timed narration of activities in first-person footage. Its authors built a coarse-to-fine annotation pipeline that produced 50,000 training examples from eight video sources and refined captions through Group Relative Policy Optimization. The paper reports leading results among evaluated open-source models across five egocentric benchmarks, as well as gains over Gemini 3.1 Pro on PhysCapBench and the tested proprietary baselines on Minerva-Ego.
The system addresses a known weakness of multimodal language models in first-person video: missing brief actions, confusing their order, and misreading wearer motion.
Computer Science > Computer Vision and Pattern Recognition
arXiv:2611.26380 (cs)
[Submitted on 19 Nov 2026]
Title:EgoCaptioner: Towards Comprehensive and Fine-Grained Egocentric Video Captioning
Authors:Yuqi Tang, Yang Shi, Yuran Wang, Tengfei Liu, Rundong Wang, Zongyang Qiu, Qixun Wang, Zhuoran Zhang, Xuanyu Zhu, Yifan Xu, Yiyan Ji, Shujie Li, Shixuan Liu, Haotian Wang, Yuanxing Zhang, Pengfei Wan, Chengwei Qin
View a PDF of the paper titled EgoCaptioner: Towards Comprehensive and Fine-Grained Egocentric Video Captioning, by Yuqi Tang and 16 other authors
#EgoCaptioner#Gemini 3.1 Pro#PhysCapBench#Minerva-Ego#Group Relative Policy Optimization