论文标题
反对：（con）跨模式视频检索的文本（tra）nsformer
ConTra: (Con)text (Tra)nsformer for Cross-Modal Video Retrieval
论文作者
论文摘要
在本文中，我们重新检查了跨模式剪辑句子检索的任务，其中剪辑是较长的未修剪视频的一部分。当剪辑短或视觉上模棱两可时，可以使用其本地时间上下文（即周围的视频段）的了解来改善检索性能。我们提出上下文变压器（Contra）；一个编码器体系结构，该体系结构对视频剪辑及其本地时间上下文之间的相互作用进行建模，以增强其嵌入式表示形式。重要的是，我们使用跨模式嵌入空间中的对比损失来监督上下文变压器。我们探索视频和文本方式的上下文变压器。结果始终证明了三个数据集的性能提高：YouCook2，Epic-Kitchens和clip-Sentence版本的ActivityNet字幕。详尽的消融研究和上下文分析表明该方法的功效。
In this paper, we re-examine the task of cross-modal clip-sentence retrieval, where the clip is part of a longer untrimmed video. When the clip is short or visually ambiguous, knowledge of its local temporal context (i.e. surrounding video segments) can be used to improve the retrieval performance. We propose Context Transformer (ConTra); an encoder architecture that models the interaction between a video clip and its local temporal context in order to enhance its embedded representations. Importantly, we supervise the context transformer using contrastive losses in the cross-modal embedding space. We explore context transformers for video and text modalities. Results consistently demonstrate improved performance on three datasets: YouCook2, EPIC-KITCHENS and a clip-sentence version of ActivityNet Captions. Exhaustive ablation studies and context analysis show the efficacy of the proposed method.
