论文标题
学习互动和电影角色之间的关系
Learning Interactions and Relationships between Movie Characters
论文作者
论文摘要
人们之间的互动通常受其关系的支配。另一方面，社会关系建立在几种互动之上。随着时间的流逝，两个陌生人更有可能在成为朋友的同时向自己介绍和自我介绍。我们对互动与人际关系之间的这种相互作用着迷，并认为这是理解社会情况的重要方面。在这项工作中，我们提出了神经模型，以学习并共同预测涉及的相互作用，关系和一对角色。我们注意到，交互是通过视觉和对话线索的混合来告知的，并提出了多模式体系结构，以从它们中提取有意义的信息。将视频中的一对互动字符定位是一个耗时的过程，我们训练模型从剪贴层弱标签中学习。我们在MovieGraphs数据集上评估了我们的模型，并展示了与地面真相标签相比，使用较长的时间上下文来预测关系的影响，并使用弱标签来实现令人鼓舞的性能。代码在线。
Interactions between people are often governed by their relationships. On the flip side, social relationships are built upon several interactions. Two strangers are more likely to greet and introduce themselves while becoming friends over time. We are fascinated by this interplay between interactions and relationships, and believe that it is an important aspect of understanding social situations. In this work, we propose neural models to learn and jointly predict interactions, relationships, and the pair of characters that are involved. We note that interactions are informed by a mixture of visual and dialog cues, and present a multimodal architecture to extract meaningful information from them. Localizing the pair of interacting characters in video is a time-consuming process, instead, we train our model to learn from clip-level weak labels. We evaluate our models on the MovieGraphs dataset and show the impact of modalities, use of longer temporal context for predicting relationships, and achieve encouraging performance using weak labels as compared with ground-truth labels. Code is online.
