论文标题
视觉常识R-CNN
Visual Commonsense R-CNN
论文作者
论文摘要
我们提出了一种新型的无监督特征表示方法，即基于视觉常识区域的卷积神经网络（VC R-CNN），可作为改进的视觉区域编码器，用于高级任务，例如字幕和VQA。给定图像中一组检测到的对象区域（例如，使用更快的R-CNN），就像任何其他无监督的特征学习方法（例如Word2Vec）一样，VC R-CNN的代理训练目标是预测区域的上下文对象。但是，它们在根本上是不同的：VC R-CNN的预测是通过因果干预措施：P（y | do（x）），而其他人则使用常规可能性：P（y | x）。这也是VC R-CNN可以学习“理解”知识（例如椅子）的核心原因 - 而如果观察到桌子，则可能存在“常见”的共同出现（例如椅子）。我们广泛地将VC R-CNN功能应用于三个流行任务的典型模型：图像字幕，VQA和VCR，并在其中观察到一致的性能提升，从而实现了许多新的最新技术。代码和功能可在https://github.com/wangt-cn/vc-r-cnn上找到。
We present a novel unsupervised feature representation learning method, Visual Commonsense Region-based Convolutional Neural Network (VC R-CNN), to serve as an improved visual region encoder for high-level tasks such as captioning and VQA. Given a set of detected object regions in an image (e.g., using Faster R-CNN), like any other unsupervised feature learning methods (e.g., word2vec), the proxy training objective of VC R-CNN is to predict the contextual objects of a region. However, they are fundamentally different: the prediction of VC R-CNN is by using causal intervention: P(Y|do(X)), while others are by using the conventional likelihood: P(Y|X). This is also the core reason why VC R-CNN can learn "sense-making" knowledge like chair can be sat -- while not just "common" co-occurrences such as chair is likely to exist if table is observed. We extensively apply VC R-CNN features in prevailing models of three popular tasks: Image Captioning, VQA, and VCR, and observe consistent performance boosts across them, achieving many new state-of-the-arts. Code and feature are available at https://github.com/Wangt-CN/VC-R-CNN.
