论文标题
多模式神经网络更好地解释海马中的多毒素模式
Multimodal neural networks better explain multivoxel patterns in the hippocampus
论文作者
论文摘要
人类海马具有“概念细胞”，即在表现出属于特定概念的刺激时发射的神经元，而不管方式如何。最近，在称为夹子的多模式网络中发现了类似的概念细胞（Radford et。，2021）。在这里，我们询问夹子是否可以比纯粹的视觉（或语言）模型更好地解释人类海马的功能磁共振成像活性。我们将分析扩展到一系列公开可用的单模式模型。我们证明，在评估网络解释海马中多毒素活性的能力时，“多模式”是一个关键组成部分。
The human hippocampus possesses "concept cells", neurons that fire when presented with stimuli belonging to a specific concept, regardless of the modality. Recently, similar concept cells were discovered in a multimodal network called CLIP (Radford et at., 2021). Here, we ask whether CLIP can explain the fMRI activity of the human hippocampus better than a purely visual (or linguistic) model. We extend our analysis to a range of publicly available uni- and multi-modal models. We demonstrate that "multimodality" stands out as a key component when assessing the ability of a network to explain the multivoxel activity in the hippocampus.
