多巴胺奖赏信号通过学习得到的价值期望塑造觅食决策
Dopaminergic reward signals shape foraging decisions through learned value expectations
神经系统的一项基本功能是引导动物趋向维持生命的资源，并使其具备获取这些资源的适应性策略。觅食，即从环境中获取这些资源所面临的挑战，需要做出在局部收益与成本之间进行权衡的决策，这为在生态情境下研究基于价值的决策打开了一扇窗口。然而，尽管觅食在生命中处于核心地位，觅食决策背后的算法与神经机制仍未得到充分理解。我们研究了小鼠在资源贫乏与资源丰富环境中、在动态的局部奖励结果下做出留驻或离开斑块的觅水觅食决策。我们发现，小鼠整合即时奖励、局部奖励历史与奖励（即价值）预期来指导觅食决策，倾向于在资源丰富环境的斑块中投入更多时间。计算模型揭示了随时间变化的偏差的影响，该偏差与习得价值函数的形状相吻合。与强化学习机制相一致，腹侧纹状体中的多巴胺信号表征了时间差分奖励预测误差，该信号可作为教学信号用于构建价值函数；而通过校准的光遗传学操控多巴胺以产生虚构奖励信号，重现了与价值相关的决策偏差。综上所述，我们的结果为了解觅食的神经计算基础提供了新的视角，揭示了价值预期在动态环境中的新作用。此外，这些结果确立了生物强化学习在生态情境中的泛化功能，证明了经典的 dopaminergic 奖励回路在塑造时间分辨的价值预期以指导行为与认知中的特定作用。这一框架与觅食的统一，为阐明动物如何学习并动态调整认知策略以应对现实世界生态问题的原理开辟了新途径。
把觅食的留走决策与多巴胺预测误差、光遗传伪造奖赏信号放在同一实验里，为强化学习如何塑造价值期望提供了行为与神经的双重证据。
A fundamental function of the nervous system is to direct animals towards life-sustaining resources and enable adaptive strategies for acquiring them. Foraging, the challenge of securing these resources from the environment, requires making decisions that balance local gains against costs, opening a window on value-based decision making in ecological context. Yet despite the centrality of foraging in life, the algorithms and neural mechanisms underlying foraging decisions are not well understood. We studied mice making stay-or-leave patch foraging decisions for water across poor- and rich-resource environments with dynamic local reward outcomes. We found that mice integrate immediate rewards, local reward history, and reward (i.e. value) expectations to guide foraging decisions, favoring greater time investment in rich-environment patches. Computational models revealed influence of a time-varying bias which mirrored the shape of a learned value function. Consistent with a reinforcement learning mechanism, dopamine in the ventral striatum signaled a temporal-difference reward prediction error, which could be used as a teaching signal to construct value functions, and calibrated optogenetic manipulation of dopamine to create fictive reward signals recapitulated the value-related decision bias. Taken together, our results shed new light on the neurocomputational basis of foraging, revealing a novel role for value expectations in dynamic environments. Further, they establish generalized function of biological reinforcement learning in ecological context, demonstrating a specific role for the canonical dopamine reward circuit in shaping temporally resolved value expectations to guide behavior and cognition. The unification of this framework with foraging opens a new avenue to elucidate principles of how animals learn and dynamically adjust cognitive strategies for real-world ecological problems.
中文翻译还没有生成，先显示原文。
来源：bioRxiv · Neuroscience · biorxiv.org