自主智能体的内在目标：分层探索捕捉果蝇未来的意志性动作
Intrinsic Goals for Autonomous Agents: Hierarchical Exploration Captures Future Volitional Action in Drosophila
动物具有一种非凡的能力，能够在没有外部感觉线索或外在目标的情况下，由内在动机驱动而自主行动，然而这一能力背后的计算机制在很大程度上仍是一个谜。为了理解生物自主性如何可能从优化内在目标中涌现，我们利用近期在果蝇（textit{Drosophila melanogaster}）中开展的全脑记录，这些记录能在超过 $10$ 秒之前识别出自发探索决策的神经表征。结合 textit{Drosophila} 的神经力学数字孪生，这些记录使我们能够检验：具有真实身体形态、由内在动机驱动的人工智能体，是否能在数量上与果蝇在探索过程中的行为和未来意志行动的内部控制表征相匹配。我们实现了一个分层的内在动机智能体，其策略在一组离散的运动基元库上运行——这些基元是驱动关节动力学的固定低层控制器——与行为音节的行为学理论高度吻合。我们使用涵盖好奇心、熵和能力方法的 $20$ 种内在动机算法，在没有外部奖励的情况下训练智能体。我们在一个仿照果蝇实验设计的虚拟拴系球实验中评估冻结的策略，从而能够在与果蝇中观察到的相同时间尺度上，比较自发运动统计量和内部表征，并从运动学和神经激活两方面解码未来的转向方向。我们发现，DIAYN——一种分层能力算法，它学习一组离散策略，以最大化行为熵及其诱导的感觉观测的可分离性——在与果蝇的联合行为和神经前瞻性编码性能上表现最强。令人惊讶的是，DIAYN 学到的策略密切捕捉了果蝇探索策略中的个体差异，这表明这种行为多样性优化了一个具体的计算目标，即在 t 时刻最大化状态空间覆盖。
把果蝇全脑记录与数字孪生结合，用 20 种内在动机算法检验哪一类能同时匹配自发探索的行为统计和未来转向的神经表征。
Animals have a remarkable ability to act autonomously---driven by intrinsic motivation---without external sensory cues or extrinsic objectives, yet the computational mechanisms behind this ability largely remain a mystery. To understand how biological autonomy may emerge from optimizing intrinsic goals, we use recent whole-brain recordings in the fruit fly (textit{Drosophila melanogaster}) that identify neural representations of spontaneous exploratory decisions more than $10$ seconds in advance. Combined with a neuromechanical digital twin of textit{Drosophila}, these recordings allow us to test whether intrinsically motivated artificial agents with realistic embodiments quantitatively match fly behavior and internal representations of future volitional action during exploration. We implement a hierarchical, intrinsically motivated agent whose policy operates over a discrete library of motor primitives---fixed low-level controllers that actuate joint dynamics---closely matching ethological theories of behavioral syllables. We train agents with no external reward using $20$ intrinsic motivation algorithms spanning curiosity, entropy, and competency methods. We evaluate frozen policies in a virtual tethered-ball assay modeled after the fruit-fly experiment, allowing us to compare spontaneous locomotor statistics and internal representations and decode future turn direction from both kinematics and neural activations on the same timescales observed in flies. We find that DIAYN---a hierarchical competency algorithm that learns a discrete set of policies maximizing behavioral entropy and the separability of their induced sensory observations---achieves the strongest joint behavioral and neural prospective coding performance with the fruit fly. Surprisingly, DIAYN's learned policies closely capture individual variability in fly exploration strategies, suggesting that this behavioral diversity optimizes a concrete computational goal of maximizing state-space coverage at t
中文翻译还没有生成，先显示原文。
来源：bioRxiv · Neuroscience · biorxiv.org