SeeMore: a spatiotemporal predictive model with bidirectional distillation and level-specific meta-adaptation

作     者：Yuqing MA Wei LIU Yajun GAO Yang YUAN Shihao BAI Haotong QIN Xianglong LIU 

作者机构：State Key Lab of Software Development Environment Beihang University 

出 版 物：《Science China(Information Sciences)》 (中国科学:信息科学(英文版))

年 卷 期：2024年第67卷第8期

页      面：147-171页

核心收录：

学科分类：12[管理学] 1201[管理学-管理科学与工程(可授管理学、工学学位)] 0808[工学-电气工程] 081104[工学-模式识别与智能系统] 08[工学] 0835[工学-软件工程] 0811[工学-控制科学与工程] 0812[工学-计算机科学与技术（可授工学、理学学位）] 

基　　金：supported by National Natural Science Foundation of China (Grant Nos. 62206010  62022009) 

主　　题：spatiotemporal predictive learning knowledge transfer bidirectional distillation network level-specific meta-adapter coarse-to-fine training 

摘      要：Predicting future frames using historical spatiotemporal data sequences is challenging and critical, and it is receiving a lot of attention these days from academic and industrial scholars. Most spatiotemporal predictive algorithms ignore the valuable backward reasoning ability and the disparate learning complexities among different layers and hence, cannot build good long-term dependencies and spatial correlations,resulting in suboptimal solutions. To address the aforementioned issues, we propose a two-stage coarse-to-fine spatiotemporal predictive model with bidirectional distillation and level-specific meta-adaptation(See More)in this paper, which includes a bidirectional distillation network(BDN) and a level-specific meta-adapter(LMA), to gain bidirectional multilevel reasoning. In the first stage, BDN concentrates on bidirectional dynamics modeling and coarsely constructs spatial correlations of different layers, while LMA is introduced in the second fine-tuning stage to refine the multilevel spatial correlations from a meta-learning *** particular, BDN mimics the forward and backward reasoning abilities of humans in a distillation manner,which aids in the development of long-term dependencies. The LMA views learning of different layers as disparate but related tasks and guides the transfer of learning experiences among these tasks through learning complexities. Thus, each layer could be closer to its solutions and could extract more informative spatial correlations. By capturing the enhanced short-term spatial correlations and long-term temporal dependencies,the proposed model could extract adequate knowledge from sequential historical observations and accurately predict future frames whose backtracking preconditions are consistent with the historical sequence. Our work is general and robust enough to be integrated into most spatiotemporal predictive models without requiring additional computation or memory cost during inference. Extensive experimen