Neural compositing for real-time augmented reality rendering in low-frequency lighting environments

作     者：Shengjie MA Qian SHEN Qiming HOU Zhong REN Kun ZHOU Shengjie MA;Qian SHEN;Qiming HOU;Zhong REN;Kun ZHOU

作者机构：State Key Lab of CAD&CGZhejiang University ZJU-FaceUnity Joint Lab of Intelligent Graphics 

出 版 物：《Science China(Information Sciences)》 (中国科学:信息科学(英文版))

年 卷 期：2021年第64卷第2期

页      面：139-153页

核心收录：

学科分类：12[管理学] 1201[管理学-管理科学与工程(可授管理学、工学学位)] 081104[工学-模式识别与智能系统] 08[工学] 080203[工学-机械设计及理论] 0835[工学-软件工程] 0802[工学-机械工程] 0811[工学-控制科学与工程] 0812[工学-计算机科学与技术（可授工学、理学学位）] 

主　　题：augmented reality neural networks differentiable renderer reflection shadow 

摘      要：We present neural compositing, a deep-learning based method for augmented reality rendering,which uses convolutional neural networks to composite rendered layers of a virtual object with a real photograph to emulate shadow and reflection effects. The method starts from estimating the lighting and roughness information from the photograph using neural networks, renders the virtual object with a virtual floor into color, shadow and reflection layers by applying the estimated lighting, and finally refines the reflection and shadow layers using neural networks and blends them with the color layer and input image to yield the output image. We assume low-frequency lighting environments and adopt PRT(precomputed radiance transfer) for layer rendering, which makes the whole pipeline differentiable and enables fast end-to-end network training with synthetic scenes. Working on a single photograph, our method can produce realistic reflections in a real scene with spatially-varying material and cast shadows on background objects with unknown geometry and material at real-time frame rates.