Pyramid-VAE-GAN:Transferring hierarchical latent variables for image inpainting

作     者：Huiyuan Tian Li Zhang Shijian Li Min Yao Gang Pan Huiyuan Tian;Li Zhang;Shijian Li;Min Yao;Gang Pan

作者机构：College of Computer Science and TechnologyZhejiang UniversityHangzhou 310027China Advanced Technology Research InstituteZhejiang UniversityHangzhou 310027China 

出 版 物：《Computational Visual Media》 (计算可视媒体(英文版))

年 卷 期：2023年第9卷第4期

页      面：827-841页

核心收录：

学科分类：08[工学] 080203[工学-机械设计及理论] 0802[工学-机械工程] 0835[工学-软件工程] 

基　　金：The authors gratefully acknowledge the financial support of the National Natural Science Foundation of China(Grant No.61925603). 

主　　题：image inpainting variational autoencoder(VAE) latent variable transfer(LTN) pyramid structure generative model 

摘      要：Significant progress has been made in image inpainting methods in recent years.However,they are incapable of producing inpainting results with reasonable structures,rich detail,and sharpness at the same time.In this paper,we propose the Pyramid-VAE-GAN network for image inpainting to address this limitation.Our network is built on a variational autoencoder(VAE)backbone that encodes high-level latent variables to represent complicated high-dimensional prior distributions of images.The prior assists in reconstructing reasonable structures when inpainting.We also adopt a pyramid structure in our model to maintain rich detail in low-level latent variables.To avoid the usual incompatibility of requiring both reasonable structures and rich detail,we propose a novel cross-layer latent variable transfer module.This transfers information about long-range structures contained in high-level latent variables to low-level latent variables representing more detailed information.We further use adversarial training to select the most reasonable results and to improve the sharpness of the images.Extensive experimental results on multiple datasets demonstrate the superiority of our method.Our code is available at https://***/thy960112/Pyramid-VAE-GAN.