SmartPaint:a co-creative drawing system based on generative adversarial networks

作     者：Lingyun SUN Pei CHEN Wei XIANG Peng CHEN Wei-yue GAO Ke-jun ZHANG 

作者机构：Key Laboratory of Design Intelligence and Digital Creativity of Zhejiang ProvinceHangzhou 310027China State Key Lab of CAD&CGZhejiang UniversityHangzhou 310058China Alibaba-Zhejiang University Joint Institute of Frontier TechnologiesHangzhou 310027China 

出 版 物：《Frontiers of Information Technology & Electronic Engineering》 (信息与电子工程前沿（英文版）)

年 卷 期：2019年第20卷第12期

页      面：1644-1656页

核心收录：

学科分类：081203[工学-计算机应用技术] 08[工学] 0835[工学-软件工程] 0812[工学-计算机科学与技术（可授工学、理学学位）] 

基　　金：the National Science and Technology Innovation 2030 Major Project of the Ministry of Science and Technology of China(No.2018AAA0100703) the National Natural Science Foundation of China(No.61672451) the Provincial Key Research and Development Plan of Zhejiang Province,China(No.2019C03137) the China Postdoctoral Science Foundation(No.2018M630658) the Ng Teng Fong Charitable Foundation in the form of ZJU-SUTD IDEA Grant 

主　　题：Co-creative drawing Deep learning Image generation 

摘      要：Artificial intelligence(AI) has played a significant role in imitating and producing large-scale designs such as e-commerce banners. However, it is less successful at creative and collaborative design outputs. Most humans express their ideas as rough sketches, and lack the professional skills to complete pleasing paintings. Existing AI approaches have failed to convert varied user sketches into artistically beautiful paintings while preserving their semantic concepts. To bridge this gap, we have developed Smart Paint, a co-creative drawing system based on generative adversarial networks(GANs), enabling a machine and a human being to collaborate in cartoon landscape painting. Smart Paint trains a GAN using triples of cartoon images, their corresponding semantic label maps, and edge detection maps. The machine can then simultaneously understand the cartoon style and semantics, along with the spatial relationships among the objects in the landscape images. The trained system receives a sketch as a semantic label map input, and automatically synthesizes its edge map for stable handling of varied sketches. It then outputs a creative and fine painting with the appropriate style corresponding to the human’s sketch. Experiments confirmed that the proposed Smart Paint system successfully generates high-quality cartoon paintings.