UI layers merger: merging UI layers via visual learning and boundary prior

作     者：Yunnong CHEN Yankun ZHEN Chuning SHI Jiazhi LI Liuqing CHEN Zejian LI Lingyun SUN Tingting ZHOU Yanfang CHANG Yunnong CHEN;Yankun ZHEN;Chuning SHI;Jiazhi LI;Liuqing CHEN;Zejian LI;Lingyun SUN;Tingting ZHOU;Yanfang CHANG

作者机构：School of Software TechnologyZhejiang UniversityHangzhou 310027China College of Computer Science and TechnologyZhejiang UniversityHangzhou 310027China Alibaba-Zhejiang University Joint Research Institute of Frontier TechnologiesHangzhou 310027China Alibaba GroupHangzhou 311121China Zhejiang-Singapore Innovation and AI Joint Research LabHangzhou 310027China 

出 版 物：《Frontiers of Information Technology & Electronic Engineering》 (信息与电子工程前沿（英文版）)

年 卷 期：2023年第24卷第3期

页      面：373-387页

核心收录：

学科分类：08[工学] 080203[工学-机械设计及理论] 0802[工学-机械工程] 

基　　金：Project supported by the National Key R&D Program of China(No.2018AAA0100703) the National Natural Science Foundation of China(Nos.62006208 and 62107035) the Ng Teng Fong Charitable Foundation in the form of ZJU-SUTD IDEA Grant the Alibaba-Zhejiang University Joint Research Institute of Frontier Technologies 

主　　题：User interface(UI)to code UI design lint UI layer merging Object detection 

摘      要：With the fast-growing graphical user interface(GUI)development workload in the Internet industry,some work attempted to generate maintainable front-end code from GUI *** can be more suitable for using user interface(UI)design drafts that contain UI ***,fragmented layers inevitably appear in the UI design drafts,which greatly reduces the quality of the generated *** of the existing automated GUI techniques detects and merges the fragmented layers to improve the accessibility of generated *** this paper,we propose UI layers merger(UILM),a vision-based method that can automatically detect and merge fragmented layers into UI *** UILM contains the merging area detector(MAD)and a layer merging *** MAD incorporates the boundary prior knowledge to accurately detect the boundaries of UI ***,the layer merging algorithm can search for the associated layers within the components’boundaries and merge them into a *** present a dynamic data augmentation approach to boost the performance of *** also construct a large-scale UI dataset for training the MAD and testing the performance of *** results show that the proposed method outperforms the best baseline regarding merging area detection and achieves decent layer merging accuracy.A user study on a real application also confirms the effectiveness of our UILM.