Full Text:   <245>

CLC number: 

On-line Access: 2022-09-28

Received: 2022-03-15

Revision Accepted: 2022-08-31

Crosschecked: 0000-00-00

Cited: 0

Clicked: 144

Citations:  Bibtex RefMan EndNote GB/T7714

-   Go to

Article info.
Open peer comments

Journal of Zhejiang University SCIENCE C 1998 Vol.-1 No.-1 P.

http://doi.org/10.1631/FITEE.2200099


UI layers merger: merging UI layers via visual learning and boundary prior


Author(s):  Yunnong CHEN, Yankun ZHEN, Chuning SHI, Jia-zhi Li, Liuqing CHEN, Zejian LI, Lingyun SUN, Tingting ZHOU, Yanfang CHANG

Affiliation(s):  School of Software Technology, Zhejiang University, Hangzhou 310027, China; more

Corresponding email(s):   chenlq@zju.edu.cn

Key Words:  UI to code, UI design lint, UI layers merging, Object detection


Yunnong CHEN, Yankun ZHEN, Chuning SHI, Jia-zhi Li, Liuqing CHEN,Zejian LI, Lingyun SUN, Tingting ZHOU, Yanfang CHANG. UI layers merger: merging UI layers via visual learning and boundary prior[J]. Frontiers of Information Technology & Electronic Engineering, 1998, -1(-1): .

@article{title="UI layers merger: merging UI layers via visual learning and boundary prior",
author="Yunnong CHEN, Yankun ZHEN, Chuning SHI, Jia-zhi Li, Liuqing CHEN,Zejian LI, Lingyun SUN, Tingting ZHOU, Yanfang CHANG",
journal="Frontiers of Information Technology & Electronic Engineering",
volume="-1",
number="-1",
pages="",
year="1998",
publisher="Zhejiang University Press & Springer",
doi="10.1631/FITEE.2200099"
}

%0 Journal Article
%T UI layers merger: merging UI layers via visual learning and boundary prior
%A Yunnong CHEN
%A Yankun ZHEN
%A Chuning SHI
%A Jia-zhi Li
%A Liuqing CHEN
%A Zejian LI
%A Lingyun SUN
%A Tingting ZHOU
%A Yanfang CHANG
%J Journal of Zhejiang University SCIENCE C
%V -1
%N -1
%P
%@ 2095-9184
%D 1998
%I Zhejiang University Press & Springer
%DOI 10.1631/FITEE.2200099

TY - JOUR
T1 - UI layers merger: merging UI layers via visual learning and boundary prior
A1 - Yunnong CHEN
A1 - Yankun ZHEN
A1 - Chuning SHI
A1 - Jia-zhi Li
A1 - Liuqing CHEN
A1 - Zejian LI
A1 - Lingyun SUN
A1 - Tingting ZHOU
A1 - Yanfang CHANG
J0 - Journal of Zhejiang University Science C
VL - -1
IS - -1
SP -
EP -
%@ 2095-9184
Y1 - 1998
PB - Zhejiang University Press & Springer
ER -
DOI - 10.1631/FITEE.2200099


Abstract: 
With the fast-growing graphical user interface (GUI) development workload in the Internet industry, some work attempted to generate maintainable front-end code from GUI screenshots. It can be more suitable for utilizing user interface (UI) design drafts that contain UI metadata. However, fragmented layers inevitably appear in the UI design drafts which greatly reduces the quality of the generated code. None of the existing automated GUI techniques detects and merges the fragmented layers to improve the accessibility of generated code. In this paper, we propose UI layers merger (UILM), a vision-based method that can automatically detect and merge fragmented layers into UI components. Our UILM contains the merging area detector (MAD) and a layer merging algorithm. The MAD incorporates the boundary prior knowledge to accurately detect the boundaries of UI components. Then, the layer merging algorithm can search for the associated layers within the components’ boundaries and merge them into a whole. We present a dynamic data augmentation approach to boost the performance of MAD. We also construct a large-scale UI dataset for training the MAD and testing the performance of UILM. The experiment shows that the proposed method outperforms the best baseline regarding merging area detection and achieves decent merged layers accuracy. The user study on a real application also confirms the effectiveness of our UILM.

Darkslateblue:Affiliate; Royal Blue:Author; Turquoise:Article

Open peer comments: Debate/Discuss/Question/Opinion

<1>

Please provide your name, email address and a comment





Journal of Zhejiang University-SCIENCE, 38 Zheda Road, Hangzhou 310027, China
Tel: +86-571-87952783; E-mail: cjzhang@zju.edu.cn
Copyright © 2000 - 2023 Journal of Zhejiang University-SCIENCE