CLC number: TP39
On-line Access: 2023-03-25
Received: 2022-03-15
Revision Accepted: 2023-03-25
Crosschecked: 2022-08-31
Cited: 0
Clicked: 1163
Yunnong CHEN, Yankun ZHEN, Chuning SHI, Jiazhi LI, Liuqing CHEN, Zejian LI, Lingyun SUN, Tingting ZHOU, Yanfang CHANG. UI layers merger: merging UI layers via visual learning and boundary prior[J]. Frontiers of Information Technology & Electronic Engineering,in press.https://doi.org/10.1631/FITEE.2200099 @article{title="UI layers merger: merging UI layers via visual learning and boundary prior", %0 Journal Article TY - JOUR
UI图层合并器:基于计算机视觉与边界先验的UI图层合并方法1浙江大学软件学院,中国杭州市,310027 2浙江大学计算机科学与技术学院,中国杭州市,310027 3阿里巴巴-浙江大学前沿技术联合研究中心,中国杭州市,310027 4阿里巴巴集团,中国杭州市,311121 5浙江-新加坡人工智能与创新设计联合实验室,中国杭州市,310027 摘要:随着互联网行业图形用户界面(GUI)开发工作量的快速增长,国内外学者试图从GUI图片生成可维护的前端代码。该任务更适合从包含有用户界面元数据的UI设计稿来实现代码生成。然而,碎片图层不可避免地会出现在UI设计稿中,这大大降低了生成代码的质量。目前尚未有自动化GUI技术来检测并合并碎片图层以提高生成代码的可维护性。本文提出UI图层合并器(UILM),一种基于计算机视觉的方法,可以自动检测并合并碎片图层为完整的UI组件。本文提出的UILM包含合并区域检测器和碎片图层合并算法。合并区域检测器结合边界先验知识来准确地检测出UI组件的边界。基于此,碎片图层合并算法可以搜索出边界内的相关碎片层并将其合并为整体。此外,我们提出一种动态数据增强算法来提高合并区域检测器的性能。我们还构建了用于训练合并区域检测器和测试UILM的大规模UI数据集。实验结果表明,本文提出的方法在合并区域检测方面优于最佳基线,并达到了有效的图层合并准确率。最后,对真实应用程序的用户研究也证实了本文方法的有效性。 关键词组: Darkslateblue:Affiliate; Royal Blue:Author; Turquoise:Article
Reference[1]Așıroǧlu B, Mete BR, Yıldız E, et al., 2019. Automatic HTML code generation from mock-up images using machine learning techniques. Scientific Meeting on Electrical-Electronics & Biomedical Engineering and Computer Science, p.1-4. [2]Behrang F, Reiss SP, Orso A, 2018. GUIFetch: supporting app design and development through GUI search. Proc 5th Int Conf on Mobile Software Engineering and Systems, p.236-246. [3]Beltramelli T, 2018. pix2code: generating code from a graphical user interface screenshot. ACM SIGCHI Symp on Engineering Interactive Computing Systems, Article 3. [4]Bunian S, Li K, Jemmali C, et al., 2021. VINS: visual search for mobile user interface design. CHI Conf on Human Factors in Computing Systems, Article 423. [5]Cai ZW, Vasconcelos N, 2018. Cascade R-CNN: delving into high quality object detection. IEEE/CVF Conf on Computer Vision and Pattern Recognition, p.6154-6162. [6]Chen CY, Su T, Meng GZ, et al., 2018. From UI design image to GUI skeleton: a neural machine translator to bootstrap mobile GUI implementation. Proc 40th Int Conf on Software Engineering, p.665-676. [7]Chen CY, Feng SD, Xing ZC, et al., 2019. Gallery D.C.: design search and knowledge discovery through auto-created GUI component gallery. Proc ACM on Human-Computer Interaction, Article 180. [8]Chen JS, Xie ML, Xing ZC, et al., 2020. Object detection for graphical user interface: old fashioned or deep learning or a combination? Proc 28th ACM Joint Meeting on European Software Engineering Conf and Symp on the Foundations of Software Engineering, p.1202-1214. [9]Chen K, Wang JQ, Pang JM, et al., 2019. MMDetection: open MMLab detection toolbox and benchmark. https://arxiv.org/abs/1906.07155v1 [10]Chen S, Fan LL, Su T, et al., 2019. Automated cross-platform GUI code generation for mobile apps. Proc IEEE 1st Int Workshop on Artificial Intelligence for Mobile, p.13-16. [11]Deka B, Huang ZF, Franzen C, et al., 2017. Rico: a mobile app dataset for building data-driven design applications. Proc 30th Annual ACM Symp on User Interface Software and Technology, p.845-854. [12]Fay MP, Proschan MA, 2010. Wilcoxon-Mann-Whitney or t-test? On assumptions for hypothesis tests and multiple interpretations of decision rules. Stat Surv, 4:1-39. [13]Feng SD, Ma SY, Yu JZ, et al., 2021. Auto-icon: an automated code generation tool for icon designs assisting in UI development. Proc 26th Int Conf on Intelligent User Interfaces, p.59-69. [14]Ge XF, 2019. Android GUI search using hand-drawn sketches. Proc IEEE/ACM 41st Int Conf on Software Engineering: Companion Proc, p.141-143. [15]Halbe A, Joshi AR, 2015. A novel approach to HTML page creation using neural network. Proc Comput Sci, 45:197-204. [16]He KM, Zhang XY, Ren SQ, et al., 2016. Deep residual learning for image recognition. IEEE Conf on Computer Vision and Pattern Recognition, p.770-778. [17]Jain V, Agrawal P, Banga S, et al., 2019. Sketch2Code: transformation of sketches to UI in real-time using deep neural network. https://arxiv.org/abs/1910.08930 [18]Li G, Baechler G, Tragut M, et al., 2022. Learning to denoise raw mobile UI layouts for improving datasets at scale. CHI Conf on Human Factors in Computing Systems, Article 67. [19]Lin TY, Maire M, Belongie S, et al., 2014. Microsoft COCO: common objects in context. Proc 13th European Conf on Computer Vision, p.740-755. [20]Lin TY, Dollár P, Girshick R, et al., 2017. Feature pyramid networks for object detection. IEEE Conf on Computer Vision and Pattern Recognition, p.936-944. [21]Liu Z, Chen CY, Wang JJ, et al., 2020. Owl eyes: spotting UI display issues via visual understanding. Proc 35th IEEE/ACM Int Conf on Automated Software Engineering, p.398-409. [22]Liu Z, Chen CY, Wang JJ, et al., 2023. Nighthawk: fully automated localizing UI display issues via visual understanding. IEEE Trans Soft Eng, 49(1):403-418. [23]Moran K, Bernal-Cárdenas C, Curcio M, et al., 2020. Machine learning-based prototyping of graphical user interfaces for mobile apps. IEEE Trans Soft Eng, 46(2):196-221. [24]Nguyen TA, Csallner C, 2015. Reverse engineering mobile application user interfaces with REMAUI. Proc 30th IEEE/ACM Int Conf on Automated Software Engineering, p.248-259. [25]Ren SQ, He KM, Girshick RB, et al., 2017. Faster R-CNN: towards real-time object detection with region proposal networks. IEEE Trans Patt Anal Mach Intell, 39(6):1137-1149. [26]Suleri S, Pandian VPS, Shishkovets S, et al., 2019. Eve: a sketch-based software prototyping workbench. CHI Conf on Human Factors in Computing Systems, Article LBW1410. [27]Vu T, Jang H, Pham TX, et al., 2019. Cascade RPN: delving into high-quality region proposal network with adaptive convolution. Proc 33rd Conf on Neural Information Processing Systems, p.1430-1440. [28]White TD, Fraser G, Brown GJ, 2019. Improving random GUI testing with image-based widget detection. Proc 28th ACM SIGSOFT Int Symp on Software Testing and Analysis, p.307-317. [29]Xu N, Price B, Cohen S, et al., 2017. Deep image matting. IEEE Conf on Computer Vision and Pattern Recognition, p.311-320. [30]Zhang XY, de Greef L, Swearngin A, et al., 2021. Screen recognition: creating accessibility metadata for mobile applications from pixels. CHI Conf on Human Factors in Computing Systems, Article 275. [31]Zhao TM, Chen CY, Liu YM, et al., 2021. GUIGAN: learning to generate GUI designs using generative adversarial networks. Proc 43rd IEEE/ACM Int Conf on Software Engineering, p.748-760. Journal of Zhejiang University-SCIENCE, 38 Zheda Road, Hangzhou
310027, China
Tel: +86-571-87952783; E-mail: cjzhang@zju.edu.cn Copyright © 2000 - 2024 Journal of Zhejiang University-SCIENCE |
Open peer comments: Debate/Discuss/Question/Opinion
<1>