Abstract: In graphic design, layout is a result of the interaction between the design elements in the foreground and background images. However, prevalent research focuses on enhancing the quality of layout generation algorithms, overlooking the interaction and controllability that are essential for designers when applying these methods in real-world situations. This paper proposes a user-centered layout design system, Iris, which provides designers with an interactive environment to expedite the workflow, and this environment encompasses the features of user-constraint specification, layout generation, custom editing, and final rendering. To satisfy the multiple constraints specified by designers, we introduce a novel generation model, multi-constraint LayoutVQ-VAE, for advancing layout generation under intra- and inter-domain constraints. Qualitative and quantitative experiments on our proposed model indicate that it outperforms or is comparable to prevalent state-of-the-art models in multiple aspects. User studies on Iris further demonstrate that the system significantly enhances design efficiency while achieving human-like layout designs.

Iris：一个满足多条件约束的图形布局生成系统

[1]Arroyo DM, Postels J, Tombari F, 2021. Variational Transformer networks for layout generation. Proc IEEE/CVF Conf on Computer Vision and Pattern Recognition, p.13637-13647.

[2]Ba JL, Kiros JR, Hinton GE, 2016. Layer normalization. https://arxiv.org/abs/1607.06450

[3]Bangor A, Kortum P, Miller J, 2009. Determining what individual SUS scores mean: adding an adjective rating scale. J Usabil Stud, 4(3):114-123.

[4]Cao YN, Ma Y, Zhou M, et al., 2022. Geometry aligned variational Transformer for image-conditioned layout generation. Proc 30^th ACM Int Conf on Multimedia, p.1561-1571.

[5]Dayama NR, Todi K, Saarelainen T, et al., 2020. GRIDS: interactive layout design with integer programming. Proc CHI Conf on Human Factors in Computing Systems, p.1-13.

[6]Deka B, Huang ZF, Franzen C, et al., 2017. Rico: a mobile App dataset for building data-driven design applications. Proc 30^th Annual ACM Symp on User Interface Software and Technology, p.845-854.

[7]Devlin J, Chang MW, Lee K, et al., 2019. BERT: pre-training of deep bidirectional Transformers for language understanding. Proc Conf of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, p.4171-4186.

[8]Dosovitskiy A, Beyer L, Kolesnikov A, et al., 2021. An image is worth 16×16 words: Transformers for image recognition at scale. Proc 9^th Int Conf on Learning Representations.

[9]Guo SN, Jin ZC, Sun FL, et al., 2021. Vinci: an intelligent graphic design system for generating advertising posters. Proc CHI Conf on Human Factors in Computing Systems, Article 577.

[10]Gupta K, Lazarow J, Achille A, et al., 2021. LayoutTransformer: layout generation and completion with self-attention. Proc IEEE/CVF Int Conf on Computer Vision, p.984-994.

[11]Hart SG, Staveland LE, 1988. Development of NASA-TLX (task load index): results of empirical and theoretical research. Adv Psychol, 52:139-183.

[12]He KM, Zhang XY, Ren SQ, et al., 2016. Deep residual learning for image recognition. Proc IEEE Conf on Computer Vision and Pattern Recognition, p.770-778.

[13]Heusel M, Ramsauer H, Unterthiner T, et al., 2017. GANs trained by a two time-scale update rule converge to a local Nash equilibrium. Proc 30^th Int Conf on Neural Information Processing Systems, p.6626-6637.

[14]Hsu H, He XT, Peng YX, et al., 2023. PosterLayout: a new benchmark and approach for content-aware visual-textual presentation layout. Proc IEEE/CVF Conf on Computer Vision and Pattern Recognition, p.6018-6026.

[15]Hui MD, Zhang ZZ, Zhang XY, et al., 2023. Unifying layout generation with a decoupled diffusion model. Proc IEEE/CVF Conf on Computer Vision and Pattern Recognition, p.1942-1951.

[16]Inoue N, Kikuchi K, Simo-Serra E, et al., 2023. LayoutDM: discrete diffusion model for controllable layout generation. Proc IEEE/CVF Conf on Computer Vision and Pattern Recognition, p.10167-10176.

[17]Jacobs C, Li W, Schrier E, et al., 2003. Adaptive grid-based document layout. ACM Trans Graph, 22(3):838-847.

[18]Jiang ZY, Sun SZ, Zhu JH, et al., 2022. Coarse-to-fine generative modeling for graphic layouts. Proc 36^th AAAI Conf on Artificial Intelligence, p.1096-1103.

[19]Jiang ZY, Guo JQ, Sun SZ, et al., 2023. LayoutFormer++: conditional graphic layout generation via constraint serialization and decoding space restriction. Proc IEEE/CVF Conf on Computer Vision and Pattern Recognition, p.18403-18412.

[20]Jing QZ, Zhou TT, Tsang Y, et al., 2023. Layout generation for various scenarios in mobile shopping applications. Proc CHI Conf on Human Factors in Computing Systems, Article 130.

[21]Kaiser L, Bengio S, Roy A, et al., 2018. Fast decoding in sequence models using discrete latent variables. Proc 35^th Int Conf on Machine Learning, p.2395-2404.

[22]Kikuchi K, Simo-Serra E, Otani M, et al., 2021. Constrained graphic layout generation via latent optimization. Proc 29^th ACM Int Conf on Multimedia, p.88-96.

[23]Kong X, Jiang L, Chang HW, et al., 2022. BLT: bidirectional layout transformer for controllable layout generation. Proc 17^th European Conf on Computer Vision, p.474-490.

[24]Li JN, Yang JM, Hertzmann A, et al., 2019. LayoutGAN: generating graphic layouts with wireframe discriminators. Proc 7^th Int Conf on Learning Representations.

[25]Li JN, Yang JM, Zhang JM, et al., 2021. Attribute-conditioned layout GAN for automatic graphic design. IEEE Trans Vis Comput Graph, 27(10):4039-4048.

[26]Lin TY, Dollár P, Girshick R, et al., 2017. Feature pyramid networks for object detection. Proc IEEE Conf on Computer Vision and Pattern Recognition, p.2117-2125.

[27]O’Donovan P, Agarwala A, Hertzmann A, 2014. Learning layouts for single-page graphic designs. IEEE Trans Vis Comput Graph, 20(8):1200-1213.

[28]Paszke A, Gross S, Massa F, et al., 2019. PyTorch: an imperative style, high-performance deep learning library. Proc 32^nd Int Conf on Neural Information Processing Systems, p.8024-8035.

[29]Schrier E, Dontcheva M, Jacobs C, et al., 2008. Adaptive layout for dynamically aggregated documents. Proc 13^th Int Conf on Intelligent User Interfaces, p.99-108.

[30]van den Oord A, Vinyals O, Kavukcuoglu K, 2017. Neural discrete representation learning. Proc 30^th Int Conf on Neural Information Processing Systems, p.6306-6315.

[31]Vaswani A, Shazeer N, Parmar N, et al., 2017. Attention is all you need. Proc 30^th Int Conf on Neural Information Processing Systems, p.5998-6008.

[32]Xu CC, Zhou M, Ge TZ, et al., 2023. Unsupervised domain adaption with pixel-level discriminator for image-aware layout generation. Proc IEEE/CVF Conf on Computer Vision and Pattern Recognition, p.10114-10123.

[33]You WT, Jiang H, Yang ZY, et al., 2020. Automatic synthesis of advertising images according to a specified style. Front Inform Technol Electron Eng, 21(10):1455-1466.

[34]Zheng XR, Qiao XT, Cao Y, et al., 2019. Content-aware generative modeling of graphic design layouts. ACM Trans Graph, 38(4):133.

[35]Zhong X, Tang JB, Yepes AJ, 2019. PubLayNet: largest dataset ever for document layout analysis. Proc Int Conf on Document Analysis and Recognition, p.1015-1022.

[36]Zhou M, Xu CC, Ma Y, et al., 2022. Composition-aware graphic layout GAN for visual-textual presentation designs. Proc 31^st Int Joint Conf on Artificial Intelligence, p.4995-5001.