<  Back to the Polytechnique Montréal portal

GUILGET : GUI layout generation with transformer

Andrey Sobolevsky, Guillaume-Alexandre Bilodeau, Jinghui Cheng and Jin L. C. Guo

Paper (2023)

Open Acess document in PolyPublie and at official publisher
Open Access to the full text of this document
Published Version
Terms of Use: Creative Commons Attribution
Download (911kB)
Show abstract
Hide abstract


Sketching out Graphical User Interface (GUI) layout is part of the pipeline of designing a GUI and a crucial task for the success of a software application. Arranging all components inside a GUI layout manually is a time-consuming task. In order to assist designers, we developed a method named GUILGET to automatically generate GUI layouts from positional constraints represented as GUI arrangement graphs (GUIAGs). The goal is to support the initial step of GUI design by producing realistic and diverse GUI layouts. The existing image layout generation techniques often cannot incorporate GUI design constraints. Thus, GUILGET needs to adapt existing techniques to generate GUI layouts that obey to constraints specific to GUI designs. GUILGET is based on transformers in order to capture the semantic in relationships between elements from GUI-AG. Moreover, the model learns constraints through the minimization of losses responsible for placing each component inside its parent layout, for not letting components overlap if they are inside the same parent, and for component alignment. Our experiments, which are conducted on the CLAY dataset, reveal that our model has the best understanding of relationships from GUI-AG and has the best performances in most of evaluation metrics. Therefore, our work contributes to improved GUI layout generation by proposing a novel method that effectively accounts for the constraints on GUI elements and paves the road for a more efficient GUI design pipeline.

Uncontrolled Keywords

Graphical User Interface; GUI arrangement graphs; deep learning; transformer; generative model; GUI layout

Department: Department of Computer Engineering and Software Engineering
PolyPublie URL: https://publications.polymtl.ca/53760/
Conference Title: 36th Canadian Conference on Artificial Intelligence (CANAI 2023)
Conference Location: Montreal, Qc, Canada
Conference Date(s): 2023-06-05 - 2023-06-09
Publisher: Canadian Artificial Intelligence Association (CAIAC)
DOI: 10.21428/594757db.08fe0a25
Official URL: https://doi.org/10.21428/594757db.08fe0a25
Date Deposited: 10 Jul 2023 16:30
Last Modified: 14 Nov 2023 01:20
Cite in APA 7: Sobolevsky, A., Bilodeau, G.-A., Cheng, J., & Guo, J. L. C. (2023, June). GUILGET : GUI layout generation with transformer [Paper]. 36th Canadian Conference on Artificial Intelligence (CANAI 2023), Montreal, Qc, Canada (12 pages). https://doi.org/10.21428/594757db.08fe0a25


Total downloads

Downloads per month in the last year

Origin of downloads


Repository Staff Only

View Item View Item