BRIDGING LANGUAGES THROUGH IMAGES

COMPUTER ENGINEERING

Research on text-to-image generation (TTI) still predominantly focuses on the English language due to the lack of annotated image caption data in other languages; in the long run,this might widen inequitable access to TTI technology. In this work, we thus investigate multilingual TTI (termed mTTI) and the potential of neural machine translation (NMT) to bootstrap mTTI systems.We provide two key contributions. 1)Relying on a multilingual multi-modal encoder, we provide a systematic empirical study of standard methods used in cross-lingual NLP when applied to mTTI: TRANSLATE TRAIN, TRANSLATE TEST, and ZERO-SHOT TRANSFER. 2)We propose Ensemble Adapter (ENSAD), a novel parameter-efficient approach that learns to weigh and consolidate the multilingual text knowledge within the mTTI framework, mitigating the language gap and thus improving mTTI performance. Development of transformer-based text-to-image models is impeded by its slow generation and complexity, for high-resolution images In this work, we put forward a solution based on hierarchical transformers and local parallel autoregressive generation. We pretrain a-parameter transformer with a simple and flexible self-supervised task, a cross- model general language model (CogLM), and fine tune it for fast super-resolution. The new text-to-image system, It,shows competitive generation performance to the concurrent state- of-the-art DALLE-2 and naturally supports interactive text-guided editing on images.

This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.