Generating Text Conditioned 3D Human Motion

Computer Science and Engineering

A novel approach to generating 3D human motions from textual descriptions using a two-stage framework that incorporates a temporal variational autoencoder (VAE). The first stage, text-to-length sampling, predicts the length of the motion sequence by approximating a probability distribution conditioned on the input text. This step enables the generation of variable-length motions that align naturally with diverse textual inputs. In the second stage, text-to-motion generation, the model synthesizes human motions consistent with the sampled lengths and input descriptions. At its core, the approach utilizes a temporal VAE, featuring a triplet structure of prior, posterior, and generator networks to learn the mapping between textual semantics and motion dynamics. To further enhance motion fidelity, a motion snippet code representation is proposed as an internal format, encapsulating localized temporal semantics to ensure smooth and realistic motions faithful to the input text. The framework’s flexibility accommodates a range of textual complexities, from simple commands to detailed narratives, ensuring both diversity and naturalness in the output. This approach is evaluated on a new large-scale dataset, HumanML3D, which includes 14,616 motion clips and 44,970 text descriptions, as well as on the KIT Motion-Language dataset. Quantitative metrics and user studies demonstrate significant advancements over baseline methods in terms of motion quality, diversity, and alignment with textual input. By addressing the challenges of variable sequence lengths, semantic diversity, and textual complexity, this two-stage VAE-based framework sets a new benchmark for text-driven 3D motion generation, with applications spanning animation, gaming, and human-computer interaction.

This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.