After beating humans in video games, AI is now taking on game design.

A team of computer programmers at IT University of Copenhagen has developed a new way to encode and generate Super Mario Bros. levels—called MarioGPT, the new approach is based on the language model GPT-2. The group outlines their work and the means by which others can use their system in a paper on the arXiv pre-print server.

Mario Brothers is a video game first introduced in 1983—it involves two Italian plumbers emerging from a sewer and attempting to rescue Princess Peach, who has been captured and held by Bowser. To rescue her, the brothers must travel (via input from the game player) across a series of obstacles made of pipes and bricks. As they travel, the terrain changes in accordance with the level they have achieved in the game. In this new effort, the team in Denmark has recreated one aspect of the game—the number of levels that can be traversed.

The researchers used Generative Pre-trained Transformer 2 (GPT-2)—an open-source language model created by a team at OpenAI, to translate user requests into graphical representations of Super Mario Brothers game levels. To do so, they created a small bit of Python code to help the language model understand what needed to be done and then trained it using samples from the original Super Mario Bros. game and one of its sequels, “Super Mario Bros.: The Lost Levels.”

The current media environment is filled with visual effects and video editing. As a result, as video-centric platforms have gained popularity, demand for more user-friendly and effective video editing tools has skyrocketed. However, because video data is temporal, editing in the format is still difficult and time-consuming. Modern machine learning models have shown considerable promise in enhancing editing, although techniques frequently compromise spatial detail and temporal consistency. The emergence of potent diffusion models trained on huge datasets recently caused a sharp increase in the quality and popularity of generative techniques for picture synthesis. Simple users may produce detailed pictures using text-conditioned models like DALL-E 2 and Stable Diffusion with only a text prompt as input. Latent diffusion models effectively synthesize pictures in a perceptually constrained environment. They research generative models suitable for interactive applications in video editing due to the development of diffusion models in picture synthesis. Current techniques either propagate adjustments using methodologies that calculate direct correspondences or, by finetuning on each unique video, re-pose existing picture models.

They try to avoid costly per-movie training and correspondence calculations for quick inference for every video. They suggest a content-aware video diffusion model with a configurable structure trained on a sizable dataset of paired text-image data and uncaptioned movies. They use monocular depth estimations to represent structure and pre-trained neural networks to anticipate embeddings to represent content. Their method gives several potent controls on the creative process. They first train their model, much like image synthesis models, so the inferred films’ content, such as their look or style, correspond to user-provided pictures or text cues (Fig. 1).

Figure 1: Video Synthesis With Guidance We introduce a method based on latent video diffusion models that synthesises videos (top and bottom) directed by text-or image-described content while preserving the original video’s structure (middle).