MIRA: Multiplayer Interactive World Models with Representation Autoencoders

The first multiplayer world model for highly dynamic environments, trained on 10,000 hours of Rocket League gameplay.

world models · diffusion · multi-agent

Contents

1. What is the problem?
2. Why is it interesting?
3. How is the problem solved?
4. What are the results?
5. Limitations
6. Additional notes
7. Key terms

What is the problem?

The problem is simulating a game of Rocket League with a world model. This not only includes all the dynamics within the game such as interactions between the cars and the soccer ball, gravity, robot controls, etc, but also event messages and the state of the game itself.

This is a challenging problem because such a world model does not contain a rendering engine or a physics engine, it all needs to be learnt directly from the data.

The simulation should produce four-player matches at a reasonable framerate and stay stable over long rollouts.

Why is it interesting?

• Collecting real-world data is expensive and dangerous
• Video game data is abundant
• Video games contain dynamics that somewhat mimic real world settings so learning these dynamics could be a good pretraining step for world models

How is the problem solved?

Specs

• End-to-end 20 FPS on a single NVIDIA B200 GPU
• Trained on 10K hours of bot gameplay
• 600 million-parameter video representation codec
• 5 billion-parameter world model
• World model operates at 10 Hz (in temporally downsampled latent space)

Codec

An encoder DINOv3 (frozen) is used to extract features from all player screens. These features are taken from multiple blocks in the model to capture both global and more detailed features. The features are aggregated and then projected into a latent space using a bottleneck that subsamples (by two) both in time and space. This is done to ensure the realtime contraint can be met and it is shown that it comes with a relatively low cost w.r.t to performance. This is later upsampled again during reconstruction. This codec operates on single input videos at a time (can be batched for all players of course).

The reasons this world model uses this latent space as a codec:

• pixel space is expensive and contains a lot of redundant information
• pixel space contains a lot of high frequency signals that can lead to collapse over long rollouts
• DINOv3 is used and frozen because the resulting features result are more stable in sustained roll outs

Loss function - codec

Full loss function:

$\mathcal{L}_{\text{codec}} = ||x - \hat{x}||_{1} + \lambda_{p}\text{LPIPS}(x, \hat{x}) + \lambda_{d} ||\phi(x) - \phi(\hat{x})||_2^2$

Lets go through each component.

$||x - \hat{x}||_{1}$

This component measures the difference between the raw predicted RGB values between the output and the expected output. The L1 loss $\mathcal{L}_{L_1} = \frac{1}{N} \sum{|x_i - \hat{x}_i|}$ is probably used for two reasons:

• it is less dominated by large errors, optimizing for larger errors can come at the cost of reducing overall quality
• L2 loss encourages averaging because mistakes are punished severely

However, L1 loss still measures only pixel-level accuracy. It does not necessarily capture whether the reconstruction looks perceptually similar to the original, which motivates the additional LPIPS and DINO feature losses.

$\text{LPIPS}(x, \hat{x})$

The idea behind Learned Perceptual Image Patch Similarity (LPIPS) is that humans preceive the similarity of images based on many features rather than individual pixel values. It turns out that distances between internal features correlate much more with the way humans preceived differences between images. So the way this loss works is it takes both the reconstructed image and the original images and passes both through a frozen perceptual network. Then the resulting feature maps are compared and used as a loss. The LPIPS loss looks as follows:

$t$

$||\phi(x) - \phi(\hat{x})||_2^2$

This component

World model

Loss function - world model

Training

This section will try to give a high level overview of how the whole system was trained.

Data

The data consists of 10K hours of 2v2 Rocket League matches of self playing bots. The data was recorded using a mod attached to the gameserver, this mod records the gamestate and applies bot actions. The bot used was Nexto trained in RLGym with reinfocement learning.

Procedure

The proposed solution is trained in three steps:

• First codec is learned by keeping the feature extractor frozen and learning the Space Time Vit Decoder. The loss function used is a combination of L1, LPIPS and P-Dino.
• Next the world model is pretrained on single player data
• Lastly the world model is trained on multi player data

What are the results?

Core research questions and answers

Limitations

• One style of play e.g. Nexto
• Fixed environment, no partially unobserved environments

Additional notes

Key lessons on the latent space design (taken from the paper)

• A latent is essential
• The feature extractor must be pretrained
• The bottleneck is best learned, but need not be
• Temporal downsampeling is free
• Perceptual losses are essential and sufficient
• Decoder quality saturates with scale
• Any reasonable pretrained extractor works
• Read multiple layers from the features extractor, not just the latent
• Compression can go in either the world model or the latent
• Adaptive loss balancing helps
• Upsample before decoding, not after

Analysis of data formats and implementation choices

Key terms

World models

World models predict how an environment evolves under actions. These world models are used both for control and training. For control the world model rolls out actions to predict corresponding futures, these can then be scored and actions can be taken based on those scores. For training, policies are developed that learn by interacting with the world model as though the world model is a real environment. Outputs of these models need not be images, they can be anything representing the state of the world, coordinates of object, learned embeddings, etc.

Generative world models

Generative world models predict observations directly instead of just the state of an environment. These will often take the shape of images or whatever form the agent would actually perceive. A camera flying through a static scene is an example of a generative world model. The state is typically not modeled explicitly but just learned internally by the model in a latent space.

Interactive game world models

Interactive game world models generate frames where actions directly change the contents of scenes. This makes the model causal, action-conditioned and (ideally) real-time. Simulating DOOM is an example of an interactive game world model.

Latent diffusion

Rectified flow recast

Flow matching

Diffusion forcing and teacher forcing

Few-step distillation

Streaming KV-cache

Representation auto-encoder

Evaluation metrics: ARR, gFID, gFVD, gFDD

Frechet Inception Distance

Inception Score