Realtime game rendering: restyle a game live with AI
Let your game run the rules and a realtime model draw the frames: how neural game rendering works, EdenLive versus Matrix-Game 4.0, and how to write the scene.
Realtime game rendering splits a game in two. A normal program runs the rules: input, physics, collisions, scoring. A realtime video model draws what the player sees, frame by frame, from what that program produces. The game stays a game, because its logic is deterministic code, while the look comes from a model that can make a box-and-capsule prototype read as a photoreal cavern or a painted city.
This is different from a navigable world model, where the model itself invents the world and decides what happens when you move. Here the model invents nothing about the rules. It only renders.
Two ways to do it on Windflow
| EdenLive | Matrix-Game 4.0 | |
|---|---|---|
| What it renders | The frames you send: your own game, a webcam or a clip | One of three bundled games |
| How you choose the look | A scene prompt and an optional visual anchor image | Fixed per scene; no prompt |
| Who runs the game | Your page or app | The server |
| Controls | Your game reads its own input | Held-key snapshots sent to the session |
| Output | 1280 × 704, 24 fps delivery target | 864 × 480 at 24 fps |
| Session length | Open-ended, or the length of a clip | 60 seconds |
Pick Matrix-Game 4.0 to show what neural game rendering feels like today, with no game of your own: choose castle, ancient-egypt or racing, send keys, and the server's game is drawn by a model trained for that scene. Each round is 60 seconds. See the Matrix-Game 4.0 API guide.
Pick EdenLive to render your own game. Your page draws a plain proxy picture, often called greybox footage, and sends its frames to the session; EdenLive streams back a generated picture that follows the proxy's motion. See the EdenLive API guide.
How EdenLive turns greybox into a scene
EdenLive's authors describe the proxy video as the interface between the program and the model: the program executes the rules and keeps the world state, and the model renders that proxy into detailed frames, guided by a target first frame and a text prompt. Three inputs decide the result:
- The proxy frames. They carry layout and motion. Whatever moves in them moves in the output.
- The visual anchor. The opening chunk is generated from it. Without one, the first proxy frame is the anchor and the output starts from the greybox look. EdenLive's own samples use a photoreal render of the proxy's first frame as the anchor.
- The scene prompt. It names the look, the place and the objects, and
startneeds one.
Build a proxy the model can read
- Render the proxy at the model's canvas, 1280 × 704, which is the size of EdenLive's own sample proxies.
- Keep shapes simple and readable: clear silhouettes, distinct surfaces, steady lighting.
- Leave text and HUD out of the proxy. Everything in the frame passes through the model; draw your interface on top of the output.
- Let the game own the camera. A smooth, game-like camera gives the model motion it can follow.
Write the scene prompt
EdenLive's samples pair one long static prompt for the whole scene with shorter dynamic prompts for moments within it. The static prompt follows a consistent order:
- The look: "Cinematic photorealism."
- The place, with its main materials: "A vast volcanic cavern with a churning molten lava lake, dark basalt formations and jagged stalactites."
- The atmosphere: "Heat haze and glowing embers fill the air."
- The viewpoint and what the player holds: "Seen first-person, a heavy industrial cryo-blaster with frosted metal plating."
- The key effects: "Projectiles flash-freeze the magma into slabs of hissing ice."
- The lighting: "Fiery magma light balanced against cool cyan glow."
Describe the scene the proxy actually shows, so the prompt and the frames point the same way.
On Windflow, a change of prompt, anchor or seed during a run takes effect from the next chunk boundary, starting fresh from the current source. Use prompt changes for deliberate beats, such as entering a new area or a change of weather, rather than for every action. Actions the proxy already shows, like firing or jumping, need no prompt at all.
Tune it for your game
Before you set gameplay timing, measure input-to-display latency on your own setup with this latency method. Exploration, puzzles, showcases and interactive scenes are a natural fit for realtime rendering.
FAQ
Do I need a game engine to use EdenLive?
You need a source of frames. A simple WebGL or canvas scene that draws greybox shapes is enough. A webcam or an uploaded clip also works as the source.
Can I use my own game with Matrix-Game 4.0?
No. Each Matrix-Game 4.0 scene is a model trained for its bundled game. To render your own game, use EdenLive.
Does the model change the game's rules?
No. In realtime game rendering, the game program owns the rules and state. The model only draws the frames, so collisions, scoring and physics behave exactly as your code says.