How to use the DreamX World API

The DreamX World API on Windflow runs DreamX-World-5B, AMAP's controllable world model, as a realtime session. A seed image and a text prompt open a world; movement and look controls steer the camera through it, and the result streams back as a portrait 704 × 1280 rollout. Controls are sampled into the model's next action sequence rather than applied frame by frame, so the world responds in steps.

Start from something runnable

You need an approved Windflow account and an API key that begins with wf_live_ (request access). Requests go to https://api.windflow.online.

1. Check the key (server). The response lists this model's commands, messages and limitations.

curl https://api.windflow.online/v1/models/dreamx-world-5b \
  -H "Authorization: Bearer $WINDFLOW_API_KEY"

2. Mint a browser token (server). The account key never reaches the browser.

// For example app/api/windflow-token/route.ts. Authenticate your own user first.
export async function POST() {
  const res = await fetch("https://api.windflow.online/v1/tokens", {
    method: "POST",
    headers: {
      Authorization: `Bearer ${process.env.WINDFLOW_API_KEY}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({ model: "dreamx-world-5b", max_sessions: 1 }),
  });
  return Response.json(await res.json(), { status: res.status });
}

The token is bound to this model (another model gets 403 model_mismatch) and can create one session. Read expires_at from the response instead of assuming a lifetime.

3. Open the session and draw frames (browser). Save this as windflow-session.js. It is the same file the Helios guide walks through line by line.

// windflow-session.js: plain fetch and WebSocket, no dependencies
const API = "https://api.windflow.online";

export async function call(path, jwt, body) {
  const res = await fetch(API + path, {
    method: body ? "POST" : "GET",
    headers: { Authorization: `Bearer ${jwt}`, "Content-Type": "application/json" },
    body: body && JSON.stringify(body),
  });
  const data = await res.json();
  if (!res.ok) throw Object.assign(new Error(data.error.message), data.error, { status: res.status });
  return data;
}

export function openSession(model, canvas, onMessage = () => {}) {
  const ctx = canvas.getContext("2d");
  let session, ws, closed = false, released = false;
  let painting = Promise.resolve();

  const paint = async (bytes) => {
    const frame = await createImageBitmap(new Blob([bytes], { type: "image/jpeg" }));
    if (canvas.width !== frame.width) Object.assign(canvas, { width: frame.width, height: frame.height });
    ctx.drawImage(frame, 0, 0);
    frame.close();
  };

  const release = () => {
    closed = true;
    ws?.close();
    if (!session || released) return;
    released = true;
    fetch(`${API}/v1/sessions/${session.session_id}`, {
      method: "POST",
      keepalive: true, // still sent while the page unloads
      headers: { Authorization: `Bearer ${session.jwt}`, "Content-Type": "application/json" },
      body: JSON.stringify({ action: "detach", connection_id: session.connection_id, release: true }),
    });
  };

  const ready = (async () => {
    const { jwt } = await (await fetch("/api/windflow-token", { method: "POST" })).json();
    session = await call("/v1/sessions", jwt, { model, transport: "ws-h264" });
    while (session.status === "queued" && !closed) {
      await new Promise((resolve) => setTimeout(resolve, 2000));
      const poll = await call(`/v1/sessions/${session.session_id}`, session.jwt);
      onMessage({ type: "queued", position: poll.queue_position });
      if (poll.status !== "queued") {
        session = await call("/v1/sessions", session.jwt, { model, session_id: session.session_id, transport: "ws-h264" });
      }
    }
    const offer = session.transport.offers.find((o) => o.kind === "ws-h264");
    if (closed || !offer) throw new Error(closed ? "released" : `no stream offer (status ${session.status})`);
    const url = new URL(offer.relay_path, API);
    url.protocol = "wss:";
    ws = new WebSocket(url);
    ws.binaryType = "arraybuffer";
    return new Promise((resolve, reject) => {
      ws.onopen = () => {
        ws.send(JSON.stringify({ type: "authenticate", token: offer.launch_token }));
        ws.send(JSON.stringify({ type: "client_hello", codecs: ["jpeg"] }));
      };
      ws.onmessage = ({ data }) => {
        if (typeof data !== "string") {
          painting = painting.then(() => paint(data)).catch(console.warn);
          return;
        }
        const msg = JSON.parse(data);
        if (msg.type === "worker_ready") resolve(msg);
        if (msg.type === "failed") reject(new Error(msg.message ?? msg.error));
        onMessage(msg);
      };
      ws.onclose = (event) => {
        onMessage({ type: "socket_closed", code: event.code });
        reject(new Error(`socket closed (${event.code})`));
      };
    });
  })();
  ready.catch(release); // a failed start must not leave a session running

  return {
    ready,
    release,
    get session() { return session; },
    get socket() { return ws; },
    command: (type, fields = {}) =>
      call(`/v1/sessions/${session.session_id}/commands`, session.jwt, { command: { type, ...fields } }),
    upload: async (file) => {
      const intent = await call("/v1/uploads", session.jwt, {
        filename: file.name, mime: file.type, size: file.size, session_id: session.session_id,
      });
      const put = await fetch(intent.upload_url, { method: "PUT", headers: intent.headers, body: file });
      if (!put.ok) throw new Error(`upload failed (${put.status})`);
      return call(`/v1/uploads/${intent.file_id}/complete`, session.jwt, {});
    },
  };
}

queued and socket_closed are messages this file generates; every other message comes from the session.

4. Stage the world and start it. DreamX needs both a prompt and a seed image before start. Stage them in one set_conditioning call:

import { openSession } from "./windflow-session.js";

const world = openSession("dreamx-world-5b", document.querySelector("canvas"), (msg) => console.log(msg.type));
await world.ready; // worker_ready: commands are accepted from here
const seed = await world.upload(document.querySelector("input[type=file]").files[0]);
await world.command("set_conditioning", {
  prompt: "A stone canal town at dusk, wet cobblestones reflecting lantern light, narrow alleys leading toward a bridge",
  image: { id: seed.id },
});
await world.command("start");

5. Send held keys as one snapshot. WASD moves and the arrow keys look around. Send the complete held set as a set_input_state snapshot with a rising input_seq, on the open socket:

const KEYS = {
  KeyW: "forward", KeyA: "left", KeyS: "back", KeyD: "right",
  ArrowUp: "look_up", ArrowLeft: "look_left", ArrowDown: "look_down", ArrowRight: "look_right",
};
const held = new Set();
let inputSeq = 0;

function sendKeys() {
  world.socket.send(JSON.stringify({
    type: "set_input_state", input_seq: ++inputSeq, pressed_keys: [...held], client_ts: Date.now(),
  }));
}
addEventListener("keydown", (event) => {
  const key = KEYS[event.code];
  if (!key || held.has(key) || event.target.closest?.("input, textarea")) return;
  event.preventDefault(); // keep the arrow keys from scrolling the page
  held.add(key);
  sendKeys();
});
addEventListener("keyup", (event) => {
  if (held.delete(KEYS[event.code])) sendKeys();
});
addEventListener("blur", () => {
  held.clear();
  sendKeys();
});

Wait until the session is ready

Capacity is brought up on demand, so POST /v1/sessions can answer status: "queued" with a queue_position and no transport offers. Keep polling GET /v1/sessions/{id}: a queue ticket nobody polls expires with error: "scheduler_queue_expired". When the status leaves queued, POST /v1/sessions with the same session_id returns the WebSocket offer without creating a second session.

On the socket, authenticate with the offer's launch_token must be the first message; client_hello with codecs: ["jpeg"] asks for one JPEG per binary message. Commands are accepted after worker_ready. Before it they fail with 409 queued or 409 no_active_session.

How controls reach the world

DreamX does not read a controller every frame. Your held keys are sampled into the next action sequence, the model generates that stretch of the world, and the frames stream back. Three things follow:

• Movement is persistent state. It keeps going through later sequences until you release it. That is why the example sends an empty snapshot on keyup and on window blur.
• Steady intent works best. One movement plus at most one look direction per sequence reads clearly. Steady keys give the clearest motion.
• Acknowledge at once, expect the picture later. input_accepted arrives for every snapshot; show the press in your interface immediately, and let the world catch up at the next sequence. A snapshot whose input_seq is not ahead of the last applied one comes back with stale: true and is not applied.

set_movement, set_look_horizontal and set_look_vertical are single-axis alternatives to the snapshot. set_prompt updates the world prompt during the run, and pause, resume, reset and set_seed control the run.

The production runtime streams portrait 704 × 1280, and the resolution is negotiated per session. Read video_config rather than hard-coding the canvas size, and size the canvas with CSS. Compose seed images for a tall frame: a path that recedes up the picture navigates better than a wide vista that gets cropped. The world model seed image and prompt guide covers seed choice in detail.

Track arrived is not first frame visible

Enable the keys at the first painted frame, not at worker_ready. The session's transport.fallback_policy.first_frame_timeout_ms (10 seconds by default) is how long the platform's own client waits for media after generation_started.

Release the session on unmount

Call world.release() when the component unmounts and on pagehide. It sends { action: "detach", connection_id, release: true } with keepalive, which ends the session at the worker and hands the capacity back; the response reports session_released: true. A world nobody is steering is still a ready session, and paused time is billed, so release is what stops the meter. If a tab dies without releasing, the worker keeps the session for a reconnect grace (30 seconds by default). For drops, see recovering a dropped stream.

Troubleshooting by the stage where startup stopped

Rejected commands return 422 with a command_… code over HTTP and a command_error message on the socket. conditions_ready reports has_prompt and has_image before start.

FAQ

Is DreamX first-person or third-person?

The controls move and turn the camera through the world, as in a first-person view. For a world where the keys drive an on-screen character, see ABot-World.

Can I use WebRTC with DreamX?

Yes. DreamX is offered over both ws-h264 and livekit. Request transport: "livekit" and join the room with the offer's url and participant_token; commands go to POST /v1/sessions/{id}/commands on both transports.

Why is the stream portrait?

The production runtime renders 704 × 1280. Resolution is negotiated per session, so read it from video_config instead of assuming it.