How to use the ABot World API

The ABot World API on Windflow runs ABot-World-0-5B as a realtime session: a seed image and a text prompt start a third-person world, and the model's eight keys drive the on-screen character and the camera. ABot-World streams over the WebSocket transport only, so the code on this page is the whole media path: a server token, a session, one socket for frames and key state, and a release when the player leaves.

Start from something runnable

You need an approved Windflow account and an API key that begins with wf_live_ (request access). Requests go to https://api.windflow.online.

1. Check the key (server). The response lists this model's commands, messages and limitations.

curl https://api.windflow.online/v1/models/abot-world-0-5b \
  -H "Authorization: Bearer $WINDFLOW_API_KEY"

2. Mint a browser token (server). The account key never reaches the browser.

// For example app/api/windflow-token/route.ts. Authenticate your own user first.
export async function POST() {
  const res = await fetch("https://api.windflow.online/v1/tokens", {
    method: "POST",
    headers: {
      Authorization: `Bearer ${process.env.WINDFLOW_API_KEY}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({ model: "abot-world-0-5b", max_sessions: 1 }),
  });
  return Response.json(await res.json(), { status: res.status });
}

The token is bound to this model (another model gets 403 model_mismatch) and can create one session. Read expires_at from the response instead of assuming a lifetime.

3. Open the session and draw frames (browser). Save this as windflow-session.js; the Helios, JoyAI-Video-Edit and LingBot guides use the same file.

// windflow-session.js: plain fetch and WebSocket, no dependencies
const API = "https://api.windflow.online";

export async function call(path, jwt, body) {
  const res = await fetch(API + path, {
    method: body ? "POST" : "GET",
    headers: { Authorization: `Bearer ${jwt}`, "Content-Type": "application/json" },
    body: body && JSON.stringify(body),
  });
  const data = await res.json();
  if (!res.ok) throw Object.assign(new Error(data.error.message), data.error, { status: res.status });
  return data;
}

export function openSession(model, canvas, onMessage = () => {}) {
  const ctx = canvas.getContext("2d");
  let session, ws, closed = false, released = false;
  let painting = Promise.resolve();

  const paint = async (bytes) => {
    const frame = await createImageBitmap(new Blob([bytes], { type: "image/jpeg" }));
    if (canvas.width !== frame.width) Object.assign(canvas, { width: frame.width, height: frame.height });
    ctx.drawImage(frame, 0, 0);
    frame.close();
  };

  const release = () => {
    closed = true;
    ws?.close();
    if (!session || released) return;
    released = true;
    fetch(`${API}/v1/sessions/${session.session_id}`, {
      method: "POST",
      keepalive: true, // still sent while the page unloads
      headers: { Authorization: `Bearer ${session.jwt}`, "Content-Type": "application/json" },
      body: JSON.stringify({ action: "detach", connection_id: session.connection_id, release: true }),
    });
  };

  const ready = (async () => {
    const { jwt } = await (await fetch("/api/windflow-token", { method: "POST" })).json();
    session = await call("/v1/sessions", jwt, { model, transport: "ws-h264" });
    while (session.status === "queued" && !closed) {
      await new Promise((resolve) => setTimeout(resolve, 2000));
      const poll = await call(`/v1/sessions/${session.session_id}`, session.jwt);
      onMessage({ type: "queued", position: poll.queue_position });
      if (poll.status !== "queued") {
        session = await call("/v1/sessions", session.jwt, { model, session_id: session.session_id, transport: "ws-h264" });
      }
    }
    const offer = session.transport.offers.find((o) => o.kind === "ws-h264");
    if (closed || !offer) throw new Error(closed ? "released" : `no stream offer (status ${session.status})`);
    const url = new URL(offer.relay_path, API);
    url.protocol = "wss:";
    ws = new WebSocket(url);
    ws.binaryType = "arraybuffer";
    return new Promise((resolve, reject) => {
      ws.onopen = () => {
        ws.send(JSON.stringify({ type: "authenticate", token: offer.launch_token }));
        ws.send(JSON.stringify({ type: "client_hello", codecs: ["jpeg"] }));
      };
      ws.onmessage = ({ data }) => {
        if (typeof data !== "string") {
          painting = painting.then(() => paint(data)).catch(console.warn);
          return;
        }
        const msg = JSON.parse(data);
        if (msg.type === "worker_ready") resolve(msg);
        if (msg.type === "failed") reject(new Error(msg.message ?? msg.error));
        onMessage(msg);
      };
      ws.onclose = (event) => {
        onMessage({ type: "socket_closed", code: event.code });
        reject(new Error(`socket closed (${event.code})`));
      };
    });
  })();
  ready.catch(release); // a failed start must not leave a session running

  return {
    ready,
    release,
    get session() { return session; },
    get socket() { return ws; },
    command: (type, fields = {}) =>
      call(`/v1/sessions/${session.session_id}/commands`, session.jwt, { command: { type, ...fields } }),
    upload: async (file) => {
      const intent = await call("/v1/uploads", session.jwt, {
        filename: file.name, mime: file.type, size: file.size, session_id: session.session_id,
      });
      const put = await fetch(intent.upload_url, { method: "PUT", headers: intent.headers, body: file });
      if (!put.ok) throw new Error(`upload failed (${put.status})`);
      return call(`/v1/uploads/${intent.file_id}/complete`, session.jwt, {});
    },
  };
}

queued and socket_closed are messages this file generates; every other message comes from the session.

4. Stage the world and start it. ABot-World needs both a prompt and a seed image before start; stage them in one set_conditioning call:

import { openSession } from "./windflow-session.js";

const world = openSession("abot-world-0-5b", document.querySelector("canvas"), (msg) => console.log(msg.type));
await world.ready; // worker_ready: commands are accepted from here
const seed = await world.upload(document.querySelector("input[type=file]").files[0]);
await world.command("set_conditioning", { prompt: "A courier walks through a rainy neon market", image: { id: seed.id } });
await world.command("start");

5. Map the eight keys. WASD moves the character and IJKL turns the camera. Send the complete held set as a set_input_state snapshot with a rising input_seq, on the open socket:

const KEYS = {
  KeyW: "forward", KeyA: "left", KeyS: "back", KeyD: "right",
  KeyI: "look_up", KeyJ: "look_left", KeyK: "look_down", KeyL: "look_right",
};
const held = new Set();
let inputSeq = 0;

function sendKeys() {
  world.socket.send(JSON.stringify({
    type: "set_input_state", input_seq: ++inputSeq, pressed_keys: [...held], client_ts: Date.now(),
  }));
}
addEventListener("keydown", (event) => {
  const key = KEYS[event.code];
  if (!key || held.has(key) || event.target.closest?.("input, textarea")) return;
  held.add(key);
  sendKeys();
});
addEventListener("keyup", (event) => {
  if (held.delete(KEYS[event.code])) sendKeys();
});
addEventListener("blur", () => {
  held.clear();
  sendKeys();
});

Wait until the session is ready

Capacity is brought up on demand, so POST /v1/sessions can answer status: "queued" with a queue_position and no transport offers. Keep polling GET /v1/sessions/{id}: a queue ticket nobody polls expires with error: "scheduler_queue_expired". When the status leaves queued, POST /v1/sessions with the same session_id returns the WebSocket offer without creating a second session. Always ask for transport: "ws-h264", the only delivery ABot-World uses.

On the socket, authenticate with the offer's launch_token must be the first message, and client_hello with codecs: ["jpeg"] asks for one JPEG per binary message. Commands are accepted after worker_ready; before it they fail with 409 queued or 409 no_active_session.

Send the prompt and seed image together

set_conditioning stages the world prompt and seed image atomically, and start begins generation once both are set. After start, a new prompt with set_prompt moves the running world to it without a restart: prompt_accepted reports live_update_queued: true when the prompt is queued for the next block boundary.

Choose seed images with care. The character's identity is anchored by pre-generated reference views that exist only for curated seed images; with your own seed, the world runs without those anchors and the character stays less consistent.

Keys are quantised to the model's 12-frame blocks: held keys are sampled at each block boundary, a tap shorter than a block is latched and lands on the next one, and every press shows up from the next boundary. Movement is persistent state, which is why the example clears keys on keyup and on window blur. Every snapshot is acknowledged with input_accepted; one whose input_seq is not ahead of the last applied one comes back with stale: true and is not applied. set_movement, set_look_horizontal and set_look_vertical are single-axis alternatives, and pause, resume, reset and set_seed control the run.

The delivered stream is 1344 × 768, produced by learned super-resolution over an 896 × 512 generation canvas. Size the canvas with CSS and let the frames set its pixel size.

Track arrived is not first frame visible

Enable the keys at the first painted frame, not at worker_ready. The session's transport.fallback_policy.first_frame_timeout_ms (10 seconds by default) is how long the platform's own client waits for media after generation_started.

A canvas paints without play(), so autoplay rules do not apply here. If you pipe the canvas into a <video> with canvas.captureStream(), set muted and playsInline and handle play(): NotAllowedError means the browser wants a user gesture, so show a play button; AbortError only means a newer play() replaced the old one.

Release the session on unmount

Call world.release() when the component unmounts and on pagehide. It sends { action: "detach", connection_id, release: true } with keepalive, which ends the session at the worker and hands the capacity back; the response reports session_released: true. An idle character is still a ready session, and paused time is billed, so release is what stops the meter. If a tab dies without releasing, the worker keeps the session for a reconnect grace (30 seconds by default). For drops, see recovering a dropped stream.

Troubleshooting by the stage where startup stopped

Rejected commands return 422 with a command_… code over HTTP and a command_error message on the socket. conditions_ready reports has_prompt and has_image before start.

FAQ

What controls does ABot-World use?

The model's eight-key vocabulary: W, A, S and D move the character, and I, J, K and L turn the camera. On the API they are the set_input_state keys forward, left, back, right, look_up, look_left, look_down and look_right, sampled at each 12-frame block.

Is ABot-World a first-person or third-person world?

Third-person. Movement drives the on-screen character rather than a free camera, and the I, J, K and L keys turn the camera.

Can I use WebRTC with ABot-World?

No. ABot-World is delivered over the ws-h264 WebSocket transport only. The socket carries frames down and key snapshots up, which suits a keyboard-driven world.