How to use the LingBot API

The LingBot API on Windflow runs LingBot-World 2.0 as a realtime session: a seed image and a text prompt become a navigable stream that the user steers with persistent movement and look. Your server mints the token; the browser opens the session, stages the scene, sends start, draws frames from the session's WebSocket and sends key state up the same socket.

Hosted API or self-hosting

LingBot-World is an open model from Robbyant, Ant Group's embodied-intelligence unit; the official repository carries the research code and license. You can serve it yourself, for example with SGLang's LingBot-World recipe, and then own the GPUs, the browser transport, session lifecycle, authentication and metering. A hosted session on Windflow skips that layer: you pay per billable second and send the same controls described below. Access is through private beta, and GPU capacity is brought up on demand. The trade-offs are laid out in managed hosting versus rented GPUs.

Start from something runnable

The public browser examples provide a local Node server and a plain JavaScript client for LingBot, SANA-Streaming, Helios and LongLive. They keep the account key on the server, request a single-session token capped at 180 seconds, and include an explicit release action. The repository states its verification limits; approved beta access and billable model capacity are required for a live run.

You need an approved Windflow account and an API key that begins with wf_live_ (request access). Requests go to https://api.windflow.online.

1. Check the key (server). The response lists this model's commands, messages and limitations.

curl https://api.windflow.online/v1/models/lingbot \
  -H "Authorization: Bearer $WINDFLOW_API_KEY"

2. Mint a browser token (server). The account key never reaches the browser.

// For example app/api/windflow-token/route.ts. Authenticate your own user first.
export async function POST() {
  const res = await fetch("https://api.windflow.online/v1/tokens", {
    method: "POST",
    headers: {
      Authorization: `Bearer ${process.env.WINDFLOW_API_KEY}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({ model: "lingbot", max_sessions: 1 }),
  });
  return Response.json(await res.json(), { status: res.status });
}

The token is bound to this model (another model gets 403 model_mismatch) and can create one session. Read expires_at from the response instead of assuming a lifetime.

3. Open the session and draw frames (browser). Save this as windflow-session.js; the Helios, JoyAI-Video-Edit and ABot-World guides use the same file.

// windflow-session.js: plain fetch and WebSocket, no dependencies
const API = "https://api.windflow.online";

export async function call(path, jwt, body) {
  const res = await fetch(API + path, {
    method: body ? "POST" : "GET",
    headers: { Authorization: `Bearer ${jwt}`, "Content-Type": "application/json" },
    body: body && JSON.stringify(body),
  });
  const data = await res.json();
  if (!res.ok) throw Object.assign(new Error(data.error.message), data.error, { status: res.status });
  return data;
}

export function openSession(model, canvas, onMessage = () => {}) {
  const ctx = canvas.getContext("2d");
  let session, ws, closed = false, released = false;
  let painting = Promise.resolve();

  const paint = async (bytes) => {
    const frame = await createImageBitmap(new Blob([bytes], { type: "image/jpeg" }));
    if (canvas.width !== frame.width) Object.assign(canvas, { width: frame.width, height: frame.height });
    ctx.drawImage(frame, 0, 0);
    frame.close();
  };

  const release = () => {
    closed = true;
    ws?.close();
    if (!session || released) return;
    released = true;
    fetch(`${API}/v1/sessions/${session.session_id}`, {
      method: "POST",
      keepalive: true, // still sent while the page unloads
      headers: { Authorization: `Bearer ${session.jwt}`, "Content-Type": "application/json" },
      body: JSON.stringify({ action: "detach", connection_id: session.connection_id, release: true }),
    });
  };

  const ready = (async () => {
    const { jwt } = await (await fetch("/api/windflow-token", { method: "POST" })).json();
    session = await call("/v1/sessions", jwt, { model, transport: "ws-h264" });
    while (session.status === "queued" && !closed) {
      await new Promise((resolve) => setTimeout(resolve, 2000));
      const poll = await call(`/v1/sessions/${session.session_id}`, session.jwt);
      onMessage({ type: "queued", position: poll.queue_position });
      if (poll.status !== "queued") {
        session = await call("/v1/sessions", session.jwt, { model, session_id: session.session_id, transport: "ws-h264" });
      }
    }
    const offer = session.transport.offers.find((o) => o.kind === "ws-h264");
    if (closed || !offer) throw new Error(closed ? "released" : `no stream offer (status ${session.status})`);
    const url = new URL(offer.relay_path, API);
    url.protocol = "wss:";
    ws = new WebSocket(url);
    ws.binaryType = "arraybuffer";
    return new Promise((resolve, reject) => {
      ws.onopen = () => {
        ws.send(JSON.stringify({ type: "authenticate", token: offer.launch_token }));
        ws.send(JSON.stringify({ type: "client_hello", codecs: ["jpeg"] }));
      };
      ws.onmessage = ({ data }) => {
        if (typeof data !== "string") {
          painting = painting.then(() => paint(data)).catch(console.warn);
          return;
        }
        const msg = JSON.parse(data);
        if (msg.type === "worker_ready") resolve(msg);
        if (msg.type === "failed") reject(new Error(msg.message ?? msg.error));
        onMessage(msg);
      };
      ws.onclose = (event) => {
        onMessage({ type: "socket_closed", code: event.code });
        reject(new Error(`socket closed (${event.code})`));
      };
    });
  })();
  ready.catch(release); // a failed start must not leave a session running

  return {
    ready,
    release,
    get session() { return session; },
    get socket() { return ws; },
    command: (type, fields = {}) =>
      call(`/v1/sessions/${session.session_id}/commands`, session.jwt, { command: { type, ...fields } }),
    upload: async (file) => {
      const intent = await call("/v1/uploads", session.jwt, {
        filename: file.name, mime: file.type, size: file.size, session_id: session.session_id,
      });
      const put = await fetch(intent.upload_url, { method: "PUT", headers: intent.headers, body: file });
      if (!put.ok) throw new Error(`upload failed (${put.status})`);
      return call(`/v1/uploads/${intent.file_id}/complete`, session.jwt, {});
    },
  };
}

queued and socket_closed are messages this file generates; every other message comes from the session.

4. Stage the world and start it. LingBot needs both a prompt and a seed image before start. Stage them in one set_conditioning call:

import { openSession } from "./windflow-session.js";

const world = openSession("lingbot", document.querySelector("canvas"), (msg) => console.log(msg.type));
await world.ready; // worker_ready: commands are accepted from here
const seed = await world.upload(document.querySelector("input[type=file]").files[0]);
await world.command("set_conditioning", { prompt: "A misty pine forest trail at dawn", image: { id: seed.id } });
await world.command("start");

5. Send held keys as ordered snapshots. Each set_input_state carries the complete set of held keys and a rising input_seq. The example sends it on the open socket, which skips an HTTP round trip per key:

const KEYS = { KeyW: "forward", KeyS: "back", KeyA: "left", KeyD: "right", ArrowLeft: "look_left", ArrowRight: "look_right" };
const held = new Set();
let inputSeq = 0;

function sendKeys() {
  world.socket.send(JSON.stringify({
    type: "set_input_state", input_seq: ++inputSeq, pressed_keys: [...held], client_ts: Date.now(),
  }));
}
addEventListener("keydown", (event) => {
  const key = KEYS[event.code];
  if (!key || held.has(key) || event.target.closest?.("input, textarea")) return;
  held.add(key);
  sendKeys();
});
addEventListener("keyup", (event) => {
  if (held.delete(KEYS[event.code])) sendKeys();
});
addEventListener("blur", () => {
  held.clear();
  sendKeys();
});

Wait until the session is ready

Capacity is brought up on demand, so POST /v1/sessions can answer status: "queued" with a queue_position and no transport offers. Keep polling GET /v1/sessions/{id}: a queue ticket nobody polls expires with error: "scheduler_queue_expired". When the status leaves queued, POST /v1/sessions with the same session_id returns the WebSocket offer without creating a second session.

On the socket, authenticate with the offer's launch_token must be the first message, and client_hello with codecs: ["jpeg"] asks for one JPEG per binary message. Commands are accepted after worker_ready; before it they fail with 409 queued or 409 no_active_session.

Send the prompt and seed image together

set_conditioning stages the world prompt and seed image atomically, which is what start reads. Changing either one after start with set_prompt, set_image or set_conditioning restarts the sampler, so the change is not seamless. To change what happens in a running world, send an event instead:

await world.command("trigger_event", { eventId: "storm-1", prompt: "A thunderstorm rolls in over the trees" });

trigger_event layers a one-shot prompt onto the world's base prompt at the next generation-block boundary without restarting the sampler. eventId is echoed back in acknowledgements, and an optional positive revision lets a later event supersede an earlier one. It is accepted only after start.

Movement and look are persistent state: the world keeps moving until a snapshot without that key arrives, which is why the example clears keys on keyup and on window blur. The worker acknowledges every snapshot with input_accepted; one whose input_seq is not ahead of the last applied one comes back with stale: true and is not applied. Vertical look is accepted, but the engine's action vocabulary is yaw-only in this release, so the example binds left and right look only. set_movement and set_look_horizontal are single-axis alternatives to snapshots.

When every chunk of a run completes and the session is still started, the server starts the next run with the same prompt and seed image, so a generation_complete followed by a new generation_started is normal.

Track arrived is not first frame visible

Enable movement controls at the first painted frame, not at worker_ready. The session's transport.fallback_policy.first_frame_timeout_ms (10 seconds by default) is how long the platform's own client waits for media after generation_started.

A canvas paints without play(), so autoplay rules do not apply here. If you move the picture into a <video> element, through canvas.captureStream() or the session's livekit offer, set muted and playsInline and handle play(): NotAllowedError means the browser wants a user gesture, so show a play button; AbortError only means a newer play() replaced the old one.

Release the session on unmount

Call world.release() when the component unmounts and on pagehide. It sends { action: "detach", connection_id, release: true } with keepalive, which ends the session at the worker and hands the capacity back; the response reports session_released: true. Stopping movement does not stop billing, and neither does pausing: the session stays ready until it is released. If a tab dies without releasing, the worker keeps the session for a reconnect grace (30 seconds by default). For drops, see recovering a dropped stream.

Troubleshooting by the stage where startup stopped

Rejected commands return 422 with a command_… code over HTTP and a command_error message on the socket. conditions_ready reports has_prompt and has_image before start.

FAQ

How do I control LingBot-World with WASD?

Send set_input_state snapshots on the session's WebSocket: the full set of held keys from forward, back, left, right, look_left and look_right, with a rising input_seq. Clear the set on key release and when the window loses focus.

Can I change the world while it runs?

Yes, with trigger_event, which applies a one-shot prompt at the next generation-block boundary without restarting. Changing the base prompt or seed image restarts the sampler.

Does stopping movement stop billing?

No. An idle world is still a ready session. Release it with { action: "detach", connection_id, release: true } when the player leaves.