Streaming a video diffusion model to the browser: WebRTC or WebSocket?

To stream a video diffusion model to a browser, generate in chunks, pace the chunks to the playout rate, and deliver them over WebRTC when the network path is clean or over a WebSocket carrying H.264 or JPEG frames when it is lossy or locked down. Both work; the choice is a property of your users' networks, not of the model.

This guide explains why the model's generation pattern drives the transport decision, compares the two options, and lists what to measure.

Generation arrives in chunks, playback does not

An autoregressive video model does not emit frames one at a time at a steady rate. It produces a chunk of frames, then the next chunk, each conditioned on the last. Models in the Windflow catalog generate chunks as short as 8 to 33 frames, and some generate windows of ten seconds or more.

That has three consequences for delivery:

1. Bursty supply. A whole chunk becomes available at once, then nothing until the next chunk is done. The viewer needs a steady 24 frames a second.
2. A playout buffer is mandatory. The client holds a short queue and drains it at the display rate. Too small and the stream stalls between chunks; too large and every control you send feels slow.
3. Time to first frame includes generation. For windowed models the first frame can only appear after the first window is complete, so it is much later than the frames that follow.

If generation is slower than playback, no transport can save the stream. The first requirement is that the model produces a second of video in less than a second, with margin; the transport decides whether that margin survives the network.

Option A: WebRTC

WebRTC carries media over UDP with congestion control, loss recovery and jitter handling built into the browser. Windflow's WebRTC path runs through LiveKit.

Strengths

• Hardware-accelerated decode and a mature jitter buffer in every browser.
• Congestion control that adapts bitrate to the path instead of queueing behind it.
• Audio and video share one clock, which matters for models that generate sound.
• Works well on clean, low-latency links.

Costs

• More moving parts: signalling, ICE, TURN relays for restrictive networks, and a media server.
• On a lossy path, recovery depends on keyframes. A lost reference frame forces a keyframe request and a burst, and the viewer sees a freeze until it lands.
• Harder to debug than a socket.

Option B: WebSocket carrying H.264 or JPEG

A single TCP connection carries encoded frames in one direction and control messages in the other. Windflow exposes this as the ws-h264 transport.

Strengths

• One connection, one port, easy to pass through proxies and firewalls.
• Controls and video share a connection, so ordering is simple.
• In-order delivery makes the client trivial to reason about, and frames can be decoded with the browser's VideoDecoder or drawn from JPEG.
• With intra-only JPEG there are no reference frames to lose, so there is no keyframe recovery to stall on.

Costs

• TCP retransmits lost packets in order, so under loss one late packet delays everything behind it. You need your own pacing and a drop policy.
• No built-in congestion adaptation for media; bitrate has to be managed by the sender.
• JPEG frames use far more bandwidth than H.264 at the same quality.

Which one to choose

Windflow negotiates per session: each model lists the transports it supports in order of preference, and the client takes the first it can use. The supported transports below come from the current catalog.

A field result: when WebSocket won

On a lossy China-to-US path, we found that sending JPEG frames over one WebSocket removed stalls that WebRTC's keyframe recovery caused, as long as generation was rate-matched to playout. WebRTC stays the default wherever the link is clean. The write-up is in the field note JPEG over WebSocket beat WebRTC on a lossy trans-Pacific link; it is a decision about one bad path, not a verdict on the transport.

Use one pacing policy and measure input age

Whatever the transport, coordinate the producer and playback clocks:

• State who owns the clock. A camera-driven engine can already pace its output. An additional transport clock must drain that stream without permanently retaining time lost during a hiccup.
• Bound latency as well as stalls. Record buffer age and growth. Keeping every old source frame indefinitely can preserve sequence numbers while making the video increasingly late. Any drop or reset policy must respect the model's state and be described to the client.
• Size the cushion from observations. Measure chunk variation, network arrival gaps and visible stalls before changing a buffer. A smooth stream is not enough evidence that its input is fresh.

The SANA camera investigation is a concrete example. Removing a second pacing clock and fixing mixed-size camera encoding restored approximately 25 presented fps in six recorded follow-up runs. A remaining, flat source backlog still corresponded to about 2.8 seconds median scene-change response in the still-camera cases. Changing the transport name alone would not explain those two separate effects.

Controls have three moments

A prompt change or key press passes through three points, and a good client tells the user about each:

1. Acknowledged: the server received it.
2. Applied: the model started using it at a chunk boundary.
3. Visible: the first frame that reflects it reached the screen.

Acknowledged is not visible. Showing an immediate acknowledgement and then updating when the change is actually on screen keeps a laggy-feeling stream from feeling broken.

What to measure

• Time to first frame after starting a session.
• Delivered frame rate against the target, per session.
• Stalls: count and total duration per minute of viewing.
• Control-to-visible delay: from a control being sent to the frame that shows it.
• Bytes per second, to size the link you ask your users to have.

Measure from the client. Server logs show the frames you sent, not the frames the viewer saw.

FAQ

Should I use WebRTC or WebSocket to stream video from a diffusion model?

Use WebRTC on clean networks and when audio must stay in sync. Use a WebSocket carrying H.264 or JPEG on lossy, long-haul or restricted networks, or when you want controls and frames on one connection. The decision depends on your users' network paths more than on the model.

Why does my generated video stutter even though the model is fast enough?

Chunked generation delivers frames in bursts. Without a playout buffer the stream stalls between chunks, and on a lossy path WebRTC keyframe recovery can freeze the picture. Rate-match generation to playout and keep a shallow buffer.

Can I send controls and receive video on the same connection?

Yes. With the ws-h264 transport, controls travel up the same WebSocket that carries frames down. With WebRTC, media uses the peer connection and controls use the session's control plane.

Which Windflow models support which transports?

Supported transports vary by model and are listed in the table above and the model catalog. Request a supported transport; a shared session API does not imply every media transport is available for every model.