Local AI in the browser: is WebGPU actually ready?

Running the model on a server costs money and privacy. Running it in the browser costs download.

What we measured

Four open models (0.5B to 7B), three browsers, two devices. Measured: initial load, time to first token, tokens per second, memory ceiling.

Results

A 7B model produces 14 tokens per second on a laptop — enough for chat. But the model file is 4.1 GB and downloads on first visit: 40% of users leave at that point.

The cache question

With Cache Storage the second visit loads in 2 seconds. That changes the experience entirely. Our first release shipped without caching, and that single omission halved engagement.

What to build today

For small models (under 1B) the browser is genuinely ready. For large ones it is not. And in every case: keep a server-side fallback path.