How do you run an AI model in the browser?

Why run AI in the browser at all

Most AI features call a server: the photo or text is uploaded, a GPU in a data center runs the model, and the result comes back. In-browser AI turns this around. The model is downloaded once, and the visitor's own device does the work.

• Privacy: photos, audio and documents stay on the device. Nothing needs to be uploaded or stored, which also simplifies compliance with privacy laws in many countries.
• No server cost per use: the site pays to host the model files, not for GPU time. This makes free tools possible.
• Works offline after the first load, and has no network delay per request.
• No account or API key is needed.

The trade-offs are real. The first visit needs a large download, speed depends on the visitor's hardware, and only small and medium models fit. Large chat models with tens of billions of parameters still need a server. Tasks that fit well today include background removal, image upscaling, depth estimation, speech to text, translation, text classification, embeddings and small image or language models. For example, cv.cm's remove-background and speech-to-text tools run BiRefNet and Whisper this way, with nothing uploaded.

The four layers: model, runtime, backend, library

ONNX is an open format for neural networks. A model trained in PyTorch can be exported to ONNX, and ONNX Runtime Web can then run it with several backends. The WASM backend supports every ONNX operator. The GPU backends support a subset, so a model can fail on WebGPU and still work on WASM. Transformers.js adds what you would otherwise write yourself: resizing images, turning text into tokens, decoding audio, and turning raw output numbers into labels, masks or text.

WebGPU versus WebAssembly

WebGPU is a W3C web standard that gives pages access to the graphics card for both drawing and general computation. For neural networks, the compute part is what matters: matrix multiplications run on many GPU cores in parallel. WebAssembly runs compiled code on the CPU at close to native speed and works in every modern browser.

In practice, try WebGPU first and fall back to WASM. Do not decide by browser name. Ask the browser for an adapter and check its features:

// Check what this device can do before loading a model.
async function pickBackend() {
  if (!("gpu" in navigator)) return { device: "wasm" };      // no WebGPU API
  const adapter = await navigator.gpu.requestAdapter();
  if (!adapter) return { device: "wasm" };                    // API exists, no usable GPU
  const f16 = adapter.features.has("shader-f16");             // half-precision support
  return { device: "webgpu", f16 };
}

console.log(await pickBackend(), "threads:", self.crossOriginIsolated);

Multi-threaded WASM uses SharedArrayBuffer, which browsers only allow when the page is cross-origin isolated. That means sending the Cross-Origin-Opener-Policy and Cross-Origin-Embedder-Policy headers. Without them, WASM runs on a single thread and is much slower.

A minimal Transformers.js example

With Transformers.js, a task is one pipeline call. The library downloads the model, picks the files for the chosen device and data type, and returns a function you can call:

import { pipeline } from "@huggingface/transformers";

const { device } = await pickBackend();     // from the snippet above

// First call downloads the model and caches it. Later visits load from cache.
const classify = await pipeline(
  "sentiment-analysis",
  "Xenova/distilbert-base-uncased-finetuned-sst-2-english",
  {
    device,                                  // "webgpu" or "wasm"
    dtype: device === "webgpu" ? "fp32" : "q8",
    progress_callback: (p) => console.log(p.status, p.file ?? "", p.progress ?? ""),
  }
);

console.log(await classify("This runs entirely on my laptop."));
// [{ label: "POSITIVE", score: 0.99... }]

Run this in a module script or, better, in a Web Worker. Model inference can block for hundreds of milliseconds or more, and in a worker the page stays responsive. WebGPU is also available inside workers. Show download progress to the user, because the first load of a large model can take a while on a slow connection.

Download size, quantization and caching

The download is roughly the number of parameters times the bytes stored for each one. This simple arithmetic tells you whether a model is practical:

download size  ~  number of parameters  x  bytes per parameter

                      fp32 (4 B)   fp16 (2 B)   int8 / q8 (1 B)   q4 (0.5 B)
 50 million params      200 MB       100 MB          50 MB            25 MB
 500 million params     2.0 GB       1.0 GB         500 MB           250 MB
 1 billion params       4.0 GB       2.0 GB         1.0 GB           500 MB

Real files differ a little: some layers stay at higher precision,
and the tokenizer and config add a few MB.

Quantization stores weights with fewer bits: fp16 halves the size of fp32, 8-bit quantization (q8) quarters it, and 4-bit (q4) reduces it by about eight times. Smaller weights download faster and need less memory. The cost is some accuracy, which is usually small at 8 bits and more visible at 4 bits, especially on small models and on tasks such as image generation where small errors show. Transformers.js selects the variant with its dtype option. Its defaults are q8 on WASM and fp32 on WebGPU, and the available options depend on what the model publisher exported.

Caching makes the second visit fast. Transformers.js stores downloaded files with the browser's Cache API by default. Browsers give each site a storage quota based on disk size, often large enough for several gigabytes on a desktop, but this storage is best-effort: the browser can delete it when the device is low on space, and Safari can remove script-created data for sites that the user has not interacted with for several days. A site can ask for persistent storage with navigator.storage.persist() and check usage with navigator.storage.estimate().

Phones and other limits

• Memory: a model needs memory for its weights plus working space. Mobile browsers give each tab much less memory than desktops and may close the tab without warning if a model is too large. Keep mobile models small, often under a few hundred megabytes.
• WebGPU on phones: support depends on the browser, the operating system version and the GPU driver. Always have a WASM path or a clear message.
• Heat and battery: long GPU work makes phones hot, and the system then slows the chip down. A task that takes 2 seconds once may take much longer the fifth time.
• Data plans: warn users before a large download on a mobile connection.
• Precision: some GPUs lack fast half-precision (fp16) support or produce slightly different numbers, so outputs can differ a little between devices.
• Model licenses still apply. Running a model in the browser does not change its terms of use.

A checklist for shipping in-browser AI

1. Choose the smallest model that meets your quality bar, and test its quantized versions side by side.
2. Detect WebGPU with an adapter request, and fall back to WASM.
3. Serve the site over HTTPS, and enable cross-origin isolation if you need WASM threads.
4. Run inference in a Web Worker and show download and processing progress.
5. Do not download the model on page load. Wait until the user starts a task, so visitors who only read the page do not pay for gigabytes.
6. Host the model files at a pinned version, and cache them.
7. Test on a low-end phone, not only on a development laptop.