# Running LLMs in the Browser with WebGPU

Source: https://www.egnworks.com/blog/running-llms-in-the-browser-with-webgpu  
Author: Jacob Val  
Published: 2026-07-04  
Updated: 2026-07-07  
Category: AI Development  
Tags: WebGPU, WebAssembly, LLM

> Learn how to run LLMs directly in the browser using WebGPU, WebLLM, and Transformers.js. A practical guide covering architecture, model selection, Web Worker patterns, Chrome Built-in AI, and when on-device inference beats a cloud API.

---

## Introduction

For most of the web's history, any AI feature required a round trip to a server. The user's input traveled to a cloud endpoint, a GPU cluster computed the response, and the result came back over the network. That model works well when latency is acceptable and the cost per request is predictable. But as AI features become more frequent and more granular autocomplete on every keystroke, real time grammar correction, local document summarization the economics and latency of server side inference start working against the product.

Browser native AI changes the equation. In 2026, it is genuinely possible to run capable language models directly in a browser tab, with no server, no API key, and no data leaving the user's device. Three developments made this practical: WebGPU reaching stable cross browser support, a generation of quantized small language models that fit in consumer GPU memory, and a maturing set of JavaScript inference libraries that abstract the complexity.

This guide covers the full technical stack for browser based LLM inference what WebGPU actually does, how to integrate WebLLM and Transformers.js into a React application, how to choose the right model for a given task, and the honest constraints that determine when browser inference is the right call and when a server API still makes more sense.

## Why This Became Practical Now

Two years ago, browser based LLM inference was an interesting demo with limited practical application. The models were too large, the GPU access was too restricted, and the developer experience was rough. Several things converged to change that.

WebGPU, the successor to WebGL, shipped stable in Chrome in 2023 and has since reached solid support across Chrome, Edge, and increasingly Firefox and Safari. Unlike WebGL, which was designed for graphics rendering, WebGPU exposes general purpose GPU compute through compute shaders. This is exactly what neural network inference needs: massively parallel matrix multiplications running across thousands of GPU threads simultaneously. The performance gap between WebGPU and WebGL for ML workloads is not incremental WebGPU is 10 to 20 times faster for the kind of compute that transformer inference requires.

Simultaneously, quantization techniques matured to the point where models like Llama 3.2 1B and Qwen 2.5 0.5B can be compressed to Q4 quantization and fit comfortably in 1 to 2GB of GPU memory. At that size, a user's integrated GPU on an M series MacBook or a mid range laptop can run interactive inference without breaking a sweat. Published benchmarks for WebLLM on an M3 Max show Llama 3.1 8B at 41 tokens per second at Q4 quantization fast enough for most production use cases.

The cost story is equally compelling. One team reported cutting $8,000 per month in inference costs to near zero by moving classification and tagging tasks, which represented 60% of their API volume, to browser inference. For high frequency, lower complexity tasks, the unit economics are dramatically better than paying per token on a hosted API.

## The Browser AI Stack

Browser based LLM inference involves several layered components, each with a specific role.

### WebGPU

WebGPU is the GPU compute API that browser inference engines use for hardware acceleration. It provides compute shaders written in WGSL (WebGPU Shading Language), storage buffers for arbitrary read/write data, and explicit GPU resource management. For frontend developers, you will rarely write WebGPU code directly the inference libraries handle this layer. But understanding that it exists, and that it is what makes performance viable, informs decisions about browser support targets and fallback strategies.

### WebAssembly as CPU Fallback

Not every user's browser supports WebGPU. Firefox's support is still maturing; Safari is behind on some features. For users without WebGPU, WebAssembly (WASM) serves as the CPU fallback path. WASM based inference is significantly slower, but it means the feature degrades gracefully rather than failing entirely. The inference libraries discussed below handle this fallback automatically.

### Web Workers

LLM inference is computationally heavy. Running it on the browser's main thread blocks the UI and makes the page unresponsive during generation. Every production browser AI implementation should run the model in a Web Worker, keeping inference on a background thread and communicating results back to the main thread via message passing. This is non negotiable for a usable experience.

## The Primary Libraries

### WebLLM

WebLLM, built by the MLC AI team at CMU, is the most mature browser native LLM inference engine. It uses WebGPU for GPU acceleration, WASM for CPU fallback, and exposes an OpenAI compatible API that makes migrating from a cloud endpoint straightforward in many cases, it is just changing the endpoint URL.

```js
import { CreateMLCEngine } from "@mlc-ai/web-llm";

// Initialize the engine with a specific model
const engine = await CreateMLCEngine(
  "Llama-3.2-1B-Instruct-q4f16_1-MLC",
  {
    initProgressCallback: (progress) => {
      console.log(`Loading: ${Math.round(progress.progress * 100)}%`);
    },
  }
);

// The API mirrors OpenAI's chat completions
const reply = await engine.chat.completions.create({
  messages: [
    { role: "system", content: "You are a helpful assistant." },
    { role: "user", content: "Summarize this text in one sentence." }
  ],
  stream: true,
});

for await (const chunk of reply) {
  const delta = chunk.choices[0]?.delta?.content ?? "";
  process.stdout.write(delta);
}
```

WebLLM achieves up to 80% of native inference performance on the same hardware. It supports streaming responses, JSON mode, and function calling the same capabilities developers rely on from cloud APIs, running entirely in a browser tab.

### Transformers.js

Transformers.js by Hugging Face takes a broader approach, supporting not just text generation but the full range of Hugging Face tasks: text classification, named entity recognition, translation, text embeddings, image classification, speech to text, and more. It uses ONNX Runtime Web under the hood, with WebGPU as the primary acceleration backend.

```js
import { pipeline } from "@huggingface/transformers";

// Load a text classification pipeline
const classifier = await pipeline(
  "text-classification",
  "Xenova/distilbert-base-uncased-finetuned-sst-2-english",
  { device: "webgpu" }
);

const result = await classifier("This product is excellent!");
// [{ label: 'POSITIVE', score: 0.9998 }]
```

For tasks beyond pure text generation classification, embeddings, audio transcription, image analysis Transformers.js is usually the better choice over WebLLM.

### Chrome Built in AI (Prompt API)

Chrome ships with Gemini Nano on device, accessible via the Prompt API. This requires no model download (the model is already on the device for eligible Chrome users), which eliminates the first load penalty entirely.

```js
// Check availability first
const capabilities = await window.ai.languageModel.capabilities();

if (capabilities.available === "readily") {
  const session = await window.ai.languageModel.create({
    systemPrompt: "You are a helpful writing assistant.",
  });

  const response = await session.prompt(
    "Improve the grammar in this sentence: He go to store yesterday."
  );
  console.log(response);
}
```

The constraint is that Chrome Built in AI is currently only available in Chrome with specific flags or in Chrome 128+ for qualifying devices. It is not a universal solution, but for Chrome heavy user bases it provides the smoothest possible first load experience.

## Integration Pattern in React

A production React integration separates the model initialization lifecycle from the component that consumes it. Model loading is expensive and should happen once, not on every render.

```tsx
// hooks/useBrowserLLM.ts
import { CreateMLCEngine, MLCEngine } from "@mlc-ai/web-llm";
import { useState, useEffect, useRef } from "react";

export function useBrowserLLM(modelId: string) {
  const engineRef = useRef<MLCEngine | null>(null);
  const [status, setStatus] = useState<"idle" | "loading" | "ready" | "error">("idle");
  const [loadProgress, setLoadProgress] = useState(0);

  useEffect(() => {
    // Check WebGPU support before attempting to load
    if (!navigator.gpu) {
      setStatus("error");
      return;
    }

    setStatus("loading");

    CreateMLCEngine(modelId, {
      initProgressCallback: ({ progress }) => {
        setLoadProgress(Math.round(progress * 100));
      },
    })
      .then((engine) => {
        engineRef.current = engine;
        setStatus("ready");
      })
      .catch(() => setStatus("error"));
  }, [modelId]);

  const generate = async (messages: { role: string; content: string }[]) => {
    if (!engineRef.current) throw new Error("Engine not ready");
    return engineRef.current.chat.completions.create({ messages, stream: true });
  };

  return { status, loadProgress, generate };
}
```

```ts
// components/LocalChatWidget.tsx
import { useBrowserLLM } from "@/hooks/useBrowserLLM";
import { useState } from "react";

export function LocalChatWidget() {
  const { status, loadProgress, generate } = useBrowserLLM(
    "Llama-3.2-1B-Instruct-q4f16_1-MLC"
  );
  const [output, setOutput] = useState("");

  const handleSubmit = async (prompt: string) => {
    setOutput("");
    const stream = await generate([{ role: "user", content: prompt }]);
    for await (const chunk of stream) {
      const delta = chunk.choices[0]?.delta?.content ?? "";
      setOutput((prev) => prev + delta);
    }
  };

  if (status === "loading") {
    return <p>Loading model... {loadProgress}%</p>;
  }

  if (status === "error") {
    return <p>Browser AI not available. Using cloud fallback.</p>;
  }

  return (
    <div>
      <button onClick={() => handleSubmit("Summarize this article...")}>
        Summarize
      </button>
      <p>{output}</p>
    </div>
  );
}
```

## Choosing the Right Model

Model selection is the most consequential decision in a browser AI integration. The tradeoffs are between capability, model size, and first load download time.

For most practical production tasks in 2026, the sweet spot is models in the 0.5B to 3B parameter range at Q4 quantization. These models are fast, load from cache in seconds after the first download, and are capable enough for classification, summarization of short texts, grammar correction, entity extraction, and simple question answering.

Recommended starting points:

**Qwen 2.5 0.5B Q4** the smallest capable model, downloads around 400MB, ideal for classification and short generation tasks

**Llama 3.2 1B Q4** a solid general purpose model for most tasks, around 700MB download

**Phi 3.5 Mini Q4** Microsoft's small model, strong instruction following, around 2GB download

**Llama 3.1 8B Q4** the ceiling of what most consumer hardware handles comfortably, around 5GB, suitable for heavier generation tasks

A 70B model is not going to run in a browser tab in any foreseeable future. The relevant question is whether a 1B or 3B model is good enough for the specific task at hand. For an increasing number of tasks, the answer is yes.

## The Fallback Strategy

Browser AI is a progressive enhancement, not a hard requirement. Some users will be on browsers without WebGPU support, on low memory devices, or on corporate machines with locked down GPU access. A robust implementation always detects capability and routes gracefully.

```ts
async function getInferenceClient(prompt: string) {
  // Try browser inference first
  if (navigator.gpu) {
    try {
      const adapter = await navigator.gpu.requestAdapter();
      if (adapter) {
        return "browser"; // Use WebLLM
      }
    } catch {
      // Fall through to server
    }
  }

  // Fall back to server API
  return "server";
}
```

The hybrid approach that many teams settle on is to use browser inference for high frequency, lower complexity tasks autocomplete, classification, short summarization and route to a server API for heavy reasoning tasks, long form generation, or multimodal features. This keeps the API bill manageable while still providing the fast, private experience of local inference where it matters most.

## When to Use Browser Inference and When Not To

Browser inference makes the most sense when one or more of these conditions apply:

The feature handles sensitive data that should not leave the user's device

The task is high frequency and low complexity, making API costs significant at scale

Offline functionality is a product requirement

Latency is critical and a network round trip is too slow for the use case

Server side inference remains the better choice when:

The task requires a large, high capability model that cannot fit in browser memory

Deterministic, reproducible outputs are required

The user base includes a significant proportion of low end devices or older browsers

The feature involves multimodal inputs that current browser libraries handle poorly

## References

[WebLLM: High-Performance In-Browser LLM Inference Engine](https://github.com/mlc-ai/web-llm)

[Run AI Models in the Browser with WebGPU and WASM](https://maddevs.io/writeups/running-ai-models-locally-in-the-browser/)

[Running LLMs in the Browser: WebGPU, Transformers.js, and Chrome Built-in AI](https://pockit.tools/blog/run-llms-browser-webgpu-transformers-js-chrome-built-in-ai-guide/)

[WebGPU Browser AI Inference: Cut Client-Side LLM Costs](https://www.buildmvpfast.com/blog/webgpu-browser-ai-inference-cost-savings-2026)

[Browser-Native Agents and LLMs: The Complete Guide to In-Browser AI](https://wowdata.science/browser-native-agents-llms-in-browser-ai-guide-2026/)

[The Complete Guide to Local-First AI: WebGPU, Wasm, and Chrome Built-in Model](https://www.sitepoint.com/local-first-ai-webgpu-chrome-guide/)

[WebLLM: Run LLMs in Your Browser at 80% Native Speed](https://localaimaster.com/blog/webllm-browser-ai-guide)