Local models

Use experimental_chrome for media. Media is off by default; enable each input kind and provide an application resolver. The resolver receives an AG-UI ImagePart or AudioPart plus the run's AbortSignal, and returns an encoded Blob and an optional dispose callback. This example accepts inline base64 only. URL and provider file handles require your application's explicit resolution policy; Hashbrown does not fetch them.

import { experimental_chrome } from '@hashbrownai/core';
import { useChat } from '@hashbrownai/react';

const transport = experimental_chrome({
  outputLanguage: 'en',
  media: {
    images: {},
    inputLanguages: ['en'],
    resolve(part, { signal }) {
      signal.throwIfAborted();
      if (part.source.type !== 'data') {
        throw new Error('This application accepts inline media only.');
      }
      const bytes = Uint8Array.from(atob(part.source.value), (c) =>
        c.charCodeAt(0),
      );
      return { blob: new Blob([bytes], { type: part.source.mimeType }) };
    },
  },
});

export function LocalImage({ base64 }: { base64: string }) {
  const chat = useChat({ transport, system: 'Describe the supplied image.' });
  return (
    <section>
      <button
        onClick={() =>
          chat.sendMessage({
            role: 'user',
            content: [
              { type: 'text', text: 'What is the dominant color?' },
              {
                type: 'image',
                source: { type: 'data', value: base64, mimeType: 'image/png' },
              },
            ],
          })
        }
      >
        Describe image
      </button>
      <p>{chat.lastAssistantMessage?.content}</p>
      {chat.error && <p>{chat.error.message}</p>}
    </section>
  );
}

AG-UI inline sources use { type: 'data', value: base64, mimeType }; value contains canonical base64 bytes, without a data: URL prefix. Keep media in user messages. Canonical AG-UI content parts survive message lowering and immutable history snapshots; ordinary JSON arrays keep their legacy representation. The transport preserves part order, validates and decodes images before creating a session, and releases its ImageBitmaps and resolver cleanup callbacks when each run settles, including errors and cancellation. Each media run gets its own native session. Replayed history is resolved again.

media.inputLanguages defaults to ['en'] and describes input independently of outputLanguage. The transport checks the exact requested input modalities and languages. Missing or unknown availability fails closed. A downloadable or downloading model requires an explicit events.downloadRequired callback; without it, media requests fail before creation. Prepare downloads through your application's user-initiated flow and retain Chrome's activation requirements.

Image input accepts PNG, JPEG and WebP, with defaults of 10 MiB encoded bytes and 16 million decoded pixels per image (images.maxBytes, images.maxPixels). Audio is a separate opt-in: add audio: { maxBytes: 10 * 1024 * 1024, maxDurationSeconds: 30 } to media, then send an audio part with the same source shape and mimeType: 'audio/wav'. Only encoded PCM16 RIFF WAV with one or two channels and an 8–48 kHz sample rate is supported. The transport preserves that encoding and sample rate. Defaults are 10 MiB and 30 seconds per audio part; media.maxTotalBytes defaults to 20 MiB across the run's media history. Raw PCM, video, documents, media tool results and media combined with transformRequest are unsupported.

Chrome 155 has transcribed the retained generated speech fixture through its raw native API. The real Hashbrown media suite passed all four Angular/React image/audio cases, with one resolution, disposal, session creation and session destruction per run. Deterministic shims check transport/framework contracts separately from real inference.

experimental_chrome_tools() selects a separate text-and-tools transport. It requires Chrome's experimental tool-use feature and its v4 model; normal Chrome 155 defaults do not expose the typed tool interfaces. The local test launcher enables AIPromptAPIToolUse,OnDeviceModelConversationBackend,AIApiFoundationalModel:model_version/v4 only for its dedicated browser profile. These flags are experimental and do not enable a hosted fallback.

Prepare the model in an explicit user-initiated download flow before starting chat. The transport checks exact tools, languages and input/output types. A model that requires downloading fails unless the application supplies an events.downloadRequired callback; that callback must honor Chrome's actual activation requirement. timeoutMs defaults to 180000 and bounds setup and inference. Stop or dispose an abandoned response to release its owned session.

import { experimental_chrome_tools, s } from '@hashbrownai/core';
import { useChat, useTool } from '@hashbrownai/react';

const transport = experimental_chrome_tools();

export function LocalCounter({ value }: { value: number }) {
  const readCounter = useTool({
    name: 'read_counter',
    description: 'Read the current application counter value.',
    schema: s.object('Read counter', {}),
    handler: async () => ({ value }),
    deps: [value],
  });
  const chat = useChat({
    transport,
    tools: [readCounter],
    system:
      'Call read_counter before answering. Return only the value property as plain text.',
  });
  return (
    <section>
      <button
        onClick={() =>
          chat.sendMessage({ role: 'user', content: 'Read the counter.' })
        }
      >
        Read counter
      </button>
      <p>{chat.lastAssistantMessage?.content}</p>
    </section>
  );
}

Tool execution remains in Hashbrown and your application. The browser receives only declarations. Tool handlers and application-owned approval/denial gates run through the existing framework workflow; no browser callback auto-executes them. Each model run owns a fresh session and replays authoritative history, including typed success/error results with their canonical call IDs.

The tested Chrome 155/v4 model emits empty native callID values. The adapter assigns a stable canonical ID only when that completed turn contains exactly one call. Ambiguous batches requiring ID repair fail before execution. Valid native IDs are preserved; malformed, conflicting and duplicate IDs fail. By default, schema-plus-tools and UI requests are rejected. The finalization opt-in also accepts component UI through the existing UI chat APIs. Media and media tool results remain unsupported. Use experimental_chrome() for the existing text/schema or explicitly enabled media paths. The adapter never removes a schema silently to make a tool request work.

Without the finalization option, final replies remain unconstrained text. Clear application instructions can ask the model to extract a particular property from a JSON tool result, but formatting is a model-quality concern: an initial native workflow returned a correct receipt inside fenced JSON despite a plain-string request. Hashbrown preserves that output. The default factory does not constrain final replies.

To opt into schema output after tool planning, use useStructuredChat with your existing schema:

import { useStructuredChat } from '@hashbrownai/react';

const transport = experimental_chrome_tools({
  schemaFinalization: 'after-tool-planning',
});

const chat = useStructuredChat({
  transport,
  tools: [readCounter],
  schema: s.object('Counter answer', { value: s.number('Current counter') }),
  system:
    'Call read_counter first. Return a JSON object with only value containing the current counter.',
});

Tool planning runs without a response constraint so it can request additional or dependent tools. Hashbrown executes validated calls through the usual handlers and permission gates. Planning prose is excluded from the public conversation. Once a planning turn contains no calls, the same native session receives one additional prompt with responseConstraint set to the chat schema. Only that final answer streams through Skillet. With no tools declared, the schema is applied directly without an extra planning pass.

This is an experimental, opt-in orchestration policy, and adds a terminal model pass when tools are declared. A finalization tool call fails before execution; unsupported schemas and invalid output surface as errors. There is no weaker constraint retry, fence removal or hosted fallback. The capability report's adapter.structuredOutput describes adapter policy; inference: 'unknown' means the actual model/schema combination still needs verification. Media composition remains unsupported.

For component UI, use useUiChat with the same transport, your existing exposeComponent(...) definitions and ordinary tools. The generated component wrapper is the final response schema; you do not supply another schema or turn components into executable tools. For an exposed counterComponent whose numeric value prop displays the result of the existing counter tool:

import { useUiChat } from '@hashbrownai/react';

const chat = useUiChat({
  transport,
  components: [counterComponent],
  tools: [readCounter],
  system:
    'Call read_counter before answering. Render the counter component with the exact value returned by the tool.',
});

Only final UI text reaches the existing Skillet parser and trusted component renderer. The model still has to select a component and copy tool data correctly. A successful simple-object response does not establish support for every UI kit, recursive children or larger catalogs. Test the exact kit on your browser/model; unsupported native constraints and transport failures surface as errors. Component parsing keeps the existing renderer behavior: unknown components are omitted and invalid props can retain a fallback without mounting the trusted component. Capability adapter.ui describes the explicit adapter policy, while inference stays unknown.

The initial real Chrome 155/v4 component canary failed strict workflow checks with a catalog offering both status and panel components: it made unrequested tool calls, attempted undeclared rendering tools, or added components. Clearer instructions passed 4/8 cases but did not establish reliability. The dated component UI research records the failures separately from the passing adapter and renderer contracts. A follow-up with workflow-specific catalogs passed 2/8 and 4/8 in two fixed production samples. Only the no-tools cases passed in both samples; tool-driven component UI has not met the native qualification gate.

Expose only the tools and components needed for the current workflow. A lookup that renders a leaf does not need a dependent tool or a container component. Dependent workflows can expose both required tools while keeping the component kit small. Tool argument validation and component schemas check structure; they do not guarantee that the model follows the requested sequence or returns the requested number of components. Keep permission checks in application handlers and measure exact tool calls and rendered output on the actual model.

Applications can opt into explicit tool phases when a workflow has known steps:

const transport = experimental_chrome_tools({
  schemaFinalization: 'after-tool-planning',
  toolPhase({ completedCalls }) {
    if (completedCalls.some((call) => call.result.type === 'error')) {
      throw new Error('The counter lookup failed.');
    }
    return completedCalls.length === 0
      ? { type: 'tool', name: 'read_counter' }
      : { type: 'finalize' };
  },
});

The selector runs once per request against an owned, deeply frozen snapshot of the current user turn. It includes the user message, thread/run IDs and completed calls with argument objects and raw success/error text. Earlier user turns do not advance the current phase. Keep this function pure; use the existing tool handlers and permission gates for application actions. The application decides how to handle failed tools; there is no automatic retry or phase counter.

A tool phase exposes only the selected tool and requires exactly one call before any handler can run. Zero calls, multiple calls or an out-of-phase name fail. Its planning prose is hidden. A final phase directly constrains the full conversation and tool results without another planning turn. Every attempted finalization tool call fails before execution, including a call after partial streamed text.

Chrome requires native tool declarations to replay typed tool history. During finalization Hashbrown retains only declarations referenced in validated history, including earlier turns, in the original catalog order. These are visible to the model for replay compatibility; the final phase permits zero executable calls. With no historical calls it retains none. History is not flattened into prose. A private phase instruction is appended to the same native prompt after the unchanged history; it adds no inference pass and is omitted from framework history.

This policy controls execution phases, not UI cardinality or model quality. The phase research records browser compatibility and exact-output results separately. Two fixed production samples each passed 8/10 phase cases. All single-component lookup, dependent, no-tools and cross-turn replay cases passed in both frameworks in both samples (16/16). Nested workflows failed either inside Chrome before a handler ran or by choosing the wrong panel heading. The full native suites remain failed; nested UI is unqualified on this measured Chrome/model configuration.

Stopping a conversational run or sending a replacement request retains the original user request and committed history with an explicit runtime cancellation marker before subsequent input. The marker stays in model history and is omitted from ordinary assistant rendering and schema/UI parsing. Both real v4 Stop cases returned the exact new lookup token with that marker; cleanup and history assertions passed independently. Keep testing follow-up instructions on the actual model; a marker does not guarantee model obedience.

detectChromeToolsCapabilities() reports serializable exact-option and model availability evidence without downloading or inferring. Its inference value stays unknown: interface/availability checks do not prove model quality. Eight raw native compatibility canaries passed exact success/error responses and fresh-session replay in both browser projects. See the local conformance commands for opt-in native and aimock targets; all local-model suites refuse CI.

Use experimental_webllm({ engine }) when the application wants to select and manage its own browser model. Hashbrown does not import WebLLM or declare it as a dependency or peer dependency. Install WebLLM in the consuming application only. Structural consumer checks cover WebLLM 0.2.85 and 0.2.86 direct and worker engine declarations. These checks establish assignability, with upstream declaration-body checking skipped; they do not establish model capability.

Create the worker in the application:

// local-worker.ts
import { WebWorkerMLCEngineHandler } from '@mlc-ai/web-llm';

const handler = new WebWorkerMLCEngineHandler();
self.onmessage = (event: MessageEvent) => handler.onmessage(event);

Prepare a model in a user-initiated action, then retain the resulting transport:

import { CreateWebWorkerMLCEngine } from '@mlc-ai/web-llm';
import { experimental_webllm } from '@hashbrownai/core';

const worker = new Worker(new URL('./local-worker.ts', import.meta.url), {
  type: 'module',
});
const engine = await CreateWebWorkerMLCEngine(worker, modelId, {
  initProgressCallback: (progress) => console.log(progress.text),
});
const transport = experimental_webllm({ engine });

Pass transport to <HashbrownProvider transport={transport}> or an individual hook's options.

The application owns model selection, downloads, worker creation, caching, and unloading. Pin the application runtime, model weights and compatible WASM library together when reproducibility matters; a model ID backed by mutable catalog URLs is not an immutable artifact identity. Stop active Hashbrown runs before calling engine.unload() and terminating the worker. The default text transport supports text, structured output, and trusted UI; it rejects tools and multimodal messages. Keep UI kits and schemas small, and test nested schemas on the real model.

Hashbrown converts the response schema to WebLLM's json_object format with a JSON-encoded schema string. This differs from OpenAI's json_schema wrapper. A truncated response raises WEBLLM_OUTPUT_TRUNCATED rather than completing successfully with partial JSON.

For an application-selected image model, experimental_webllm({ engine, media: { images: {}, resolve: resolveImage } }) enables image input through an image-capable engine. The application must qualify that model and implement resolveImage for every image source, including inline data. Hashbrown does not fetch image sources implicitly. Text-only input remains the default. See the vision qualification report for the tested model, exact cases, and limits.

The image opt-in accepts user-message images followed by trailing text; interleaved text/image layouts, audio, documents and media tool results are rejected. The resolver owns source access and receives an abort signal. Defaults bound each encoded image to 10 MiB and 16 million decoded pixels, with at most eight images and 20 MiB of encoded images across the run's complete history. Set smaller limits for your application and keep the resolver's network permissions explicit. These transport bounds are not a model's context or VRAM capacity.

Use experimental_webllm_tools for opt-in, application-controlled tool phases. The application selects one declared tool; the model generates its arguments as JSON. Hashbrown buffers the complete response, validates the arguments, and then dispatches through the existing tool handler. The application selects finalization when it is ready for schema output or trusted component UI. This protocol uses structured output rather than WebLLM's native tool mode.

import { experimental_webllm_tools } from '@hashbrownai/core';

const transport = experimental_webllm_tools({
  engine,
  toolPhase({ completedCalls }) {
    if (completedCalls.some((call) => call.result.type === 'error')) {
      throw new Error('The lookup failed.');
    }
    return completedCalls.length === 0
      ? { type: 'tool', name: 'lookup_record' }
      : { type: 'finalize' };
  },
});

Declare lookup_record through the normal tool API and supply a final response schema through structured chat or UI chat. The selector receives an owned, frozen view of the current user turn and completed calls, including their exact arguments and success/error result text. Use those results to choose dependent phases. Tool selection does not grant permission: handlers still own application authorization, confirmation, and side effects.

Tool phases accept text input only. Argument schemas must use closed objects with every property required, plus supported primitive, array, enum, and anyOf shapes and constraints. References, optional object fields, format annotations, and reserved property names (__proto__, constructor, prototype) are rejected before inference. Final response schemas retain the normal structured-output/UI path. Invalid, truncated, oversized, or cancelled argument generation dispatches no complete tool call; arguments are not streamed into handlers or repaired automatically. Keep component catalogs and workflows small and qualify them on the selected model.

WebLLM 0.2.86 with Phi-3.5-vision-instruct-q4f16_1-MLC passed six fixed text-only cases: lookup to schema, dependent lookup to schema, and lookup to one trusted Status component in both frameworks on an M1 Max. See the structured tool qualification report for exact assertions, timings, and limits. Broader catalogs, nested tool-plus-UI, and other devices remain unqualified.

The image and structured-tool opt-ins are available since Hashbrown 0.8.3. All engines remain application-owned; no WebLLM dependency is added to Hashbrown. The local conformance commands separate deterministic contracts from opt-in real-model qualification. Hashbrown WebLLM audio interop and cross-hardware qualification remain deferred. Upstream WebLLM 0.2.86 already lists Gemma E2B text/audio prebuilds; that availability does not add an audio transport to Hashbrown or qualify a model on your device. The earlier research assumption that no upstream prebuilt audio model existed was incomplete.

Upstream releases and Hashbrown qualification are separate records. The npm registry published WebLLM 0.2.87 on October 30, 2026; the Qwen JSON-termination fix is an example of why runtime/model/library versions matter for constrained output. Our retained real-model runs used 0.2.86, so 0.2.87 is not a newly qualified configuration here.

Native WebLLM tool mode is not interchangeable with the structured-phase transport. The 0.2.87 upstream protocol still restricts tool models and rejects caller-supplied system messages and response formats in that mode. Passing tools through would not preserve Hashbrown's system/schema/history contract. The application-selected protocol above keeps those contracts explicit.

Other browser runtimes are evaluation candidates, not additional Hashbrown support claims. Transformers.js has experimental constrained generation, and wllama has WASM/WebGPU and multimodal/tool interfaces. Neither has been qualified by these WebLLM tests. An integration must separately establish message mapping, exact tool replay, schema completion, cancellation and resource ownership.

Only one Hashbrown run may reserve a given engine at a time, including across transport instances. Overlap fails with MODEL_BUSY. Do not invoke that engine directly while Hashbrown owns a run. Cancellation interrupts the active generation without unloading the engine; pending engine creation or an interrupted stream keeps the reservation until it drains through completion. Cancellation during media preparation releases the reservation promptly before engine creation. The UI can stop before draining finishes, so an immediate replacement run may briefly receive MODEL_BUSY. An ignored or abandoned transport response must be disposed.

Local inference does not enable a hosted fallback automatically. Keep permission to send prompts off-device an explicit application choice. Model downloads and application telemetry have their own network behavior; local generation alone does not make the entire application offline.

The local-only real-worker suite checks structured output, nested trusted UI, and generation after Stop or token-limit truncation. The separate aimock suite checks transport contracts without downloading models. See the conformance test commands to run either locally; these suites reject CI execution.