Skip to main content

Observability & tracing

Ok. Observability for a cloud model is handed to you in the response: logprobs, the token bill, the model version, a server-side span, all riding back over the wire. On-device inference hands you none of it. No request over the wire. No confidence score. No per-response token bill. No model version to even ask about.

So you build the trace yourself.

Client-side, from the handful of signals Chrome actually exposes — about thirty lines of code — and nothing leaves the device unless you decide to send it. This lesson stacks that tracer up in four levels: a console.log, a structured span with the same shape every call, a traceStream wrapper that captures real latency and time-to-first-token, and a hand-off that ships the span as OpenTelemetry gen_ai.* attributes.

What you'll build​

  • Manufacture on-device AI observability from the few signals Chrome gives you — latency, TTFT, output size, context usage, and errors.
  • Define one AiSpan shape and emit the same record on every call, then fan it out to sinks with addSink.
  • Wrap a streaming call in traceStream so it stays a pass-through — the caller renders incrementally while the span captures TTFT and output chars.
  • Map an AiSpan onto OpenTelemetry gen_ai.* attributes as an INTERNAL span, and know the honest limits of shipping it anywhere.
  • Gate tracing behind an opt-in and keep raw prompts out of it, so production visitors pay nothing and nothing private leaks.
Prerequisites

Desktop Chrome with built-in AI switched on, and a call worth tracing. This page builds on the Prompt API and streams from the Summarizer, so it helps to have run one already. It also leans on the confidence field from LanguageDetector, and name-drops Translator. New here? Start with Setup & the availability lifecycle, and keep the compatibility matrix handy for what's stable where. It assumes you already know the four availability states — it won't re-teach them.

What you can actually capture​

So what does Chrome give you to work with? Not much, and that scarcity is the whole lesson. There's no server span to read, so every row below is something you measure in your own JavaScript, around your own await.

SignalHow you get itWhich APIs
Wall-clock latencyperformance.now() around the awaitall
Time to first token (TTFT)first chunk of a *Streaming() callstreaming APIs
Output size — a char proxy, not tokensaccumulate chunk.lengthstreaming APIs
Context / token accountingsession.contextUsage and session.contextWindowPrompt API
Availabilityavailability() stateall
Download progressmonitor(m) then downloadprogress, e.loaded is 0..1all
ErrorsDOMException.name (QuotaExceededError, AbortError…)all
Real confidence{ detectedLanguage, confidence }LanguageDetector only

That last row is the tell. The only genuine confidence number in the entire built-in AI surface comes from LanguageDetector. The Prompt API, the Summarizer, the Translator, the lot — no logprobs, no per-token probabilities, no output-token count, no way to query which Gemini Nano build answered you.

Timings and sizes. That's the raw material, so that's what the tracer is built from.

One thing to fix in your head before the code: context accounting is Prompt API only. Read session.contextUsage against session.contextWindow. Nothing else in the built-in AI surface exposes it.

Level 0 — time the call with console.log​

Start dumb. Wrap the await in a performance.now() pair, log the milliseconds and the output length, walk away.

level0.js
async function summarize(text) {
  const summarizer = await Summarizer.create({ type: 'key-points', outputLanguage: 'en' });
  const t0 = performance.now();
  const result = await summarizer.summarize(text);
  console.log('[ai] summarize', {
    ms: Math.round(performance.now() - t0),
    chars: result.length, // a char proxy — on-device gives you no token count
  });
  summarizer.destroy();
  return result;
}

Push a few of those into an array and the DevTools console hands you a table for free:

level0-table.js
console.table(window.__aiSpans);

That's a real, working baseline. It's also ephemeral, unstructured, and impossible to ship. So give it a shape.

Level 1 — give every call a structured span​

One record per AI call, the same fields every time, so you can filter it, sort it, render it, and later map it onto anything. This is the AiSpan, and it carries only fields Chrome actually gives you — no invented confidence, no fake token bill.

demo.js
// One record per AI call. Only signals Chrome actually exposes.
function newSpan(api, op, stream) {
  const id = (crypto.randomUUID && crypto.randomUUID()) || 'ai-' + Date.now();
  return { id, ts: Date.now(), api, op, stream, latencyMs: 0, finish: 'ok' };
  // later, optionally: ttftMs, outChars, contextUsage, contextWindow, errorName
}

A span nobody reads is a console.log with extra steps, so give it somewhere to go. A sink is just a function that takes a span; addSink registers one and hands you back an unsubscribe. Wrap each sink in a try/catch — a misbehaving sink must never break the actual call.

sinks.js
const sinks = [];
function addSink(sink) {              // register a consumer; returns an unsubscribe
  sinks.push(sink);
  return () => sinks.splice(sinks.indexOf(sink), 1);
}
function emit(span) {
  for (const sink of sinks) {
    try { sink(span); } catch { /* a broken sink never breaks the traced call */ }
  }
}

// Two sinks, both get every span:
addSink((span) => console.log('[ai]', span.api + '.' + span.op, span));
addSink((span) => renderRow(span)); // your in-page panel, IndexedDB, whatever

Level 2 — capture latency, TTFT, and errors​

Here's where the numbers get real. The stream is where the time goes, so the signal you actually want is the gap before the first chunk lands — time to first token — separate from the total. You measure it by wrapping the stream.

traceStream takes the same (api, op, session, run) and returns a pass-through ReadableStream. The caller reads that stream exactly like the original and still renders chunk by chunk; the span emits once — when the stream ends, errors, or gets cancelled — carrying TTFT, accumulated outChars, the finish state, and the error name if it blew up.

demo.js
function traceStream(api, op, session, run) {
  const span = newSpan(api, op, true);
  const t0 = performance.now();
  let first = true, chars = 0, done = false;

  const finalize = (finish, errorName) => {
    if (done) return;
    done = true;
    span.latencyMs = performance.now() - t0;
    span.outChars = chars;
    span.finish = finish;
    if (errorName) span.errorName = errorName;
    try { // Prompt API only — getters can throw on a destroyed session
      span.contextUsage = session?.contextUsage;
      span.contextWindow = session?.contextWindow;
    } catch { /* session gone — skip the numbers */ }
    emit(span);
  };

  let reader;
  return new ReadableStream({
    start(controller) {
      try { reader = run().getReader(); }
      catch (e) { finalize(classify(e), e && e.name); controller.error(e); }
    },
    async pull(controller) {
      if (!reader) return;
      try {
        const { done: end, value } = await reader.read();
        if (end) { finalize('ok'); controller.close(); return; }
        if (first) { span.ttftMs = performance.now() - t0; first = false; } // TTFT
        chars += value.length;
        controller.enqueue(value); // pass the chunk straight through — caller still renders
      } catch (e) { finalize(classify(e), e && e.name); controller.error(e); }
    },
    cancel(reason) { finalize('abort', reason && reason.name); return reader && reader.cancel(reason); },
  });
}

const classify = (e) => (e && e.name) === 'AbortError' ? 'abort' : 'error';

The one-shot calls — prompt(), summarize(), a plain await — get the simpler sibling, traceCall, which times the promise, reads result.length for outChars, records the error, and re-throws it unchanged. Same span, no stream plumbing.

demo.js
async function traceCall(api, op, session, run) {
  const span = newSpan(api, op, false);
  const t0 = performance.now();
  try {
    const result = await run();
    span.latencyMs = performance.now() - t0;
    if (typeof result === 'string') span.outChars = result.length; // char proxy
    try { // Prompt API only — getters can throw on a destroyed session
      span.contextUsage = session?.contextUsage;
      span.contextWindow = session?.contextWindow;
    } catch { /* session gone — skip the numbers */ }
    emit(span);
    return result;                       // hand the caller its value, untouched
  } catch (e) {
    span.latencyMs = performance.now() - t0;
    span.finish = classify(e);
    if (e && e.name) span.errorName = e.name;
    emit(span);
    throw e;                             // re-throw unchanged — the tracer never swallows
  }
}

Errors are first-class here, not an afterthought. On-device, the most useful thing you can log is why it failed: QuotaExceededError (input over the window — it carries requested and contextWindow), NotSupportedError (an unsupported input or output — a language tag or expectedInputs/expectedOutputs combo the model can't do; the generic can't-run-here case isn't an error at all, it's availability() returning unavailable), and AbortError when someone cancels. Capture DOMException.name and you've turned a silent "works but subtly wrong" into a searchable field.

Level 3 — ship it as OpenTelemetry spans​

You can keep every span local forever — an in-page panel, IndexedDB, done, no backend. But if you want the gen_ai.* vocabulary the rest of the industry already speaks, it ports cleanly. The tooling mostly doesn't, but the attribute names do.

Emit an INTERNAL span, not a client span — the model runs in-process, there's no server.address to point at. Then map the fields you captured onto the GenAI semantic conventions.

demo.js
function genAi(span) {
  const attributes = {
    'gen_ai.operation.name': span.op,
    'gen_ai.provider.name': 'chrome.builtin',
    'gen_ai.request.model': 'gemini-nano',       // Chrome exposes no version
    'gen_ai.request.stream': span.stream,
    'gen_ai.response.finish_reasons': [span.finish],
  };
  if (span.ttftMs != null)
    attributes['gen_ai.response.time_to_first_chunk'] = Number((span.ttftMs / 1000).toFixed(3));
  if (span.contextUsage != null)
    attributes['gen_ai.usage.input_tokens'] = span.contextUsage; // approximate (cumulative)
  if (span.errorName) attributes['error.type'] = span.errorName;
  return { name: 'gen_ai.' + span.op, kind: 'INTERNAL', attributes };
}

Two honest caveats baked into that mapping. gen_ai.request.model is hard-coded to 'gemini-nano' because Chrome won't tell you the build number — provenance is unobservable from JS. And gen_ai.usage.input_tokens is the Prompt API's cumulative contextUsage, which is approximate and session-wide, not a per-call output count. Ship it, but label it in your head as a proxy.

Now, "ship it somewhere" has a catch at every door. Here's the state of the ecosystem, no spin.

TargetNo-backend, in the browser?Catch
console.log / console.tableyesephemeral
In-page panel + IndexedDByesyou build the UI
OpenTelemetry gen_ai.*, local exporteryesOTLP export needs a collector (CORS)
Sentryyes, to a DSNdata leaves the device
Langfusescores onlyfull tracing needs a Node backend

The nuance behind the rows: a ConsoleSpanExporter or an IndexedDB exporter keeps OpenTelemetry fully local, but pushing to an OTLP collector from a browser is a CORS-and-CSP minefield that needs a running collector — a backend. Sentry runs client-side and takes manual spans, but every one of them ships to a DSN and off the machine. Langfuse's browser SDK is scores-and-feedback only — perfect for a thumbs-up button on a trace some server already created, useless for tracing on its own.

One more thing before any of this ships: make tracing opt-in. Off by default means production visitors never pay for a wrapper they don't need, and nothing gets captured unless a developer turns it on.

opt-in.js
function isTracingEnabled() {
  if (typeof globalThis.__AI_TRACE__ === 'boolean') return globalThis.__AI_TRACE__;
  try { return localStorage.getItem('ai:trace') === '1'; } catch { return false; }
}

// In your call site: only wrap when tracing is on — zero overhead otherwise.
const stream = isTracingEnabled()
  ? traceStream('summarizer', 'summarize', s, () => s.summarizeStreaming(text))
  : s.summarizeStreaming(text);

Keep raw prompts out of the trace​

Everything through Level 2 stays on the device. That's not a nice-to-have — it's the entire reason someone picked on-device AI over a cloud API in the first place, and a sloppy trace throws it away.

The rule is short. Log sizes and timings, not content. outChars and latencyMs, yes. The user's actual prompt and the model's actual answer, no — not by default. The moment you add Sentry, Langfuse, or an OTLP collector, spans leave the machine, so make every external sink an explicit opt-in the user or the developer chooses, never the default path. Keep the content local and you keep the promise you shipped on.

Try it

Run it locally: open 15-observability-and-tracing/index.html from the chrome-ai-course repo in desktop Chrome. Or use the hosted demo: on-device trace viewer (with the API walkthrough).

Expected: you run a traced Summarizer call; the summary streams in, and the same captured span renders three ways side by side — the console.log line, the structured AiSpan object, and its OpenTelemetry gen_ai.* attributes. Switch to the Prompt API to watch contextUsage show up.

Requires: desktop Chrome with Gemini Nano available — see Setup & the availability lifecycle.

Gotchas & troubleshooting​

Most of these come from tracing the wrong kind of call, or expecting a signal on-device simply doesn't emit.

contextUsage is always undefined

Symptom: your span never carries contextUsage or contextWindow.

Cause: one of two. You're tracing the Summarizer or Translator, which don't expose context accounting — that's Prompt API only. Or you read it after destroy(), and the getter threw.

Fix: read session.contextUsage before you destroy the session, and expect it to be absent on every API except the Prompt API.

TTFT is missing or equals latency

Symptom: ttftMs is undefined, or it lands within a millisecond of latencyMs.

Cause: you wrapped a one-shot prompt() or summarize() with traceStream, or the whole reply arrived in a single chunk. TTFT only exists when there's a stream to be first in.

Fix: use traceStream on the *Streaming() variant and traceCall on the one-shot. If TTFT still hugs latency, the output was short enough to arrive in one chunk — that's real, not a bug.

The stream renders nothing after wrapping

Symptom: you pipe a call through traceStream and the UI stays blank.

Cause: you consumed the stream inside the wrapper, or you never read the stream it returned. The wrapper is a pass-through, not a sink.

Fix: read the returned stream — for await or getReader() — and render each chunk yourself. The span emits on its own when the stream ends; your job is still to render it.

outChars doesn't match your token dashboard

Symptom: outChars is nowhere near the token count you expected.

Cause: outChars is a character count — accumulated chunk.length — not tokens. On-device inference gives you no output-token number at all.

Fix: treat outChars as a proxy for output size. The only token-ish figure available is the Prompt API's contextUsage, and even that is cumulative and approximate.

Recap​

  • On-device inference has no server span — no logprobs, no confidence, no token bill, no model version. You manufacture the trace from Chrome's own signals: latency, TTFT, output chars, context usage, and errors.
  • One AiSpan shape per call, fanned out through addSink, so every call reads the same and a broken sink never breaks the call.
  • traceStream wraps a *Streaming() call as a pass-through — the caller still renders incrementally while the span captures TTFT and outChars; traceCall handles the one-shot.
  • Map the span onto OpenTelemetry gen_ai.* as an INTERNAL span to ship it; the honest matrix tells you where each backend costs you a round-trip or a leak.
  • Everything through Level 2 is local; gate tracing behind an opt-in, log sizes and timings, and keep raw prompts and responses on the device.

Everyone else is paying Datadog to trace a call that never left the tab.

Next steps​

Next: Evaluation.

Last updated on