Skip to main content

Multimodal: image input

So here's the upgrade: multimodal image input is one option at create(), not a new API. You declare that pictures are coming, hand prompt() a Blob, and the same Gemini Nano that read your text now tells you what it sees — every pixel staying on the machine. Up to now it was working blind; now you give it eyes.

What you'll build​

  • Probe multimodal availability defensively, so an unfamiliar build can't throw in your face.
  • Opt a session into image input with expectedInputs so the runtime loads the vision tower.
  • Turn a dropped, pasted, or picked file into a Blob and preview it.
  • Downsample the picture to a modest size, then ask with role-wrapped content parts.
  • Stream the answer back and destroy() the session on teardown.
Prerequisites

Desktop Chrome with built-in AI enabled, and you've already met the Prompt API — this is that same LanguageModel, plus eyes. New to any of this? Start with Setup & the availability lifecycle and skim the compatibility matrix.

Probe for image support​

Same availability() call as every other API, with one option bolted on: you ask specifically about images. Wrap it in try/catch anyway — you can't count on every build recognising the option, and one that doesn't may throw instead of politely returning a string. So you treat a throw as a no.

demo.js
async function getMultimodalAvailability() {
  if (typeof LanguageModel === 'undefined') return 'unavailable';
  try {
    return await LanguageModel.availability({ expectedInputs: [{ type: 'image' }] });
  } catch {
    return 'unavailable'; // an unrecognised option may throw — treat it as a no
  }
}

Opt into image input at create()​

One option turns a text session into a vision session. expectedInputs declares that both text and images are coming, and that declaration is what makes the runtime load the vision tower up front. Skip it and the session rejects every image you send it. The languages ride inside those same declarations — en in, en out, no extra option needed. Wire a monitor — the first multimodal create() may pull the model down, and that download blocks the promise until it finishes.

demo.js
let sessionPromise = null;

function getSession() {
  if (sessionPromise) return sessionPromise;
  sessionPromise = LanguageModel.create({
    expectedInputs: [{ type: 'text', languages: ['en'] }, { type: 'image' }],
    expectedOutputs: [{ type: 'text', languages: ['en'] }],
    monitor(m) {
      m.addEventListener('downloadprogress', (e) => {
        console.log(`Downloading model ${Math.round(e.loaded * 100)}%`);
      });
    },
  }).catch((err) => {
    sessionPromise = null; // let the next call retry a failed download
    throw err;
  });
  return sessionPromise;
}

Get a Blob from the user​

The model takes a handful of image types — an ImageBitmap, an ImageData, an <img>, a <canvas> — but a Blob is the one you'll reach for, because it's what the browser already hands you. A file picker. A drag-and-drop. A clipboard paste. All three normalise to the same Blob, so one setImage() covers every path.

demo.js
const fileInput = document.getElementById('file');
const drop = document.getElementById('drop');
const preview = document.getElementById('preview');
let currentImage = null;

function setImage(blob) {
  if (!blob || !blob.type.startsWith('image/')) return;
  currentImage = blob; // a Blob — hand it straight to prompt() later
  preview.src = URL.createObjectURL(blob);
}

// 1. File picker
fileInput.addEventListener('change', () => setImage(fileInput.files[0]));

// 2. Drag and drop
drop.addEventListener('dragover', (e) => e.preventDefault());
drop.addEventListener('drop', (e) => {
  e.preventDefault();
  setImage(e.dataTransfer.files[0]);
});

// 3. Clipboard paste
document.addEventListener('paste', (e) => {
  for (const item of e.clipboardData.items) {
    if (item.type.startsWith('image/')) { setImage(item.getAsFile()); return; }
  }
});

Downsample before you ask​

Behind the glass the model works from a small internal view of the picture, not your full-resolution file. So a giant phone photo buys you little but a slower prompt and more memory — the answer barely moves. Shrink it yourself first. Draw the image into a canvas capped on the long side (512px is plenty here), then reach for canvas.toBlob, which hands you null on failure — so check for it.

demo.js
async function downsample(blob, max = 512) {
  const bitmap = await createImageBitmap(blob);
  const scale = Math.min(1, max / Math.max(bitmap.width, bitmap.height));
  const canvas = document.createElement('canvas');
  canvas.width = Math.round(bitmap.width * scale);
  canvas.height = Math.round(bitmap.height * scale);
  canvas.getContext('2d').drawImage(bitmap, 0, 0, canvas.width, canvas.height);
  bitmap.close();
  return new Promise((resolve, reject) => {
    canvas.toBlob((out) => (out ? resolve(out) : reject(new Error('toBlob returned null'))), 'image/jpeg', 0.9);
  });
}

Ask about the image​

Here's where the shape changes. A multimodal prompt() doesn't take a string — it takes an array of role-wrapped content parts: one user message whose content is a list of {type, value} pieces. The part key is value. Not image, not data, not src. Get that key wrong and the call throws, so it's worth staring at once.

demo.js
async function ask(question) {
  const session = await getSession();
  const image = await downsample(currentImage, 512);
  const reply = await session.prompt([
    {
      role: 'user',
      content: [
        { type: 'text', value: question },
        { type: 'image', value: image }, // key is `value` — not image/data/src
      ],
    },
  ]);
  return reply; // the full string reply
}

Stream the answer instead​

For a one-word label, prompt() is fine. For a description, you don't want the user staring at a spinner while the whole paragraph assembles offscreen — so swap in promptStreaming() and paint the text as it lands. One thing that bites everyone exactly once: the chunks are deltas, the new slice each time, not the whole answer so far. Append, never replace.

demo.js
const output = document.getElementById('output');

async function askStreaming(question) {
  const session = await getSession();
  const image = await downsample(currentImage, 512);
  const stream = session.promptStreaming([
    {
      role: 'user',
      content: [
        { type: 'text', value: question },
        { type: 'image', value: image },
      ],
    },
  ]);
  output.textContent = '';
  for await (const chunk of stream) {
    output.textContent += chunk; // chunks are deltas — append, don't replace
  }
}

Destroy the session on teardown​

The vision tower is heavier than a plain text session — it holds GPU memory the entire time it's alive. So you keep one session, reuse it for every question, and hand it back when the page goes away. Leave sessions orphaned and you starve the GPU until the tab reloads.

demo.js
window.addEventListener('beforeunload', () => {
  if (sessionPromise) sessionPromise.then((s) => s.destroy()).catch(() => {});
});
Try it

Run it locally: open 05-multimodal/index.html from the chrome-ai-course repo in desktop Chrome. Or use the hosted demo: Multimodal on windowai.danduh.me.

Expected: you drop, paste, or pick an image, type a question, and the answer streams in underneath — with nothing leaving the machine (open DevTools → Network and watch it stay silent). The hosted API docs walk the same calls.

Requires: desktop Chrome with Gemini Nano and image input available — see Setup.

Gotchas & troubleshooting​

The session rejects your image

Cause: you called create() without expectedInputs, so the runtime never loaded the vision tower and the session is text-only.

Fix: declare expectedInputs: [{type:'text', languages:['en']}, {type:'image'}] at create time. It has to be there up front — you can't teach a text session to see after it's born.

availability() throws instead of answering

Cause: a build that doesn't recognise the expectedInputs option may throw rather than return a string.

Fix: wrap the probe in try/catch and treat a throw as "unavailable". Don't assume every build accepts the option.

The first prompt hangs for a long time

Cause: the first multimodal create() triggers the Gemini Nano download, and create() doesn't resolve until that finishes — several GB on a cold machine.

Fix: pass a monitor and show progress. e.loaded is a 0..1 fraction, so multiply by 100 for a percentage. (e.total is there too, but it's always 1 — loaded already is your fraction.)

create() rejects before the model even downloads

Cause: the model wasn't on disk yet and you called create() outside a user gesture. Starting the download needs one.

Fix: kick off that first create() from a click or similar. Once the model is downloaded, later create() calls don't need a fresh gesture. In this demo the first create() rides the Ask button's click, so you're already covered.

The answer is confidently wrong

Cause: Nano is a generalist. It's weak on dense or handwritten text, on counting a crowd of things, and on faces — and it won't warn you that it's guessing.

Fix: keep the ask to "roughly what's here". For exact counts, precise coordinates, handwriting, or recognising a specific person, reach for a specialised model — this isn't the tool.

The tab gets sluggish after a few images

Cause: orphaned sessions keep the vision tower resident in GPU memory. Spin up a fresh session per prompt and you leak one every time.

Fix: create one session, reuse it, and destroy() it on teardown. One in, one out.

Recap​

  • availability({expectedInputs:[{type:'image'}]}), wrapped in try/catch, tells you whether images are on the table.
  • expectedInputs at create() loads the vision tower; without it the session stays blind.
  • A Blob from a picker, a drop, or a paste is all the same to prompt().
  • Downsample before you ask — the answer barely moves, at a fraction of the cost.
  • Content parts are role-wrapped and keyed by value; stream the deltas and append them.
  • Keep one session, destroy() it on the way out.

Cloud vision rents you a pair of eyes by the photo. This one forgets the picture the second you close the tab.

Next steps​

Next: The Summarizer API

Last updated on