Skip to content
G Gemma Embedding Playground Sound, photos and text in one space

How your audio becomes words

Your browser Lab server Modal (the model)

  1. 1 · Your browser

    Record

    Microphone, up to 60 s. Stops on its own at the limit.

  2. 2 · Your browser

    Convert

    Decoded and resampled to 16 kHz mono (what the model expects), sent as one WAV.

  3. 3 · Lab server

    Split

    10-second segments, plus the whole clip. Nothing is saved.

  4. 4 · Modal

    Embed

    The audio encoder turns each piece into 768 numbers: a point in the space.

  5. 5 · Lab server

    Compare

    Cosine similarity against 91,194 stored word vectors and the other input, at the size you pick.

  6. 6 · Your browser

    Show

    Nearest words for the clip and each segment, and how close A and B are.

Built once, ahead of time: every whole-word token in the model's 262,144-token vocabulary (91,194 words, all languages) the model's text encoder on Modal (as title: none | text: <word>) 91,194 vectors stored on the lab server, so step 5 costs nothing.

How it works

EmbeddingGemma 2 maps text, images, video and audio into one shared 768-dimensional space. Things that mean similar things should land near each other, whatever form they arrive in and whatever language they use.

The playground asks two questions of that space: how close are A and B, and which words does each land nearest?

The verdict

Cosine similarity is the real distance, but its scale depends on what is being compared: two unrelated sentences can still score 0.6, and a photo and a word that match can score lower than that. So the verdict ranks the pair against the vocabulary instead: how many of the 91,194 words sit closer to A than B does (or closer to B than A does, whichever is fewer). Fewer than 10 is “Same meaning”, fewer than 200 “Closely related”, fewer than 3,000 “Related”, and anything else “Unrelated”. The thresholds are rules of thumb from trying known pairs, not part of the model.

The vocabulary

The words come from the model itself: every whole-word token in its 262,144-token vocabulary, in every language. Each one was run through the model once (as a document, title: none | text: <word>) and stored on this server, so searching costs nothing. Expect fragments and non-English words in the results; that is the raw space. Language tags come from word-frequency lists (the wordfreq project) and are left off when a word is ambiguous; words shared by close languages get both, like ES/PT.

Your inputs

Text goes to the model as a search query. Photos are shrunk to 1024 px in your browser. Recordings are converted to 16 kHz mono (the rate the model expects); the server sends the whole clip plus each 10-second segment to the model. Nothing is stored: not the audio, the photo, the text or the results.

Smaller vectors

The model is trained so the first 512, 256 or 128 of its 768 numbers still work as an embedding. Switch the size to see how much of the answer survives at a sixth of the storage.

The map

The map in “For the curious” is a 2D projection (PCA) fitted to just A, B and their nearest words. Two components can only show a sliver of the space: points close in the real space can look far apart. Use it for rough orientation; trust the scores.

Example media

The ready-made examples were embedded once ahead of time, with the same model calls the live inputs use. Their media are from Wikimedia Commons, resized or trimmed:

The model runs on Modal and shuts down after two idle minutes, so the first request after a quiet spell takes a minute or so while it starts.