pandora-embeddings

Semantic search over AI-generated shot descriptions in pan.do/ra video archives — pad.ma and indiancine.ma.

A vision model has watched these films and written a description of every shot. This project makes those descriptions searchable by meaning rather than by keyword, so you can ask for "a woman waiting alone at a railway platform at night" and get the shot, even though not one of those words appears in the description.

It is a standalone Django service, deliberately not part of the pan.do/ra codebase.

Talk to the machine, get hallucinations back. Every description here was written by a model watching the film. It invents names, misreads on-screen text, and is confidently wrong on a regular basis. You are searching the descriptions, not the films.

---

What are embeddings? (the short version)

Suppose you had to arrange every shot description in the archive on an enormous noticeboard, with one rule: things that mean similar things must be pinned near each other. Descriptions of boats end up in one region, descriptions of crowded markets in another, and "a puppy sprints across grass" lands right next to "a dog runs through a field" — not because they share words, but because they mean nearly the same thing.

An embedding is that pin's position, written down as a long list of numbers (1,024 of them here). A model reads a piece of text and produces the coordinates. Text that means similar things gets similar coordinates.

Searching is then simple: your query gets turned into coordinates by the same model, and the database returns whatever is pinned nearest. That is what a vector database is — one that can answer "what's near this point?" over millions of points, quickly. We use PostgreSQL with the pgvector extension.

Why bother, when the archive already has search? Because pan.do/ra's search matches substrings and cannot rank. Ask it for an aeroplane in flight and it finds nothing — the descriptions say "aircraft" and "airplane". Worse, when it does match, results come back in arbitrary order, because there is no relevance score anywhere in the API. Meaning-based search has neither problem.

What it costs. Two honest limitations:

1. It can be confidently wrong. Nearness in meaning is not the same as correctness. A query about grief will happily return a shot of someone merely looking downward.
2. It cannot find what was never described. If the model didn't notice the tiger, no amount of clever searching will surface it. The ceiling here is the quality of the machine's descriptions, and that ceiling is uneven.

Both are why the UI keeps a keyword mode alongside the semantic one, and why every result links straight back to the actual video in the archive.

---

What it does

Search shots	Describe a shot; get the moments that match, ranked.
Search scenes	The same over merged runs of shots — "find this sequence".
Search objects	Detected objects rather than whole shots, with bounding boxes — "a person on a bicycle, in the left third".
Search on-screen text	Title cards, credits, censor certificates and burned-in subtitles, indexed separately from the visual description.
Say when nothing matches	Search always returns its nearest neighbours; a calibrated confidence band distinguishes answers from mere neighbours.
Similar shots	Given one shot, find where else in the archive that image recurs.
Similar films	Films whose overall visual character resembles another's.
Scene structure	Where a film's shots stop resembling each other — semantic scene boundaries, which are not the same as camera cuts.
Motif clusters	Recurring visual patterns within a film, each with a representative frame.
Embedding map	Every shot as a point on a 2-D map of the embedding space, regions named by what they are about; hover for the frame, click for true nearest neighbours, search to fly to the hits, or walk neighbour-to-neighbour in graph mode. Several layouts to switch between, and a map of whole films.
Axes	Plot every shot on two axes you name — indoors ↔ outdoors, celebration ↔ grief, or the corpus's own principal directions — so that position means something.
Paths and steering	The straight line in meaning between two shots, stepping through real shots; or "this shot, but more like night".
Stand inside it	/3dmap: the archive arranged around wherever you stand — a shot, a sentence you type, or either one nudged by a phrase. Distance is true similarity, direction is two lenses you choose, and the nearest shots are their actual frames, playing. Click anything to stand there.

Current corpus: the archive's qwenie:Qwenhallus list — 243 films, every shot described and its objects detected, all embedded. (Counts on /api/healthz.)

---

Quick start (local)

No Docker needed. Postgres.app bundles pgvector; any PostgreSQL 16+ with pgvector 0.8+ works.

uv venv && uv pip install -e '.[dev,embed,viz]'
cp .env.example .env
createdb pandora_embeddings
./manage.py migrate

./manage.py sync_items            # the films in the archive's qwenie:Qwenhallus list
./manage.py sync_annotations      # their shot descriptions (machinedescriptions layer)
./manage.py sync_regions          # detected objects, attached to those shots
./manage.py check_quality         # flag films whose descriptions degenerated
./manage.py build_scenes
./manage.py embed                 # shots; then --target scenes, items, regions
./manage.py build_indexes --sparse
./manage.py project_map --all     # behind /map + /3dmap: layouts, vectors, axes, films

./manage.py runserver             # UI at /, maps at /map and /3dmap, API at /api/docs

./manage.py pipeline runs everything up to build_indexes in order; the archive is the source of truth and every stage is idempotent, so re-running it is the refresh (see docs/embeddings.md §1a). ingest_srt srts/ and ingest_bboxes load the same data from local files when the archive is unreachable.

Deploying

On the server:

git clone https://code.with.camp/CAMP/pandora-embeddings.git
sudo ./pandora-embeddings/deploy/bootstrap.sh        # first time
sudo /srv/pandoraembed/app/deploy/update.sh          # every time after

bootstrap.sh keeps its own checkout at /srv/pandoraembed/app; the one you cloned is only used to find the script. Everything the deployment owns — code, virtualenv, model cache, static files, dumps, config — lives under /srv/pandoraembed, owned by the pandoraembed user. The database starts empty — ship it with ./deploy/make-dump.sh locally and restore-dump.sh on the server.

See docs/deployment.md for the full runbook.

---

Does it work?

15 golden queries over the full corpus, k=10:

mode	recall@10	MRR	+ rerank (MRR)
lexical (Postgres FTS)	0.60	0.600	0.600
sparse (BGE-M3 lexical)	0.80	0.800	0.800
dense	1.00	0.956	1.000
hybrid	1.00	1.000	1.000

Relevance is derived from corpus content, not hand-assigned film ids. That distinction matters: a first attempt named one "correct" film per query and scored hybrid at 0.58 — but in 199 Indian films a bullock cart on a dirt path appears in nine and a typewriter in forty-four, so single-film judgments baked false negatives into every score.

Plus a false-confidence rate over queries whose correct answer is nothing — currently 0.00. That metric exists because recall alone cannot see over-confidence, which is the failure people actually hit.

Caveat: the positive set is saturated. At 1.00/1.00 it can no longer tell hybrid from dense or measure what reranking is worth. Harder queries are needed before the retrieval knobs can be tuned on evidence rather than argument. If you hit a search that feels wrong, that is exactly what should be added to search/eval_queries.py.

---

Documentation

docs/embeddings.md	Technical reference: the pipeline, every tunable parameter, model options, evaluation.
docs/how-it-works.md	Why the design is shaped the way it is — the data's quirks and what they forced.
docs/deployment.md	Server runbook: bootstrap, update, restore, rollback, troubleshooting.
docs/pandora-api.md	Working with the live pan.do/ra API, and its traps.

Layout

Path	What
archive/	Items, annotations, scenes; the pan.do/ra client and ingest
archive/ingest/normalize.py	Boilerplate stripping and facet extraction
embeddings/	Model registry, vector tables, BGE-M3 backend
search/	Retrieval, fusion, diversification, reranking, the API
search/views.py, search/templates/	The server-rendered UI
deploy/	Deployment scripts and service configuration
docs/	Documentation
srts/	Sample corpus: 199 films, 152,593 shot descriptions