The Met Search Lab Technical Guide

The guide distinguishes the documented production approach, the current public lab default, and options exposed specifically for experimentation. The lab reproduces architectural decisions rather than the complete production application, evidence set, or undistributed prompts.

Each search variant can use keyword search, semantic search, or a hybrid of both.

From collection object to search result

Elasticsearch stores the Met object record, generated text such as visual descriptions, and embeddings derived from generated visual descriptions or images. The pipeline first prepares that evidence for the index, then combines it at search time to produce ranked results and navigable filter counts.

The live lab keeps The Met's script-score execution and 50 / 50 hybrid weights, while its default evidence recipe is intentionally simplified. A Lab default badge marks the active Medium Visual Description recipe; changing one of its settings removes the badge.

1. Prepare complementary evidence

Keep concise catalog metadata as the primary literal signal. Generate an objective visual description for each artwork image, then embed that description as the semantic signal. The production pipeline currently uses The Met's internal Agentic Vision v4 visual-description generator and a 768-dimensional Gemini text embedding. This public lab instead defaults to the Medium field from its reproducible visual-description generator. The internal v4 strategy remains documented as the historical production-aligned baseline used to calibrate the completed study, but it is not an active search option or field in the current lab index.

Use the same embedding model, dimensions, normalization, and task conventions for indexed documents and live queries. Store the model and source-text version with each vector so a reindex cannot silently mix incompatible embeddings.

2. Map the vector field

Set index: false because script score reads stored vector values directly and does not need an approximate vector index. The lab indexer also excludes embeddings from _source to avoid retaining and returning large arrays there; scripts can still access the vector values.

{
  "mappings": {
    "_source": {
      "excludes": ["embeddings.*"]
    },
    "properties": {
      "embeddings": {
        "properties": {
          "embedding__visual_description_lengths_v1_medium_gemini_768": {
            "type": "dense_vector",
            "dims": 768,
            "index": false
          }
        }
      }
    }
  }
}

3. Build a complete match universe

The lexical clause saturates BM25 with score / (score + 8). The vector clause rejects documents below the similarity threshold, rescales the remaining cosine range to 0–1, and applies its weight. A bool.should union means a document can enter through either literal or semantic evidence.

Abridged structural example: the production and lab implementations use the larger field set documented below, plus eligibility and facet clauses appropriate to their collections.

{
  "query": {
    "bool": {
      "should": [
        {
          "script_score": {
            "query": { "multi_match": { "query": "<query>", "fields": ["title^4", "artistDisplayName^3", "medium^1.5"] } },
            "script": {
              "source": "(_score / (_score + params.saturation_k)) * params.weight",
              "params": { "saturation_k": 8, "weight": 0.5 }
            }
          }
        },
        {
          "script_score": {
            "query": { "bool": { "filter": [{ "exists": { "field": "embeddings.embedding__visual_description_lengths_v1_medium_gemini_768" } }] } },
            "script": {
              "source": "double c = cosineSimilarity(params.vector, 'embeddings.embedding__visual_description_lengths_v1_medium_gemini_768'); if (c < params.threshold) return 0; return ((c - params.threshold) / (1 - params.threshold)) * params.weight;",
              "params": { "vector": "<query vector>", "threshold": 0.65, "weight": 0.5 }
            },
            "min_score": 0.000001
          }
        }
      ],
      "minimum_should_match": 1
    }
  }
}

Elasticsearch documents the per-match scoring and linear vector scan behavior in its script-score query reference.

Put collection-wide hard constraints—rights, publication state, date bounds, or mandatory AND filters—inside both scoring clauses. Keep ordinary navigational facet selections out of the base query so aggregations can still calculate alternatives.

4. Add disjunctive facets

Elasticsearch calculates the scored query first, aggregations second, and post_filter last. Apply every selected facet to returned hits in the post-filter. For each facet aggregation, apply all selected facets except that facet itself. This is AND across facets and, in a multi-select interface, normally OR within one facet.

{
  "aggs": {
    "department": {
      "filter": {
        "bool": {
          "filter": [
            { "term": { "classification.keyword": "Paintings" } },
            { "term": { "isHighlight": true } }
          ]
        }
      },
      "aggs": {
        "values": {
          "terms": {
            "field": "department.keyword",
            "size": 50,
            "show_term_doc_count_error": true
          }
        }
      }
    }
  },
  "post_filter": {
    "bool": {
      "filter": [
        { "term": { "department.keyword": "European Paintings" } },
        { "term": { "classification.keyword": "Paintings" } },
        { "term": { "isHighlight": true } }
      ]
    }
  }
}

Repeat that filtered aggregation for every facet. Preserve selected values with an explicit filter bucket when they may fall outside the top terms. In a multi-shard index, inspect doc_count_error_upper_bound; script score gives a complete query match universe, but a distributed terms aggregation can still report bounded bucket-count error. The lab index uses one shard, so its local terms counts report an error bound of zero. See Elasticsearch’s post-filter guide and terms aggregation count-error guidance.

5. Benchmark before production

Vector functions in script score linearly scan the documents admitted by each clause. Disjunctive facets deliberately keep navigational filters out of the scored query, so they can cost more than a query that pushes every selected facet into the vector filter. Measure empty, broad, and narrow queries against the full collection, with production shard counts and concurrent traffic.

Start with similarity-threshold and relevance evaluation, then measure latency. If your institution prototypes kNN separately, compare it only after the exhaustive script-score baseline is stable: its bounded top-k window is not a substitute for complete collection-navigation counts.

Keyword search is lexical search: it ranks documents by matching the words in your query against indexed text fields. Elasticsearch scores those matches with BM25, a relevance formula that rewards query words appearing in a document while accounting for field length and term frequency.

Multi-word queries use a minimum-match rule: two terms must both match, three to five terms may miss one term, and longer queries must match 75% of terms. Adjacent phrase matches still get a smaller boost, so phrases like pink textile are treated as phrase intent without overwhelming the term match.

Optional fuzzy keyword matching is conservative: one-to-five-character terms get no edits, six-to-ten get one edit, and longer terms get two edits. That keeps short-name searches like monet from drifting into broad typo expansions.

Queries that look like accession numbers, such as 1979.486, are boosted heavily for exact accession matches and can also match by prefix.

Keyword Fields And Weights

The list below is sorted by effective boost, not by source-code declaration order.

The ^ number is a boost. Higher numbers make a match in that field count more. For example, a title match matters much more than a credit-line match.

Each metadata field searches both a stemmed English-analyzed version, such as title.en, and a lower-weight raw text version, such as title.

Fuzzy keyword matching is on by default for minor spelling mistakes. It uses a conservative prefix and edit-distance window so a query like qumrat can recover a close spelling such as qumran without drifting into broader neighbors like quart. Turn it off when you want stricter keyword comparisons.

Generated-field boosts are intentionally modest. Generated visual descriptions help recall visual subjects that are missing from museum metadata, but concise museum fields should still anchor keyword ranking.

Semantic search turns your query into an embedding and compares it with stored artwork embeddings. Instead of requiring the same words to appear in the object record, it looks for objects that are close to the meaning or visual idea of the query.

This is useful for descriptive searches like person looking into a mirror, scene searches like sparsely furnished interior, or subject searches where catalog metadata may not use the words you typed.

The semantic side depends on the selected embedding strategy, which controls whether the search uses generated-description vectors or image-only vectors. Min similarity controls how strict the semantic match must be before a result is shown.

Semantic variants always use script score, evaluating the similarity threshold across the complete eligible vector-bearing query scope. Their variants can still compare embeddings, thresholds, filters, generated evidence, and other settings within that exhaustive execution model.

Embeddings can also support cross-language searches because the query and indexed artwork evidence are compared by meaning rather than by shared words. That is why queries such as floresor 猫 can retrieve visually relevant objects even when the catalog text is in English.

Hybrid search combines keyword and semantic retrieval. It is useful when you want literal catalog evidence to anchor the ranking while still allowing semantic search to find visually or conceptually relevant objects that use different words.

Hybrid variants always use a plain script-score query that combines a saturated BM25 keyword clause with a thresholded cosine-similarity clause. It is more expensive than approximate retrieval, but it gives Elasticsearch a complete hybrid match set for collection navigation.

How script score differs from other fusion approaches

The lab implements only script score, but Linear and RRF are useful reference points when evaluating another architecture. Script score combines fixed transformed scores across the complete query scope. Linear generally combines scores normalized relative to bounded keyword and kNN windows; RRF combines rank positions instead of raw scores. A 50 / 50Linear sum therefore is not equivalent to this project's fixed 50 / 50 script-score formula.

Elasticsearch describes Linear as a weighted sum of normalized child-retriever scores and RRF as a sum of reciprocal-rank contributions. See the Linear retriever reference and RRF retriever reference. For the broader execution tradeoff and the HNSW and DiskBBQ controls, see Elastic's kNN search documentation.

Approximate kNN contributes a bounded top-k semantic result set rather than every object above a semantic threshold. For HNSW-backed fields, num_candidates sets how many nearest neighbors each shard gathers before the top k is chosen. Raising it improves recall, which can change which documents make the top k and, when the similarity cutoff leaves fewer than k, find more that qualify, but the top k remains the limit. k cannot exceed num_candidates, which is capped at 10000. DiskBBQ instead exposes visit_percentage. kNN's own similarity parameter can only remove documents from that top k. Aggregations therefore count at most k semantic matches (globally for a top-level knn search, per shard for a knn query), not every collection object that might satisfy a semantic threshold. Disjunctive hybrid retrieval adds every keyword match: a query beside top-level knn, or knn and keyword queries as bool should clauses with minimum_should_match: 1. Combining them under bool mustcounts only their intersection. With RRF, aggregations likewise cover every child retriever's matches; rank_window_size limits ranking and paging, not aggregations. A separate keyword aggregation cannot repair the gap because it cannot count semantic-only matches. Script score evaluates the semantic threshold in the query itself, so aggregations can count keyword matches plus semantic matches in one query context.

The primary variant's hits and shared filter counts therefore come from the same script-score or keyword request. There is no secondary aggregation query, top-k window count, or cross-variant bucket merge.

The tradeoff is that script score scans matching vector-bearing documents instead of using the approximate nearest-neighbor index. This is reasonable for the default 768-dimensional embedding strategy, but higher dimensional strategies should be treated as benchmark targets before relying on them for broad script-score comparisons.

Script score compared with approximate kNN

Neither execution strategy is universally better. Script score buys a complete, threshold-defined search universe for navigation; kNN buys efficient top-neighbor retrieval. Choose against the product requirement, then benchmark with the institution's own index, filters, traffic, and hardware.

Use hybrid for searches like pink textile, woman and child, or ship in a storm, where catalog metadata and visible description can both carry useful evidence.

An embedding is a numeric representation of text, an image, or both. Similar meanings or visual patterns end up near each other in vector space, which lets semantic search retrieve objects even when the query and catalog record do not share exact words.

Semantic and hybrid variants search one stored vector field at a time in Elasticsearch.

Google's published dimension results show why 768 is a practical default: its MTEB score is 67.99, only 0.18 points below the best listed score, while using one-quarter as many raw vector values as 3072 dimensions. See the official Gemini Embeddings documentation.

The percentages compare raw vector length, not total Elasticsearch index storage, which also includes document and mapping overhead; an approximate-search design would add vector-index overhead as well. MTEB is a broad text-embedding benchmark rather than a museum-search relevance test, and Google's dimension table does not report a separate 3072 score; institutions should still evaluate dimensions with their own collections and queries.

Why Embed Generated Descriptions?

A generated description inserts an inspectable textual representation between an artwork image and its retrieval vector. That intermediate layer changes what can be understood, governed, reused, and tested even when the final text embedding is still a learned and imperfect representation.

The Met Search Lab uses AI-generated text to make parts of the collection searchable in ways that catalog metadata alone may not support. For selected Open Access objects, the pipeline generates structured descriptions from object images and metadata, then stores that generated text alongside the original Met object record.

The generated visual descriptions in this lab are produced by vision-language models. The current prompts use Agentic Vision in Gemini 3 Flash, a capability that combines visual reasoning with code execution so the model can zoom, inspect, and ground answers in visual evidence.

Generated text is treated as search evidence, not museum catalog data. It feeds the keyword query alongside catalog fields and serves as source text for selected embedding strategies. The controls let you compare variants with generated keyword content on or off, and generated content panels in results help show what the model contributed.

The public reference prompt lives in a versioned Markdown file. Historical Agentic Vision artifacts remain in repository history and the frozen evaluation record, but they are not indexed or exposed by the live lab and their exact production prompts are not distributed in the current tree.

The executable public reference prompt below is loaded from its versioned generator Markdown file. It is a simplified, reproducible experiment inspired by The Met's production principles, not the exact production prompt.

These controls are the main levers for changing retrieval behavior. Use them to decide which evidence is searched, how strict the result set should be, and how keyword and semantic signals are combined.

Result cards link to an object detail page that keeps the search context in the URL, including the query, variant, rank, and score. The page is meant for inspecting why an object appeared in a result set, not only for viewing the artwork record.

Similar Objects is useful for comparing embedding strategies directly. Text-description embeddings often group works by depicted subjects and visual language, while image-only embeddings can emphasize composition, medium, or visual appearance. Results are filtered to objects with images and should be read as model neighborhoods rather than curated related works. Object-page similarity scans the eligible vector-bearing documents with script_score, using the same exhaustive execution approach as semantic collection search.

These queries are useful for checking whether a search change improves the right behavior without causing regressions elsewhere.

These examples open the search page with multiple variants already configured so one search question can be compared across embedding strategies.

The Met Search Lab is built using AI coding agents. Their contributions include application code, tests, evaluation tooling, technical documentation, and this technical guide. Human collaborators set the goals, provide institutional and production context, review changes, validate behavior, and decide what is published.

The guide draws on The Met's production code to explain architectural decisions and implementation details that others can reproduce and adapt without access to the production application source. AI coding agents may continue to help update the lab and guide as the project and The Met's practices evolve.

Treat this as a working technical reference, not a universal recipe. Models, prompts, thresholds, infrastructure, latency observations, and governance practices must be evaluated against each institution's collection, policies, editorial capacity, and users.

The data in this app comes from The Metropolitan Museum of Art Collection API and includes only Open Access images and data.