About The Met Search Lab

LLMs helped build The Met Search Lab, including this guide, which draws on The Met's production code to explain key architectural decisions and implementation details for others to compare and adapt. LLMs will also be used periodically to help update the project as The Met's practices evolve. Treat it as technical guidance, not a universal recipe: the models, thresholds, infrastructure, and latency observations reflect The Met's needs and should be tested against each institution's collection and users.

Each search variant can use keyword search, semantic search, or a hybrid of both.

From collection object to search result

Elasticsearch stores the Met object record, generated text such as visual descriptions, and embeddings derived from generated visual descriptions or images. The pipeline first prepares that evidence for the index, then combines it at search time to produce ranked results and navigable filter counts.

The live lab keeps The Met's script-score execution and 75 / 25 hybrid weights, but its selectable evidence is intentionally narrower. A Lab default badge marks the active Medium Visual Description recipe; changing one of its settings removes the badge.

1. Prepare complementary evidence

Keep concise catalog metadata as the primary literal signal. Generate an objective visual description for each artwork image, then embed that description as the semantic signal. The production pipeline currently uses Agentic Vision v4 visual descriptions and a 768-dimensional Gemini text embedding. This public lab instead defaults to the Medium field from its reproducible visual-description generator. V4 remains documented as the historical production-aligned baseline used to calibrate the completed study, but it is not an active search option or field in the current lab index.

Use the same embedding model, dimensions, normalization, and task conventions for indexed documents and live queries. Store the model and source-text version with each vector so a reindex cannot silently mix incompatible embeddings.

2. Map the vector field

Set index: false because script score reads stored vector values directly and does not need an HNSW graph. The lab indexer also excludes embeddings from _source to avoid retaining and returning large arrays there; scripts can still access the vector values.

{
  "mappings": {
    "_source": {
      "excludes": ["embeddings.*"]
    },
    "properties": {
      "embeddings": {
        "properties": {
          "agentic_vision": {
            "type": "dense_vector",
            "dims": 768,
            "index": false
          }
        }
      }
    }
  }
}

3. Build a complete match universe

The lexical clause saturates BM25 with score / (score + 8). The vector clause rejects documents below the similarity threshold, rescales the remaining cosine range to 0–1, and applies its weight. A bool.should union means a document can enter through either literal or semantic evidence.

{
  "query": {
    "bool": {
      "should": [
        {
          "script_score": {
            "query": { "multi_match": { "query": "<query>", "fields": ["title^4", "artist^3", "medium^1.5"] } },
            "script": {
              "source": "(_score / (_score + params.k)) * params.weight",
              "params": { "k": 8, "weight": 0.75 }
            }
          }
        },
        {
          "script_score": {
            "query": { "bool": { "filter": [{ "exists": { "field": "embeddings.agentic_vision" } }] } },
            "script": {
              "source": "double c = cosineSimilarity(params.vector, 'embeddings.agentic_vision'); if (c < params.threshold) return 0; return ((c - params.threshold) / (1 - params.threshold)) * params.weight;",
              "params": { "vector": "<query vector>", "threshold": 0.65, "weight": 0.25 }
            },
            "min_score": 0.000001
          }
        }
      ],
      "minimum_should_match": 1
    }
  }
}

Elasticsearch documents the per-match scoring and linear vector scan behavior in its script-score query reference.

Put collection-wide hard constraints—rights, publication state, date bounds, or mandatory AND filters—inside both scoring clauses. Keep ordinary navigational facet selections out of the base query so aggregations can still calculate alternatives.

4. Add disjunctive facets

Elasticsearch calculates the scored query first, aggregations second, and post_filter last. Apply every selected facet to returned hits in the post-filter. For each facet aggregation, apply all selected facets except that facet itself. This is AND across facets and, in a multi-select interface, normally OR within one facet.

{
  "aggs": {
    "department": {
      "filter": {
        "bool": {
          "filter": [
            { "term": { "classification.keyword": "Paintings" } },
            { "term": { "isHighlight": true } }
          ]
        }
      },
      "aggs": {
        "values": {
          "terms": {
            "field": "department.keyword",
            "size": 50,
            "show_term_doc_count_error": true
          }
        }
      }
    }
  },
  "post_filter": {
    "bool": {
      "filter": [
        { "term": { "department.keyword": "European Paintings" } },
        { "term": { "classification.keyword": "Paintings" } },
        { "term": { "isHighlight": true } }
      ]
    }
  }
}

Repeat that filtered aggregation for every facet. Preserve selected values with an explicit filter bucket when they may fall outside the top terms. In a multi-shard index, inspect doc_count_error_upper_bound; script score gives a complete query match universe, but a distributed terms aggregation can still report bounded bucket-count error. The lab index uses one shard, so its local terms counts report an error bound of zero. See Elasticsearch’s post-filter guide and terms aggregation count-error guidance.

5. Benchmark before production

Vector functions in script score linearly scan the documents admitted by each clause. Disjunctive facets deliberately keep navigational filters out of the scored query, so they can cost more than a query that pushes every selected facet into the vector filter. Measure empty, broad, and narrow queries against the full collection, with production shard counts and concurrent traffic.

Start with similarity-threshold and relevance evaluation, then measure latency. If your institution prototypes kNN separately, compare it only after the exact script-score baseline is stable: its bounded candidate window is not a substitute for complete collection-navigation counts.

Keyword search is lexical search: it ranks documents by matching the words in your query against indexed text fields. Elasticsearch scores those matches with BM25, a relevance formula that rewards query words appearing in a document while accounting for field length and term frequency.

Multi-word queries use a minimum-match rule: two terms must both match, three to five terms may miss one term, and longer queries must match 75% of terms. Adjacent phrase matches still get a smaller boost, so phrases like pink textile are treated as phrase intent without overwhelming the term match.

Optional fuzzy keyword matching is conservative: one-to-five-character terms get no edits, six-to-ten get one edit, and longer terms get two edits. That keeps short-name searches like monet from drifting into broad typo expansions.

Queries that look like accession numbers, such as 1979.486, are boosted heavily for exact accession matches and can also match by prefix.

Keyword Fields And Weights

The list below is sorted by effective boost, not by source-code declaration order.

The ^ number is a boost. Higher numbers make a match in that field count more. For example, a title match matters much more than a credit-line match.

Each metadata field searches both a stemmed English-analyzed version, such as title.en, and a lower-weight raw text version, such as title.

Fuzzy keyword matching is on by default for minor spelling mistakes. It uses a conservative prefix and edit-distance window so a query like qumrat can recover a close spelling such as qumran without drifting into broader neighbors like quart. Turn it off when you want stricter keyword comparisons.

Generated-field boosts are intentionally modest. Generated visual descriptions help recall visual subjects that are missing from museum metadata, but concise museum fields should still anchor keyword ranking.

Semantic search turns your query into an embedding and compares it with stored artwork embeddings. Instead of requiring the same words to appear in the object record, it looks for objects that are close to the meaning or visual idea of the query.

This is useful for descriptive searches like person looking into a mirror, mood searches like quiet interior, or subject searches where catalog metadata may not use the words you typed.

The semantic side depends on the selected embedding strategy, which controls whether the search uses generated text vectors, image vectors, or image-plus-text vectors. Min similarity controls how strict the semantic match must be before a result is shown.

Semantic variants always use script score, evaluating the similarity threshold across the complete eligible vector-bearing query scope. Their variants can still compare embeddings, thresholds, filters, generated evidence, and other settings within that exact execution model.

Embeddings can also support cross-language searches because the query and indexed artwork evidence are compared by meaning rather than by shared words. That is why queries such as floresor 猫 can retrieve visually relevant objects even when the catalog text is in English.

Hybrid search combines keyword and semantic retrieval. It is useful when you want literal catalog evidence to anchor the ranking while still allowing semantic search to find visually or conceptually relevant objects that use different words.

Hybrid variants always use a plain script-score query that combines a saturated BM25 keyword clause with a thresholded cosine-similarity clause. It is more expensive than approximate retrieval, but it gives Elasticsearch a complete hybrid match set for collection navigation.

How script score differs from other fusion approaches

The lab implements only script score, but Linear and RRF are useful reference points when evaluating another architecture. Script score combines fixed transformed scores across the complete query scope. Linear generally combines scores normalized relative to bounded keyword and kNN windows; RRF combines rank positions instead of raw scores. A 75 / 25Linear sum therefore is not equivalent to this project's fixed 75 / 25 script-score formula.

Elasticsearch describes Linear as a weighted sum of normalized child-retriever scores and RRF as a sum of reciprocal-rank contributions. See the Linear retriever reference and RRF retriever reference. For the broader execution tradeoff, see Elastic's guide to exact and approximate kNN.

kNN-backed retrievers work from a bounded candidate set, and Elasticsearch caps num_candidates at 10000 per shard. That means Linear and RRF cannot produce exact facets for the full embedded collection once the matching set is larger than the candidate window. A separate keyword aggregation cannot repair that gap because it cannot count semantic-only matches. Script score evaluates the semantic threshold in the query itself, so aggregations can count keyword matches plus semantic matches in one query context.

The primary variant's hits and shared filter counts therefore come from the same script-score or keyword request. There is no secondary aggregation query, candidate-window count, or cross-variant bucket merge.

The tradeoff is that script score scans matching vector-bearing documents instead of using the approximate nearest-neighbor index. This is reasonable for the default 768-dimensional embedding strategy, but higher dimensional strategies should be treated as benchmark targets before relying on them for broad script-score comparisons.

Script score compared with kNN

Neither execution strategy is universally better. Script score buys a complete, threshold-defined search universe for navigation; kNN buys efficient top-neighbor retrieval. Choose against the product requirement, then benchmark with the institution's own index, filters, traffic, and hardware.

Use hybrid for searches like pink textile, woman and child, or ship in a storm, where catalog metadata and visible description can both carry useful evidence.

An embedding is a numeric representation of text, an image, or both. Similar meanings or visual patterns end up near each other in vector space, which lets semantic search retrieve objects even when the query and catalog record do not share exact words.

Semantic and hybrid variants search one stored vector field at a time in Elasticsearch.

Google's published dimension results show why 768 is a practical default: its MTEB score is 67.99, only 0.18 points below the best listed score, while using one-quarter as many raw vector values as 3072 dimensions. See the official Gemini Embeddings documentation.

The percentages compare raw vector length, not total Elasticsearch index storage, which also includes document and mapping overhead; an approximate-search design would add HNSW overhead as well. MTEB is a broad text-embedding benchmark rather than a museum-search relevance test, and Google's dimension table does not report a separate 3072 score; institutions should still evaluate dimensions with their own collections and queries.

The Met Search Lab uses AI-generated text to make parts of the collection searchable in ways that catalog metadata alone may not support. For selected Open Access objects, the pipeline generates structured descriptions from object images and metadata, then stores that generated text alongside the original Met object record.

The generated visual descriptions in this lab are produced by vision-language models. The current prompts use Agentic Vision in Gemini 3 Flash, a capability that combines visual reasoning with code execution so the model can zoom, inspect, and ground answers in visual evidence.

Generated text is treated as search evidence, not museum catalog data. It feeds the keyword query alongside catalog fields and serves as source text for selected embedding strategies. The controls let you compare variants with generated keyword content on or off, and generated content panels in results help show what the model contributed.

The public reference prompt lives in a versioned Markdown file. Historical Agentic Vision artifacts remain in repository history and the frozen evaluation record, but they are not indexed or exposed by the live lab and their exact production prompts are not distributed in the current tree.

The executable public reference prompt below is loaded from its versioned generator Markdown file. It is a simplified, reproducible experiment inspired by The Met's production principles, not the exact production prompt.

These controls are the main levers for changing retrieval behavior. Use them to decide which evidence is searched, how strict the result set should be, and how keyword and semantic signals are combined.

Result cards link to an object detail page that keeps the search context in the URL, including the query, variant, rank, and score. The page is meant for inspecting why an object appeared in a result set, not only for viewing the artwork record.

Similar Objects is useful for comparing embedding strategies directly. Text-description embeddings often group works by depicted subjects and visual language, while image-only and image-plus-text embeddings can emphasize composition, medium, or visual appearance. Results are filtered to objects with images and should be read as model neighborhoods rather than curated related works. Object-page similarity scans the eligible vector-bearing documents with script_score, using the same exact execution approach as semantic collection search.

These queries are useful for checking whether a search change improves the right behavior without causing regressions elsewhere.

These examples open the search page with multiple variants already configured so one search question can be compared across embedding strategies.

The data in this app comes from The Metropolitan Museum of Art Collection API and includes only Open Access images and data.