Search Evaluation Guide

Offer evaluation after the search pipeline is configured and working:

“Would you like to evaluate the search quality? I can run test queries, measure relevance metrics, and suggest improvements.”

Evaluation Workflow

Step 1: Generate Test Queries

Ask the user to provide test queries. Assign a capability to each query based on its form:

Step 2: Run Queries

Run all test queries through the search pipeline and collect top-k results for each.

Step 3: Judge Relevance

For each query, review the returned documents and assign a relevance grade to each query-document pair. Grade every document in the top-k results — do not skip any.

Grading scale:

Judgment prompt — for each query-document pair, evaluate:

1. Intent match: What is the user trying to find with this query? Does this document satisfy that intent?
2. Content relevance: How well does the document’s content relate to the query?
3. Would a real user click this? If yes, grade >= 2. If maybe, grade 1. If no, grade 0.

Step 4: Compute Metrics

Three metrics are computed per query per method, all at cutoff k:

Target Thresholds

Step 5: Diagnose Issues

Apply diagnostic rules comparing across methods:

Rule 1: All methods fail (nDCG < 0.3 for every method)

• Severity: HIGH
• Meaning: No retrieval strategy can find relevant documents for this query
• Fix: Check field mappings, analyzers, or upgrade embedding model

Rule 2: Pairwise method gaps

• Severity: MEDIUM
• Triggers when: A vector method fails (nDCG < 0.3) while a lexical method succeeds (nDCG > 0.5), or vice versa
• Fix: Upgrade embedding model, or add proper text analyzers/boosting

Rule 3: Hybrid worse than single signals

• Severity: MEDIUM/LOW
• Triggers when: A hybrid method’s nDCG is > 0.15 below the best non-hybrid method
• Fix: Adjust hybrid weights, or use query-type-aware routing

Rule 4: Irrelevant docs in top-2

• Severity: MEDIUM
• Triggers when: An irrelevant document (grade 0) appears in positions 1-2 and nDCG < 0.8
• Fix: Reduce field boosts, restructure query, or upgrade model

Rule 5: Missed relevant documents

• Severity: LOW
• Triggers when: High-relevance documents (grade >= 2) don’t appear in any method’s top-k
• Fix: Embed more fields, use a higher-capacity model

Finding Tags

Completion Criteria

The evaluation passes if any of:

• Mean nDCG@k across all methods > 0.7
• All findings are LOW severity only
• No HIGH severity findings and setup matches the use case

After Evaluation

Present results, then offer:

1. Restart with improvements — Apply recommended fixes and rebuild the search setup
2. Deploy as-is — Current configuration is acceptable
3. Done for now — Keep experimenting

If HIGH severity findings exist, recommend option 1 and explain the specific fix.