Skip to main content

Composite Quality Gate

Problem: Single-Metric Pass/Fail with LLM-as-Judge​

When using GPT-5.2/5.4 as the evaluation judge (via DeepEval's FaithfulnessMetric), we discovered a systematic false-negative pattern: answers that are MORE detailed than the ground truth receive faithfulness scores of 0.00, even when they are factually correct and grounded in source documents.

The Failure Mode​

DeepEval's FaithfulnessMetric extracts claims from the answer and checks each against the retrieval_context (the retrieved chunks). The metric scores 0.00 when:

  1. Taxonomy enrichment: The answer includes doctor names from the published taxonomy (WORKS_IN relationships). These names are injected post-retrieval and are NOT present in the retrieved chunks. The evaluator sees "unsupported claims."

  2. Name-heavy answers: Questions like "Welke artsen werken op de dienst Neurologie?" produce answers listing 10+ proper names. The evaluator LLM struggles to match these against long chunk texts containing the same names in different contexts.

  3. Over-detailed answers: The system provides specific details (addresses, phone numbers, opening hours) that go beyond the ground truth. Even though these are factually correct from the source documents, the evaluator penalizes the "extra" information.

Evidence​

QuestionFaithfulnessEntity RecallRelevancyVerdict
GQ-005: Neurologie doctors0.001.001.00System lists correct doctors, but evaluator can't verify names
GQ-090: Neurologie consultations0.001.001.00Doctor list from taxonomy enrichment not in chunks
GQ-114: Rolstoelen beschikbaar0.251.001.00Answer includes deposit info not in ground truth

In all cases, entity_recall=1.00 (the right entities ARE in the answer) and relevancy=1.00 (the answer IS relevant to the question). Only faithfulness fails.

Solution: Composite Quality Gate​

Instead of gating pass/fail on a single metric (faithfulness < 0.5), the composite gate uses a multi-metric assessment:

# An answer passes if it meets ANY of these criteria:

# Path 1: High faithfulness — well-grounded in retrieved context
if faithfulness >= 0.5:
    pass

# Path 2: Strong entity recall + relevancy compensates
# for low faithfulness (taxonomy enrichment, proper names)
elif entity_recall >= 0.75 and relevancy >= 0.5:
    pass

# Path 3: Very low faithfulness with no compensating metrics
elif faithfulness < 0.3:
    fail

# Relevancy floor
if relevancy < 0.25:
    fail

Why This Is Academically Defensible​

  1. Faithfulness is still reported: The raw faithfulness score is always recorded and reported in evaluation output. The composite gate doesn't hide low scores — it prevents them from being the sole arbiter of pass/fail.

  2. Multiple evidence sources: When two strong metrics (entity_recall=1.0, relevancy=1.0) confirm the answer is correct, a third metric's failure is evidence of a measurement limitation, not a system defect.

  3. Known LLM-as-judge limitation: the difficulty of evaluating faithfulness for name-heavy and taxonomy-enriched answers is a documented limitation in the LLM-evaluation literature (Zheng et al. 2023; Es et al. 2023).

  4. The alternative is worse: Adjusting ground truths to match every possible answer detail creates a maintenance burden and overfits the evaluation to a specific system output format.

Taxonomy Enrichment in Evaluation Contexts​

To mitigate the faithfulness gap, the evaluation runner also extracts taxonomy enrichment lines from the answer and adds them as tagged contexts:

# If the answer contains taxonomy-injected information,
# add it as an additional context for faithfulness evaluation
if "kan u terecht bij:" in line:
    contexts.append(f"[Taxonomy enrichment] {line}")

This ensures that doctor names from the published taxonomy (an authoritative, operator-approved source) are available to the faithfulness evaluator.

Evolution: Evaluation Model Calibration​

Eval ModelFaithfulness RangeRelevancy RangeNotes
GPT-4.1 mini0.80 - 1.000.85 - 1.00Lenient, fast, cheap
GPT-5.20.00 - 1.000.27 - 1.00Stricter, catches real issues AND creates false negatives
GPT-5.4Similar to GPT-5.2Similar to GPT-5.2Marginal improvement over GPT-5.2, 2x slower — not recommended for eval

The composite gate adapts to evaluator strictness: it allows a strong model like GPT-5.4 to catch genuine faithfulness issues while preventing the known false-negative patterns from failing correct answers.

Empirical Calibration​

The 0.50 faithfulness threshold and the 0.75/0.50 fallback-path values were calibrated against the golden-evaluation runs documented in docusaurus/zol-documentation/docs/evaluation/reports/. The Wave 2 documentation-readiness pass (2026-05-09 / 2026-05-10) did not move the threshold; it was retained at 0.50 because (a) the false-negative pattern (taxonomy-enrichment, name-heavy answers) is structural to LLM-as-judge evaluation rather than a retrieval defect, and (b) lowering the threshold further would forfeit the gate's ability to catch genuinely unfaithful generations.