fix(agent): persist the LLM confidence through metadata writes

The post-processor stored the LLM's confidence as `_llm_confidence`, but
that value could never be read back: `BaseModelMetadata.from_dict()`
deliberately excludes underscore-prefixed keys from `_unknown_fields` and
`to_dict()` strips private fields, so it was erased by the next metadata
write and was invisible to `read_metadata()`.  The enrichment evaluation
harness reads this field to score runs, so confidence was always scored
as blank.

Store it as `llm_confidence`, which round-trips as an ordinary unknown
field — the same mechanism `llm_enriched_at` already relies on.  Nothing
else consumed the old name, and the harness still accepts it so sidecars
written by earlier versions keep evaluating.

Covered by a metadata load/save round-trip regression test plus
assertions that the post-processor writes the persisted key and no longer
writes the private one.
This commit is contained in:
Will Miao
2026-09-14 20:42:14 +08:00
parent 4064ea7d3a
commit 51de85a6ca
4 changed files with 92 additions and 3 deletions
@@ -108,7 +108,12 @@ def evaluate_model(
model_description: str = metadata.get("modelDescription") or ""
base_model: str = metadata.get("base_model") or ""
preview_url: str = metadata.get("preview_url") or ""
confidence: str = metadata.get("_llm_confidence") or ""
# `_llm_confidence` is the legacy key: underscore-prefixed metadata keys are
# deliberately not persisted through `BaseModelMetadata`, so older sidecars
# may still carry it while current ones use `llm_confidence`.
confidence: str = (
metadata.get("llm_confidence") or metadata.get("_llm_confidence") or ""
)
# --- base_model ---
base_model_valid = base_model in SUPPORTED_BASE_MODELS