fix(agent): persist the LLM confidence through metadata writes

The post-processor stored the LLM's confidence as `_llm_confidence`, but
that value could never be read back: `BaseModelMetadata.from_dict()`
deliberately excludes underscore-prefixed keys from `_unknown_fields` and
`to_dict()` strips private fields, so it was erased by the next metadata
write and was invisible to `read_metadata()`.  The enrichment evaluation
harness reads this field to score runs, so confidence was always scored
as blank.

Store it as `llm_confidence`, which round-trips as an ordinary unknown
field — the same mechanism `llm_enriched_at` already relies on.  Nothing
else consumed the old name, and the harness still accepts it so sidecars
written by earlier versions keep evaluating.

Covered by a metadata load/save round-trip regression test plus
assertions that the post-processor writes the persisted key and no longer
writes the private one.
This commit is contained in:
Will Miao
2026-09-14 20:42:14 +08:00
parent 4064ea7d3a
commit 51de85a6ca
4 changed files with 92 additions and 3 deletions
+5 -2
View File
@@ -273,10 +273,13 @@ class PostProcessor:
updates["metadata_source"] = "agent:enrich_hf_metadata"
updates["llm_enriched_at"] = datetime.now(timezone.utc).isoformat()
# Store LLM confidence in metadata so it's accessible for evaluation
# LLM confidence, stored for the enrichment evaluation harness. The key
# must NOT start with an underscore: `BaseModelMetadata.from_dict()`
# deliberately drops underscore-prefixed keys so they never round-trip,
# which silently erased this field on the next metadata write.
raw_confidence = (llm_output.get("confidence") or "").strip()
if raw_confidence:
updates["_llm_confidence"] = raw_confidence
updates["llm_confidence"] = raw_confidence
# Fallback: use the trigger words the site records for this exact file,
# then the README's YAML `instance_prompt`, when the LLM returned none.