ComfyUI-Lora-Manager/docs/agent_skills.md

Agent Skills System

The LoRA Manager agent skills system enables LLM-powered metadata enrichment and other AI-driven tasks. Users configure their own LLM provider (BYOK), and skills are executed through right-click context menu actions.

Architecture

┌──────────────────────────────────────────────┐
│              LoRA Manager Backend             │
│                                               │
│  ┌──────────────┐    ┌────────────────┐       │
│  │ LLMService    │───▶│ LLM Provider   │       │
│  │ (BYOK config, │◀───│ (OpenAI/Ollama │       │
│  │  API calls)   │    │ /custom)       │       │
│  └───────┬───────┘    └────────────────┘       │
│          │                                     │
│  ┌───────▼───────────────────────┐             │
│  │     AgentService              │             │
│  │  (orchestration: validate     │             │
│  │   → LLM call → post-process   │             │
│  │   → WebSocket broadcast)      │             │
│  └───────┬───────────────────────┘             │
│          │                                     │
│  ┌───────▼───────────────────────┐             │
│  │     SkillRegistry             │             │
│  │  ┌─────────────────────────┐  │             │
│  │  │ enrich_hf_metadata:     │  │             │
│  │  │  - skill.yaml           │  │             │
│  │  │  - prompt.md            │  │             │
│  │  │  - handler.py           │  │             │
│  │  └─────────────────────────┘  │             │
│  └───────────────────────────────┘             │
└──────────────────────────────────────────────┘

Key Design Principle

Skills define what to do (prompt + post-processing). The AgentService handles how (LLM calls, validation, progress).

Skills never call the LLM directly. This keeps BYOK configuration centralized and provider-agnostic.

BYOK Configuration

Users configure their LLM provider in Settings → AI Provider:

Setting	Description	Example
llm_provider	Provider type	openai, ollama, or custom
llm_api_key	API key (not needed for local Ollama)	sk-...
llm_api_base	Custom API base URL (empty = provider default)	https://api.openai.com/v1
llm_model	Model name	gpt-4o-mini

Environment variable overrides: LLM_API_KEY, LLM_MODEL, LLM_API_BASE, LLM_PROVIDER.

Supported Providers

• OpenAI: Uses https://api.openai.com/v1 by default
• Ollama (local): Uses http://localhost:11434/v1, no API key required
• Custom: Any OpenAI-compatible endpoint (vLLM, LM Studio, etc.) — set llm_api_base explicitly

Available Skills

enrich_hf_metadata

Enriches models linked to an external model site with metadata extracted by an LLM from the site's model card (README).

Entry point: Right-click context menu → "Enrich Metadata with AI"

Supported model sources:

Platform	Link	AI enrichment	Direct download
Hugging Face	yes	yes	yes
ModelScope	yes	yes	no
TensorArt	yes	no (see below)	no

TensorArt is link-only: tensor.art sits behind a Cloudflare managed challenge and its internal API requires session authorization, so the backend cannot read its model pages. Linking still stores the canonical page URL and the "View on TensorArt" link works.

What it does:

1. Reads the model's .metadata.json to get the source (source_platform + source_url, or the legacy hf_url)
2. Fetches the model card through the provider in py/services/model_sources/
3. Sends the README + local metadata to the LLM for structured extraction
4. Writes extracted fields to .metadata.json:
   • base_model — only if current value is empty
   • trainedWords — trigger words (LoRA only, if none exist)
   • modelDescription — concise summary (if none exists)
   • tags — merged with existing tags, deduplicated
   • metadata_source — audit trail: agent:enrich_hf_metadata
   • llm_enriched_at — ISO timestamp
5. Downloads and optimizes preview image (if LLM found one in the README)
6. Updates the scanner cache
7. Broadcasts WebSocket progress events

Models with no source, an unknown source, or a source without model-card access (TensorArt) are skipped with an explicit reason and counted in the run summary.

Model types: LoRA, Checkpoint, Embedding

Adding a New Skill

1. Create the skill directory

py/services/agent/skills/<skill_name>/
├── skill.yaml      # Skill metadata and schemas
├── prompt.md       # LLM prompt template
└── handler.py      # Pre-processing and post-processing

2. Write skill.yaml

name: my_skill
title: "My Skill"
description: "What this skill does"
llm_required: true
model_type_filter: ["lora"]  # or null for all types
input_schema:
  type: object
  properties:
    model_paths:
      type: array
      items:
        type: string
  required:
    - model_paths
output_schema:
  type: object
  properties:
    # ... JSON schema for LLM output
permissions:
  write_metadata: true
  write_previews: false
  network_domains:
    - "example.com"

3. Write prompt.md

Use {{variable}} placeholders that will be replaced with data from the prepare function:

You are an expert assistant...

Model URL: {{source_url}}
README content:
{{readme_content}}

Current metadata:
{{current_metadata}}

4. Write handler.py

async def prepare(model_path: str, input_data: dict) -> dict:
    """Gather context for the LLM prompt. Returns variables for template rendering."""
    return {
        "model_path": model_path,
        # ... other variables used in prompt.md
    }

async def post_process(context) -> dict:
    """Apply the LLM-extracted data to the model."""
    llm_response = context.llm_response
    # ... write metadata, download previews, update cache
    return {
        "success": True,
        "updated_fields": ["base_model", "tags"],
        "errors": [],
    }

Important: Use absolute imports (from py.utils.metadata_manager import MetadataManager) because skills are loaded via importlib.util.spec_from_file_location, which doesn't support relative imports.

5. Test

The skill is automatically discovered by SkillRegistry on startup. Test with:

pytest tests/services/test_agent_service.py

API Endpoints

Method	Path	Description
GET	/api/lm/agent/skills	List available skills
POST	/api/lm/agent/execute/{skill_name}	Execute a skill (body: {"model_paths": [...]})
POST	/api/lm/agent/cancel	Cancel running skill (stub)

WebSocket Events

Type	When	Key fields
agent_progress	Skill started/processing	skill, status, total, processed, success, current_path
agent_progress	Skill completed	skill, status, updated_models, errors, summary
agent_progress	Skill error	skill, status, error

Security Model

Skills declare permissions in skill.yaml:

• write_metadata — can write .metadata.json files
• write_previews — can download/replace preview images
• network_domains — allowed domains for HTTP requests

These are declarative constraints checked by AgentService. They are defense-in-depth, not a sandbox — the Python process can technically do anything, but the contract is clear and auditable.

File Locations

Component	Path
LLMService	py/services/llm_service.py
AgentService	py/services/agent/agent_service.py
SkillRegistry	py/services/agent/skill_registry.py
SkillDefinition	py/services/agent/skill_definition.py
Skills directory	py/services/agent/skills/
Route handlers	py/routes/handlers/agent_handlers.py
Frontend manager	static/js/managers/AgentManager.js
Settings UI	templates/components/modals/settings_modal.html
Context menu	templates/components/context_menu.html