基于阿里云 AgentLoop 的 Skill 评估与优化最佳实践
引言
Cloud Native
在 AI Agent 应用中,Skill(技能)是赋予 Agent 特定领域能力的关键资产。然而,一个 Skill 从编写到发布,如何确保它在真实场景中表现良好?如何量化它的质量?如何系统性地持续优化?
本文介绍一套基于阿里云 Agent 观测与优化平台 AgentLoop 的 Skill 评估与优化最佳实践,覆盖从 Skill 创建、可观测接入、离线评估、Bad Case 分析到迭代优化的完整闭环,帮助开发者以数据驱动的方式交付高质量 Skill。
整体流程概览
Cloud Native
┌─────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌──────────┐ ┌────────┐│ ① 创建 │───▶│ ② 可观测 │───▶│ ③ 离线 │───▶│ ④ Bad │───▶│ ⑤ 迭代 │───▶│ ⑥ 发布 ││ Skill │ │ 接入 │ │ 实验 │ │ Case优化 │ │ 验证 │ │ 上线 │└─────────┘ └──────────┘ └──────────┘ └──────────┘ └──────────┘ └────────┘ ▲ │ └──────────────────────────────────────────────┘ 评分持续提升,直到达标
这套流程的核心理念是:用可观测数据感知现状,用离线实验量化质量,用 Bad Case 驱动优化,用评分趋势验证效果。
第一步:创建 Skill 并接入可观测
Cloud Native
▍1.1 Skill 的创建与管理
在 AgentLoop 的 Agent 资产管理中,开发者可以创建和管理 Skill。Skill 本质上是一组结构化的指令和约束,用于指导 Agent 在特定领域(如 CMS 运维、代码审查、工单处理等)中执行任务。
一个典型的 Skill 包含以下要素:
Prompt 指令:定义 Agent 的操作流程、步骤和约束;
工具调用规范:指定 Agent 应调用的 CLI 命令或 API;
输出约束:规定输出的格式、禁止出现的内容等;
安全边界:明确哪些操作是被禁止的、哪些信息不应暴露。
Skill 创建后以草稿状态存在,此时尚未发布上线,开发者可以在本地或测试环境中先行验证。本文以云监控2.0全生命周期管理 Skill alibabacloud-cms-manage 为例,进行真实线上业务数据的 Skill 调优。
▍1.2 可观测接入
Skill 投入使用后,AgentLoop 的可观测能力会自动采集 Skill 的执行数据。通过 Skill 大盘,开发者可以实时掌握:
这些数据通过探针自动采集,以 Trace Span 的形式记录在链路追踪中——每个工具调用的 Span 里会附带 Skill 相关的属性标识,开发者无需额外埋点即可获得完整的执行视图。
最佳实践:在 Skill 发布前就完成可观测接入,确保从第一次线上调用开始就有数据沉淀,为后续的评估和优化提供基线。
第二步:构建评估数据集
Cloud Native
▍2.1 数据集的结构
评估数据集(Dataset)是离线实验的基础。每个测试用例(Case)由以下几部分构成:
# 一个典型的评估用例case: name: "查询 Prometheus 监控数据" prompt: "帮我查看当前工作空间下 Prometheus 的监控数据" # 期望的行为:Agent 应该执行的命令 expected_behaviors: - "arms-prometheus query" - "kubectl get pods" # 期望的输出:Agent 响应中应包含的内容 expected_outputs: - "监控数据查询成功" - "工作空间信息" # 禁止出现:Agent 响应中不应包含的内容 forbidden_outputs: - "AccessKey" - "SecretKey" - "内部 IP 地址"
▍2.2 数据集的设计原则
覆盖真实业务场景:数据集应来源于真实的业务需求,涵盖 Skill 要处理的核心操作场景。例如 CMS 运维场景下,应包括查询报警、查询 Prometheus、查询工作组列表等典型操作;
正反例兼顾:既要包含期望行为的正例,也要包含禁止行为的反例,确保评估的全面性;
可验证性:每个用例的期望结果应该是明确可验证的,便于自动化评估器进行判定。
最佳实践:初始数据集建议覆盖 10-20 个核心场景,随着 Skill 的迭代不断扩充。每个场景至少包含一个正例和一个边界用例。
本次Skill 调优实践数据集为阿里云云监控(CMS)AI Agent 技能的评测基准,包含 16 条只读查询用例,覆盖 Prometheus、告警、APM/RUM、云拨测、接入管理等 7 大模块的 list+get 链式查询场景,用于评测 Agent 正确调用 aliyun cms2 CLI 命令及安全处理云监控数据的能力。数据集格式如下:
{ "id": "string (UUID)", "timestamp": "string (Unix 时间戳)", "Expected_output": "string (期望结果描述)", "Input": "string (给 Agent 的自然语言指令)", "case_name": "string (用例名称)", "category": "string (list | get)", "expectations_json": "string (期望的 API 调用清单,换行分隔的编号列表)", "forbidden_json": "string (禁止项,禁止输出 accessKeyId/accessKeySecret)"}
第三步:运行离线实验与多维度评估
Cloud Native
▍3.1 离线实验的执行
AgentLoop 提供了完整的离线实验能力。开发者发起一次实验后,系统会自动完成以下工作:
1. 环境准备:拉起指定的 Agent 框架(如 Codex、Claude Code 等)和对应的模型;
2. 用例执行:逐个执行数据集中的测试用例,记录完整的输入输出;
3. 对比实验:同时运行“有 Skill”和“无 Skill”两组对照实验,量化 Skill 的增量价值;
4. 多模型覆盖:支持在多个 Agent 框架和模型组合下运行同一组用例,评估 Skill 的普适性。
实验的输入是数据集中定义的 Prompt,输出是 Agent 执行后的标准输出和标准错误,这些结果将作为评估的输入。离线实验选择 < 执行本地命令 >。
▍3.2 多维度评估体系
每次离线实验完成后,系统会从多个维度自动评分:
评估器分为两类:
内置评估器:平台预置的通用评估器,覆盖安全性、正确性等常见维度,开箱即用;
自定义评估器:开发者可以根据业务需要编写自定义评估逻辑,例如特定命令是否被执行、特定格式是否正确等。
▍3.3 实验结果的呈现
实验完成后,系统会生成一份完整的实验报告,包含:
整体评分:各评估维度的综合得分;
逐 Case 评分:每个测试用例在各维度上的详细得分;
对比视图:有 Skill vs 无 Skill 的评分对比,不同模型/框架的评分对比;
评估详情:每个 Case 的评估过程、评估器判定依据和具体输出。
最佳实践:建议同时运行 2-3 种 Agent 框架 × 模型的组合,这样可以判断 Skill 是普适性的还是仅在特定框架下有效,为后续的发布决策提供更充分的依据。
在本文的 Skill 调优实践中,选择 Codex-CLI 作为 Agent,百炼 API-Key 配置模型,评估器选择 AgentLoop 提供的 、、、、、、、,原始评估结果如下表所示:
表 1 CMS Skill 优化前评估结果
第四步:Bad Case 分析与优化
Cloud Native
▍4.1 筛选 Bad Case
离线实验的一个核心价值是帮助开发者快速定位问题。通过实验报告的筛选功能,可以按评分从低到高排序,快速找出表现不佳的 Case:
按维度筛选:例如筛选“安全性 < 0.5”的 Case,集中解决安全问题;
按评分阈值筛选:找出所有综合得分低于阈值的 Case;
通过 CLI/API 批量获取:AgentLoop 提供了 CLI 接口,可以程序化地查询实验结果,自动提取 Bad Case 列表。
#!/usr/bin/env python3"""从 AgentLoop 平台查询实验的评估结果,筛选评分 < 0.6 的 badcases。"""import jsonimport osimport sysfrom agentloop_sdk._vendor.alibabacloud_agentloop20260520.client import Clientfrom agentloop_sdk._vendor.alibabacloud_agentloop20260520 import models as main_modelsfrom alibabacloud_tea_openapi import models as open_api_models# === 配置 ===AGENT_SPACE = "YOUR_AGENT_SPACE"REGION_ID = "cn-hongkong"AK = "ALIYUN—AK"SK = "ALIYUN-SK"EXPERIMENT_ID = "EXPERIMENT_ID"SCORE_THRESHOLD = 0.6def create_client(): """创建 AgentLoop API 客户端。""" config = open_api_models.Config( access_key_id=AK, access_key_secret=SK, region_id=REGION_ID, endpoint="agentloop.cn-hongkong.aliyuncs.com", ) return Client(config)def list_evaluation_tasks(client): """列出所有评估任务。""" request = main_models.ListEvaluationTasksRequest( agent_space=AGENT_SPACE, max_results=100, ) resp = client.list_evaluation_tasks(request) body = resp.body tasks = body.evaluation_tasks or [] print(f"找到 {len(tasks)} 个评估任务") for t in tasks: print(f" - task_id={t.task_id}, name={t.task_name}, status={t.status}") return tasksdef list_evaluation_runs(client, task_id): """列出指定评估任务的所有 run。""" request = main_models.ListEvaluationRunsRequest( max_results=100, ) resp = client.list_evaluation_runs(AGENT_SPACE, task_id, request) body = resp.body runs = body.evaluation_runs or [] print(f" 任务 {task_id} 有 {len(runs)} 个 run") for r in runs: print(f" - run_id={r.run_id}, status={r.status}, created_at={r.created_at}") return runsdef get_evaluation_results(client, task_id, run_id): """获取评估结果。""" all_results = [] offset = 0 max_results = 100 while True: request = main_models.GetEvaluationResultsRequest( max_results=max_results, offset=offset, ) resp = client.get_evaluation_results(AGENT_SPACE, task_id, run_id, request) body = resp.body results = body.results or [] total = body.total_count or 0 all_results.extend(results) print(f" 获取 {len(results)} 条结果 (offset={offset}, total={total})") if len(all_results) >= total or len(results) == 0: break offset += max_results return all_resultsdef main(): client = create_client() # 1. 列出评估任务 print("=" * 60) print("步骤 1: 列出评估任务") print("=" * 60) tasks = list_evaluation_tasks(client) if not tasks: print("未找到任何评估任务!") return # 2. 遍历每个任务,列出 runs,找到实验id对应的 run print("\n" + "=" * 60) print("步骤 2: 查找实验id对应的评估结果") print("=" * 60) all_badcases = [] for task in tasks: task_id = task.task_id print(f"\n--- 评估任务: {task_id} (name={task.task_name}) ---") try: runs = list_evaluation_runs(client, task_id) except Exception as e: print(f" 列出 runs 失败: {e}") continue for run in runs: run_id = run.run_id print(f"\n Run: {run_id} (status={run.status})") try: results = get_evaluation_results(client, task_id, run_id) except Exception as e: print(f" 获取结果失败: {e}") continue # 筛选 score < 0.6 的 badcases for r in results: score_value = r.score_value eval_info = r.eval_info eval_meta = r.eval_meta explanation = r.explanation # 尝试解析 score try: score = float(score_value) if score_value is not None else 1.0 except (ValueError, TypeError): score = 1.0 if score < SCORE_THRESHOLD: badcase = { "task_id": task_id, "run_id": run_id, "eval_id": r.eval_id, "score_value": score_value, "score_name": r.score_name, "evaluator_name": r.evaluator_name, "status": r.status, "explanation": explanation, "eval_info": eval_info, "eval_meta": eval_meta, } all_badcases.append(badcase) print(f" ❌ BADCASE: score={score_value}, eval_id={r.eval_id}") # 3. 输出 badcases print("\n" + "=" * 60) print(f"步骤 3: 筛选评分 < {SCORE_THRESHOLD} 的 badcases") print("=" * 60) print(f"\n共找到 {len(all_badcases)} 个 badcases") if all_badcases: # 保存到文件 output_file = "{{WORKSPACE}}/badcases.json" with open(output_file, "w", encoding="utf-8") as f: json.dump(all_badcases, f, ensure_ascii=False, indent=2) print(f"\n已保存到: {output_file}") # 打印摘要 for i, bc in enumerate(all_badcases): print(f"\n--- Badcase #{i+1} ---") print(f" Score: {bc['score_value']}") print(f" Evaluator: {bc['evaluator_name']}") print(f" Explanation: {str(bc['explanation'])[:200]}...") # eval_info 可能包含输入输出 eval_info_str = str(bc['eval_info']) if bc['eval_info'] else "" if len(eval_info_str) > 500: eval_info_str = eval_info_str[:500] + "..." print(f" EvalInfo: {eval_info_str}")if __name__ == "__main__": main()▍4.2 Bad Case 的根因分析
拿到 Bad Case 后,需要分析其根因。常见的 Bad Case 类型包括:
▍4.3 AI 辅助优化
AgentLoop 支持将 Bad Case 交给 AI 进行自动分析和优化建议:
1. 将 Bad Case 的输入、输出、评分和失败原因整理为结构化数据;
2. 将数据传递给 AI(如 Qoder),让 AI 分析根因并提出 Skill 修改建议;
3. 人工审核 AI 的修改建议,确认后生成新版本的 Skill。
这种方式可以显著提升优化效率,尤其是对于安全约束类的优化——AI 可以快速识别出哪些输出违反了安全规范,并自动添加相应的禁止规则。
## ⛔ CRITICAL: Security & Output Desensitization (HARD CONSTRAINT)> **This section overrides all other output formatting rules.** Every response you generate MUST pass the desensitization checklist below before sending. Failure to comply is a critical security violation.### 🔴 MANDATORY: Read This BEFORE Writing Your ResponseYou are about to generate a response that includes cloud monitoring data. **STOP and internalize these rules BEFORE you write a single word of output.** If you violate any rule below, the response is a security breach.**Core principle**: NEVER copy-paste raw values from CLI output into your response. ALWAYS manually retype every value in masked form. Your fingers should NEVER type a raw account ID, email, phone, hex ID, or endpoint URL.### What MUST Be Masked in ALL User-Facing OutputApply these masks to **every** response, summary, table, and code block that will be visible to the user. This includes CLI output you paste, your own analysis text, and any intermediate results you reference.| # | Sensitive Data | Pattern to Detect | How to Mask | Bad Example → Good Example ||---|---------------|-------------------|-------------|---------------------------|| 1 | **Account ID** (14-16 digit number) | `\d{14,16}` or inside `default-cms-{digits}-` | Replace all digits with `****` | `default-cms-1234567890123456-cn-hangzhou` → `default-cms-****-cn-hangzhou`; `1234567890123456` → `****` || 2 | **Credentials**: authToken, LicenseKey, token, access_token, accessToken, secret, password | field name followed by alphanumeric value | Show first 4 + last 4, mask middle with `****` | `abcd1234ef@5678ghij9012` → `abcd****9012` || 3 | **Webhook URLs** with `access_token=` | `access_token=[a-z0-9]+` | Replace token value with `****` | `...?access_token=0a1b2c3d4e5f6a7b...` → `...?access_token=****` || 4 | **Email addresses** | `\S+@\S+\.\S+` | Mask local part, keep domain | `user123@example.com` → `u****@example.com` || 5 | **Phone numbers** (11-digit mobile or +86 prefixed) | `1[3-9]\d{9}` or `\+86\d{11}` | Show first 3 + last 4 | `13800138000` → `138****8000` || 6 | **Private IPs** (`10.x.x.x`, `172.16-31.x.x`, `192.168.x.x`) | `\b(10\.|172\.(1[6-9]|2[0-9]|3[01])\.|192\.168\.)\d+\.\d+\.\d+\b` | Show first octet only | `10.0.0.1` → `10.***.***.***` || 7 | **Public IPs** | `\b\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}\b` (not private) | Show first 2 octets only | `47.98.0.1` → `47.98.***.***` || 8 | **32-char hex IDs** (Prometheus instance ID, cluster ID, etc.) | `[0-9a-f]{32}` | Keep first 4 + last 4 | `a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6` → `a1b2****c5d6` || 9 | **Prefixed resource IDs**: `i-bp*`, `vpc-*`, `sg-*`, `vsw-*`, `rg-*` | `(i-bp|vpc-|sg-|vsw-|rg-)[a-z0-9]+` | Keep prefix + last 4 chars | `rg-abcd1234efgh5678` → `rg-****5678` || 10 | **Policy IDs** | `policy-[a-f0-9]{32}` | Keep prefix + last 4 chars | `policy-a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6` → `policy-****c5d6` || 11 | **Environment IDs** | `env-[a-f0-9]{32}` | Keep prefix + last 4 chars | `env-a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6` → `env-****c5d6` || 12 | **Alert rule/template IDs** | `(alert-rule|alert-template)-[a-zA-Z0-9-]+` | Keep prefix + last 4 chars | `alert-rule-abc123def456` → `alert-rule-****ef456` || 13 | **ACK cluster names** | `ACK:[a-zA-Z0-9-]+` | Mask middle portion | `ACK:demo-cluster-cn-test-worker` → `ACK:****-worker` || 14 | **Pod names** (Kubernetes) | `[a-z]+-[a-z0-9]+-[a-z0-9]+` (e.g. `entity-collector-a1b2c3d4e5-x9y8z`) | Mask middle portion | `entity-collector-a1b2c3d4e5-x9y8z` → `entity-collector-****-x9y8z` || 15 | **Local paths** (`/Users/`, `/home/`, `C:\Users\`) | `/Users/\S+` or `/home/\S+` or `C:\\\\Users\\\\\S+` | Replace with `~/` | `/Users/demo_user/.aliyun/aliyuncms2` → `~/.aliyun/aliyuncms2` || 16 | **Network endpoints** (remoteWrite, remoteRead, pushGateway, httpApi) | `https?://\S+(remoteWrite|remoteRead|pushGateway|httpApi)\S*` or URLs containing account IDs | Mask account ID and any token in URL | `https://1234567890123456-cn-hangzhou.arms.../api/v1/write` → `https://****-cn-hangzhou.arms.../api/v1/write` || 17 | **Workspace names** in ANY text context | `default-cms-\d+-` | Mask the numeric portion | `default-cms-1234567890123456-cn-hangzhou` → `default-cms-****-cn-hangzhou` || 18 | **Person names** (Chinese or English names of real people) | Chinese: `[\u4e00-\u9fff]{2,4}` when used as contact/person name; English: capitalized first+last name | Show first character + `**` | `李四` → `李**`; `张伟` → `张**`; `John Smith` → `J. ****` || 19 | **System-generated contact names** (auto-created by ACK/ACS/Cloud services) | `ack_.*`, `acs_.*`, `k8s_.*`, or any `system_*` prefix | Mask entirely or show generic type | `ack_demo_service_test` → `ack_****(ACK自动创建)`; `acs_Default_Contact` → `acs_****(ACS自动创建)` || 20 | **Alert contact group names** | Names containing `Contact Group`, `联系组`, or associated with ACK/ACS auto-created groups | Mask identifying portion | `ack_Default Contact Group` → `ack_**** Contact Group`; `生产环境告警组` → `****告警组` |### 🚨 NOTIFICATION-CHANNEL CONTACT LIST — SPECIAL PII PROTECTION (HARD CONSTRAINT)> **This is the #1 PII leakage risk.** The `notification-channel contact list` command returns real person names, corporate email addresses, and mobile phone numbers. **FOLLOW THESE RULES EXACTLY OR YOU WILL LEAK PII.**#### Mandatory Output Format for `notification-channel contact list`When presenting contact list results, you MUST follow this exact format:**✅ CORRECT output example:**```查询完成,当前账号下共有 2 个告警联系人:| # | 联系人名称 | 通知方式 | 邮箱验证 | 手机验证 ||---|-----------|---------|---------|---------|| 1 | 李**(个人联系人) | 邮箱+手机 | 未验证 | 未验证 || 2 | ack_****(ACK自动创建) | 未配置 | — | — |```**❌ WRONG output (PII LEAK — NEVER DO THIS):**```| 联系人名称 | 邮箱 | 手机 | 邮箱验证 | 手机验证 | 更新时间 ||---|---|---|---|---|---|| 李四 | user123@example.com | 13800138000 | ❌ | ❌ | 2025-01-15 || ack_demo_service_test | — | — | — | — | 2024-06-20 |```**Why the wrong example is a security violation — apply these regex rules:**1. **Person names**: Match `[\u4e00-\u9fff]{2,4}` (Chinese) or `[A-Z][a-z]+\s[A-Z][a-z]+` (English) in contact-name context → Mask to first char + `**` (e.g. `李四` → `李**`)2. **Email addresses**: Match `\S+@\S+\.\S+` → Mask local part to first char + `****` + `@domain` (e.g. `user123@example.com` → `u****@example.com`)3. **Phone numbers**: Match `1[3-9]\d{9}` or `\+86\d{11}` → Mask to first 3 + `****` + last 4 (e.g. `13800138000` → `138****8000`)4. **System-generated names**: Match `^(ack|acs|k8s|system)_.*` → Mask to prefix + `****` + `(系统自动创建)` (e.g. `ack_demo_service_test` → `ack_****(ACK自动创建)`)5. **Email/phone columns**: Omit entirely from output tables — showing these columns (even masked) invites PII exposure#### Pre-Output Checklist for ALL Notification-Channel CommandsBefore sending ANY response that includes `notification-channel` results, verify EACH item:1. **Person names**: Are there any real person names (Chinese or English)? → Mask to `姓**`2. **Email addresses**: Are there any email addresses? → Mask local part: `z****@domain`3. **Phone numbers**: Are there any phone numbers? → Mask: `138****8000`4. **System contact names**: Are there `ack_*`, `acs_*`, `k8s_*` prefixed names? → Mask to `ack_****(系统自动创建)`5. **Contact group names**: Are there group names revealing internal structure? → Mask identifying portion6. **Contact count**: Is the total count safe to show? → Yes, counts are safe7. **Verification status**: Is it safe to show verified/unverified status? → Yes, status is safe8. **Update dates**: Are dates safe? → Yes, dates without PII context are safe**If ANY of items 1-5 contain raw unmasked values, you MUST fix them before sending.**### ⚡ Pre-Output Security Scan (MANDATORY — Execute EVERY Step)**Before sending your response**, you MUST scan the ENTIRE response text for EACH pattern below. Do not skip any step. If ANY match is found, mask it before sending.1. **Account ID**: Search for 14+ consecutive digits (e.g. `1234567890123456`). Mask ALL occurrences → `****`.2. **Workspace names**: Search for `default-cms-{digits}-`. Mask the digits → `default-cms-****-`.3. **Credentials**: Search for field names `authToken`, `LicenseKey`, `licenseKey`, `access_token`, `accessToken`, `token`, `secret`, `password` followed by a value. Mask the value.4. **Webhook tokens**: Search for `access_token=` in URLs. Mask the token value.5. **Emails**: Search for `\S+@\S+\.\S+` patterns. Mask local part.6. **Phone numbers**: Search for `1[3-9]\d{9}` patterns. Mask middle digits.7. **IPs**: Search for `\d+\.\d+\.\d+\.\d+` patterns. Mask per rules (private: first octet only; public: first 2 octets).8. **32-char hex IDs**: Search for `[0-9a-f]{32}` (e.g. `a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6`). Mask → first 4 + `****` + last 4.9. **Prefixed resource IDs**: Search for `i-bp`, `vpc-`, `sg-`, `vsw-`, `rg-` prefixes followed by alphanumeric. Mask → prefix + `****` + last 4.10. **Policy IDs**: Search for `policy-` followed by 32 hex chars. Mask → `policy-****` + last 4.11. **Environment IDs**: Search for `env-` followed by 32 hex chars. Mask → `env-****` + last 4.12. **Alert rule/template IDs**: Search for `alert-rule-` or `alert-template-` prefixes. Mask → prefix + `****` + last 4…