Compare commits

...

12 Commits

Author SHA1 Message Date
cfdaily 67cad2dd96 fix: _REASON_MAP 补 agent_error 条目(G2)
CI / lint (pull_request) Successful in 6s
CI / test (pull_request) Successful in 8s
CI / notify-on-failure (pull_request) Successful in 0s
spawner 会产生 agent_error reason,之前缺映射走到 _default 显示'未知原因'。
2026-06-13 09:35:15 +08:00
pangtong-fujunshi 79da0bd07e Merge PR #56 2026-06-13 01:34:39 +00:00
cfdaily a116f7e6c0 fix: 注释拼写 must_hives → must_haves
CI / lint (pull_request) Successful in 7s
CI / test (pull_request) Successful in 8s
CI / notify-on-failure (pull_request) Successful in 0s
2026-06-13 09:33:59 +08:00
cfdaily 7fb4d988ec fix: lint 修复 + api_error 测试更新
CI / lint (pull_request) Successful in 6s
CI / test (pull_request) Successful in 8s
CI / notify-on-failure (pull_request) Successful in 0s
- mail_notify: f-string 反斜杠修复、行过长修复、unused import
- test_classify_outcome: api_error should_retry 改 True
2026-06-13 09:29:52 +08:00
cfdaily f4dd9ff78d feat(daemon): Mail 失败通知 v2.0 — api_error retry + 通知增强
CI / lint (pull_request) Failing after 7s
CI / test (pull_request) Has been skipped
CI / notify-on-failure (pull_request) Successful in 1s
P1: api_error rate_limit/500/503 改为可恢复 retry(should_retry=True,60s cooldown)
P2: 通知模板动态化(reason 人话翻译 + detail 信息 + 重试情况 + AI Native 知识库)

设计文档:§20.7 (20-task-type-architecture.md)
2026-06-13 09:27:17 +08:00
pangtong-fujunshi 6520e78c0b Merge PR #55 2026-06-13 01:23:33 +00:00
cfdaily 0169823b72 chore(docs): 合并 mail-failure-notification 到 §20,更新设计方案
CI / lint (pull_request) Successful in 6s
CI / test (pull_request) Successful in 9s
CI / notify-on-failure (pull_request) Successful in 0s
- mail-failure-notification.md → archive-3.0/
- §20 新增 §20 Mail 失败通知机制(v2.0 AI Native)
  - 失败场景与重试耗时完整表
  - reason 人话翻译映射
  - 通知模板增强(detail 传入 + 重试情况)
  - api_error rate_limit 待改为可恢复 retry
- §18→§21,§19→§22 编号顺延
2026-06-13 09:22:32 +08:00
pangtong-fujunshi 77252c39c6 Merge PR #54 2026-06-13 00:59:11 +00:00
cfdaily 5a80d6c5cd chore(docs): gateway-watchdog.md 改编号 99
CI / lint (pull_request) Successful in 6s
CI / test (pull_request) Successful in 9s
CI / notify-on-failure (pull_request) Successful in 0s
2026-06-13 08:58:04 +08:00
pangtong-fujunshi 322263585d Merge PR #53 2026-06-13 00:54:39 +00:00
cfdaily c7b4b262b1 chore(docs): 归档 §13-sim §18 §21 §25 至 archive-3.0
CI / lint (pull_request) Successful in 7s
CI / test (pull_request) Successful in 9s
CI / notify-on-failure (pull_request) Successful in 0s
- 13-toolchain-and-dev-workflow-simulation.md → archive-3.0/(模拟报告,§16 已覆盖)
- 18-toolchain-e2e-test.md → archive-3.0/(E2E 测试记录,§13 已引用)
- 21-e2e-verification-handler.md → archive-3.0/(Handler 验证,§20 §19 已覆盖)
- 25-gitea-mention-toolchain.md → archive-3.0/(@mention 集成,§13 §16 已覆盖)
2026-06-13 08:53:23 +08:00
pangtong-fujunshi e43d87f3db Merge PR #52 2026-06-13 00:53:09 +00:00
10 changed files with 271 additions and 26 deletions
+146 -2
View File
@@ -950,7 +950,151 @@ handler.post_complete(task_id, agent_id, outcome, db_path)
---
# §18 设计决策记录
## §20. Mail 失败通知机制
### 20.1 背景
Mail 是 A→B 点对点通信,失败应通知发件人 A,而非统一 @pangtong。
当前机制(v1.3 已实现):
- `_mark_task("failed")` 对 _mail 项目:调用 `mail_notify.notify_mail_failed` 通知发件人
- `_mark_task("failed")` 对 Task 项目:@pangtong-fujunshiF2 原逻辑不变)
- `_mail_auto_complete` 的 no_reply_found:标 failed 后通知发件人
- 防递归:`must_haves.system_notify=true` 的邮件失败不再递归通知
### 20.2 失败场景与重试机制
所有可能的失败路径及其重试/等待机制(重试上限 max_retries=3agent_timeout=630s):
| 失败类型 | 机制 | 重试次数 | 每次耗时 | cooldown | 最长总耗时 |
|---|---|---|---|---|---|
| `gateway_timeout` | 续杯 | 3 | 630s | 无 | ~31.5 分钟 |
| `crashed` | ticker 兜底 | 3 | ~2-5 分钟 | 60s + 30s ticker | ~15 分钟 |
| `api_error`rate_limit | 推 pending**待改为续杯** | 3 | ~2.5 分钟 | 120s | ~8 分钟 |
| `compact_interrupted` | 续杯 | 3 | 630s | 60s | ~34 分钟 |
| `gateway_unreachable` | 续杯 | 3 | 630s | 60s | ~34 分钟 |
| `lock_conflict` | 续杯 | 3 | 630s | 60s | ~34 分钟 |
| `fallback_timeout` | 续杯(A3b) | 3 | 630s | 60s | ~34 分钟 |
| `compact_wait` | monitor 等待 | 3 | 630s | 无 | ~31.5 分钟 |
| `compact_hanging` | monitor → release | 3 | 630s | 300s | ~31.5 分钟 + ticker |
| `max_monitor_timeouts` | monitor 上限 | 3 | 630s | 无 | ~31.5 分钟 |
| `session_stuck` | revive 1 次 | 1 | ~30s | 无 | ~30 秒 |
| `compact_failed` | 无重试 | 0 | — | 300s | 立刻 failed |
| `auth_failed` | 无重试 | 0 | — | — | 立刻 failed |
| `agent_error` | 无重试 | 0 | — | 300s | 立刻 failed |
| `no_reply_found` | 无重试 | 0 | — | — | 立刻 failed |
### 20.3 触发点
| 触发点 | 文件 | 说明 |
|---|---|---|
| `_mark_task(failed)` | spawner.py | _mail 项目 → notify_mail_failedTask 项目 → @pangtong |
| `_mail_auto_complete` no_reply_found | dispatcher.py | Agent 正常退出但没回复 request → 标 failed → 通知发件人 |
### 20.4 实现位置
- `src/daemon/mail_notify.py``notify_mail_failed` + `_is_mail_project` + 通知模板
- `src/daemon/spawner.py``_mark_task` 中 _mail/Task 分流
- `src/daemon/dispatcher.py``_mail_auto_complete` 中 no_reply_found 后调 notify
### 20.5 通知设计(v2.0 — AI Native
通知提供充足事实信息,不做硬编码处理建议。收件 AI 自行判断下一步。
**通知结构**
```
邮件投递失败通知
📧 原始邮件:「{title}」
👤 收件人:{to_agent}
❌ 失败原因:{reason_human_readable}{reason_raw}
📊 重试情况:{attempt_info}
📋 上下文信息:
{detail_formatted}
常见失败原因参考:
• no_reply_found:收件人未回复(Agent 未能识别或处理此邮件)
• crashed / max_crash_count:收件人处理时进程崩溃(已自动重试 3 次)
• max_retries:续杯耗尽(已自动重试 3 次,共约 34 分钟)
• max_api_retry_countAPI 连续失败达上限(rate_limit/500/503
• max_monitor_timeouts:处理超时达上限(共约 31.5 分钟)
• gateway_timeoutAgent 执行超时(已续杯重试)
• session_stuckAgent 会话假死(lock PID 死亡,revive 失败)
• revive_failed:会话假死后恢复失败
• auth_failedAgent 认证失败(配置问题)
• fallback_exhausted:主模型和备用模型均失败
• agent_failed:收件人主动标记失败
• compact_failed:上下文压缩失败
• compact_hanging:上下文压缩长时间未完成(等待超 31.5 分钟)
• compact_interrupted:上下文压缩被中断(已自动重试 3 次)
• gateway_unreachableGateway 不可达(已自动重试 3 次)
• lock_conflict:会话锁冲突(已自动重试 3 次)
• 其他:建议排查系统日志
——系统自动通知
```
**reason 人话翻译映射**
| reason_raw | reason_human_readable | detail 提取 |
|---|---|---|
| `no_reply_found` | 收件人未回复 | 无额外信息 |
| `crashed` | 处理时进程崩溃 | stderr_preview 前 200 字 |
| `max_crash_count` | 连续崩溃达上限 | count + stderr_preview |
| `max_retries` | 续杯耗尽 | count + retry_field |
| `max_api_retry_count` | API 连续失败达上限 | count |
| `max_monitor_timeouts` | 处理超时达上限 | count + elapsed_seconds |
| `gateway_timeout` | Agent 执行超时 | retry_count |
| `session_stuck` | 会话假死 | stuck_count |
| `revive_failed` | 假死后恢复失败 | stuck_count |
| `auth_failed` | 认证失败 | stderr_preview |
| `fallback_exhausted` | 模型全部失败 | fallback_count + fallback_reason |
| `agent_failed` | 收件人主动标失败 | 无 |
| `compact_failed` | 上下文压缩失败 | stderr_preview |
| `compact_hanging` | 压缩长时间未完成 | compact_wait_count |
| `compact_interrupted` | 压缩被中断 | 无 |
| `gateway_unreachable` | Gateway 不可达 | stderr_preview |
| `lock_conflict` | 会话锁冲突 | 无 |
| 默认 | 未知原因 | reason + stderr_preview(如有) |
**重试情况格式**
- 有重试:`"已自动重试 {count} 次,共耗时约 {total_time}"`
- 无重试:`"无法重试({reason_human_readable}"`
### 20.6 防递归
系统通知邮件(from=system)本身也可能失败:
- 检查 `must_haves.system_notify=true` → 跳过递归通知
- system 不是有效 Agent → 通知路由到 pangtong-fujunshi 代处理
### 20.7 待实现改动
#### P1api_error rate_limit 改为可恢复 retry
**当前**`_classify_outcome` 中 rate_limit/500/503 → `api_error``should_retry=False`,走推 pending 路径。
**改为**`should_retry=True`,走续杯路径。cooldown 60s。上限仍 3 次。
**改动文件**`src/daemon/spawner.py` `_classify_outcome` 的 `api_error` 分支。
**影响**`api_retry_count` 机制可以废弃(统一用 `retry_count`),但保持向后兼容暂不删除。
#### P2:通知模板更新(v2.0
**当前**`mail_notify.py` 的 `_NOTIFY_TEMPLATE` 是静态模板,不传 detail。
**改为**:动态模板,根据 reason 选择人话翻译 + 提取 detail 信息 + 格式化重试情况。
**改动文件**`src/daemon/mail_notify.py`。
**新增**`_REASON_MAP` 字典(reason → 人话 + detail 提取函数)。
### 20.8 不改的
| 项目 | 原因 |
|---|---|
| F2 @pangtong 对 Task 的逻辑 | Task failed 仍 @pangtong,只对 Mail 不同 |
| no_reply_found 的判定逻辑 | 只在判定后加通知,不改判定本身 |
| inform 类型邮件的完成逻辑 | inform 直接 done,不存在 no_reply_found |
| 外部 API 的 from 校验 | system 不走 HTTP,外部无法伪造 |
---
# §21 设计决策记录
本节记录设计过程中的关键讨论和决策,便于未来回顾。
@@ -1010,7 +1154,7 @@ handler.post_complete(task_id, agent_id, outcome, db_path)
---
## §19. 审查与验证历史
## §22. 审查与验证历史
### Step 2-5 背靠背审查(2026-06-10/11
+117 -21
View File
@@ -1,4 +1,4 @@
"""Mail 失败通知 — 以 system 身份通知发件人"""
"""Mail 失败通知 v2.0 — 以 system 身份通知发件人AI Native"""
from __future__ import annotations
@@ -6,7 +6,7 @@ import json
import logging
from datetime import datetime
from pathlib import Path
from typing import Optional
from typing import Dict, Optional
from src.blackboard.models import Task
from src.blackboard.operations import Blackboard
@@ -15,21 +15,121 @@ from src.config.agents import AGENT_IDS
logger = logging.getLogger(__name__)
# 邮件通知正文模板(统一模板,包含所有可能的失败原因和建议)
_NOTIFY_TEMPLATE = """你的邮件投递失败了。
# ── Reason 人话翻译 + detail 提取 ──────────────────────────────
📧 原始邮件:「{title}
👤 收件人:{to_agent}
❌ 失败原因:{reason}
def _extract_stderr(detail: dict, max_len: int = 200) -> str:
"""从 detail 中提取 stderr_preview"""
preview = (detail or {}).get("stderr_preview", "")
if preview and len(preview) > max_len:
preview = preview[:max_len] + "..."
return preview
常见失败原因及处理建议:
• no_reply_found:收件人未回复。建议重发邮件,或通过黑板任务方式联系
• auth_failed:收件人认证失败。需检查 Agent 配置,联系姜维(jiangwei-infra)排查
• crash_limit:收件人处理时多次崩溃。系统异常,建议稍后重试
• task_timeout:处理超时。建议重发或通过其他方式联系
• 其他原因:建议联系副军师(pangtong-fujunshi)排查
——系统自动通知"""
def _fmt_retry_info(reason: str, detail: dict) -> str:
"""格式化重试情况描述"""
_NO_RETRY_REASONS = {
"no_reply_found", "auth_failed", "agent_error",
"agent_failed", "compact_failed",
}
if reason in _NO_RETRY_REASONS:
reason_human = _REASON_MAP.get(reason, _REASON_MAP.get("_default", ("未知原因", lambda d: "")))[0]
return f"无法重试({reason_human}"
count = (detail or {}).get("count", 0)
fallback_count = (detail or {}).get("fallback_count", 0)
if count > 0:
return f"已自动重试 {count}"
if fallback_count > 0:
return f"已自动重试 {fallback_count} 次(fallback"
return "系统已尝试恢复,但仍失败"
# reason_raw → (reason_human_readable, detail_format_fn)
_REASON_MAP: Dict[str, tuple] = {
"no_reply_found": ("收件人未回复(Agent 未能识别或处理此邮件)", lambda d: ""),
"crashed": ("处理时进程崩溃", lambda d: f"stderr: {_extract_stderr(d)}" if _extract_stderr(d) else "无 stderr 输出"),
"max_crash_count": ("连续崩溃达上限", lambda d: f"崩溃 {d.get('count', '?')}"),
"max_retries": ("续杯耗尽(已自动重试)", lambda d: f"重试 {d.get('count', '?')}"),
"max_api_retry_count": ("API 连续失败达上限", lambda d: f"API 重试 {d.get('count', '?')}"),
"max_monitor_timeouts": (
"处理超时达上限",
lambda d: f"超时 {d.get('count', '?')} 次,"
f"共约 {d.get('elapsed_seconds', 0) // 60} 分钟"),
"gateway_timeout": ("Agent 执行超时(已续杯重试)", lambda d: ""),
"session_stuck": ("会话假死(lock PID 死亡)", lambda d: f"假死 {d.get('stuck_count', '?')}"),
"revive_failed": ("会话恢复失败", lambda d: f"假死 {d.get('stuck_count', '?')}"),
"auth_failed": ("Agent 认证失败(配置问题)", lambda d: f"stderr: {_extract_stderr(d)}" if _extract_stderr(d) else ""),
"fallback_exhausted": (
"主模型和备用模型均失败",
lambda d: f"fallback {d.get('fallback_count', '?')} 次,"
f"原因: {d.get('fallback_reason', '未知')}"),
"agent_error": (
"Agent 内部错误",
lambda d: f"stderr: {_extract_stderr(d)}" if _extract_stderr(d) else ""),
"agent_failed": ("收件人主动标记失败", lambda d: ""),
"compact_failed": ("上下文压缩失败", lambda d: f"stderr: {_extract_stderr(d)}" if _extract_stderr(d) else ""),
"compact_hanging": ("上下文压缩长时间未完成", lambda d: ""),
"compact_interrupted": ("上下文压缩被中断(已自动重试)", lambda d: ""),
"gateway_unreachable": (
"Gateway 不可达(已自动重试)",
lambda d: f"stderr: {_extract_stderr(d)}"
if _extract_stderr(d) else ""),
"lock_conflict": ("会话锁冲突(已自动重试)", lambda d: ""),
"max_retry_count": ("重试耗尽", lambda d: f"重试 {d.get('count', '?')}"),
"max_lock_retry_count": ("锁冲突重试耗尽", lambda d: f"重试 {d.get('count', '?')}"),
"max_connect_retry_count": ("连接重试耗尽", lambda d: f"重试 {d.get('count', '?')}"),
"_default": ("未知原因", lambda d: f"stderr: {_extract_stderr(d)}" if _extract_stderr(d) else ""),
}
# 常见失败原因参考(AI Native:提供知识库让收件 AI 自行判断)
_REASON_REFERENCE = """常见失败原因参考:
• no_reply_found:收件人未回复(Agent 未能识别或处理此邮件)
• crashed / max_crash_count:收件人处理时进程崩溃(已自动重试 3 次)
• max_retries:续杯耗尽(已自动重试 3 次,共约 34 分钟)
• max_api_retry_countAPI 连续失败达上限(rate_limit/500/503
• max_monitor_timeouts:处理超时达上限(共约 31.5 分钟)
• gateway_timeoutAgent 执行超时(已续杯重试)
• session_stuckAgent 会话假死(lock PID 死亡,revive 失败)
• revive_failed:会话假死后恢复失败
• auth_failedAgent 认证失败(配置问题)
• fallback_exhausted:主模型和备用模型均失败
• agent_failed:收件人主动标记失败
• compact_failed:上下文压缩失败
• compact_hanging:上下文压缩长时间未完成(等待超 31.5 分钟)
• compact_interrupted:上下文压缩被中断(已自动重试 3 次)
• gateway_unreachableGateway 不可达(已自动重试 3 次)
• lock_conflict:会话锁冲突(已自动重试 3 次)
• 其他:建议排查系统日志"""
def _build_notify_text(title: str, to_agent: str, reason: str,
detail: Optional[dict] = None) -> str:
"""构建通知正文(v2.0 AI Native"""
reason_human, detail_fn = _REASON_MAP.get(reason, _REASON_MAP["_default"])
detail_info = detail_fn(detail or {})
retry_info = _fmt_retry_info(reason, detail or {})
lines = [
"邮件投递失败通知",
"",
f"📧 原始邮件:「{title}",
f"👤 收件人:{to_agent}",
f"❌ 失败原因:{reason_human}{reason}",
f"📊 重试情况:{retry_info}",
]
if detail_info:
lines.append("📋 上下文信息:")
lines.append(f" {detail_info}")
lines.append("")
lines.append(_REASON_REFERENCE)
lines.append("")
lines.append("——系统自动通知")
return "\n".join(lines)
def _is_mail_project(db_path: Path) -> bool:
@@ -43,7 +143,7 @@ def notify_mail_failed(db_path: Path, original_mail_id: str,
"""Mail 失败后以 system 身份给发件人发通知邮件
直接通过 Blackboard 创建 Task,不走 HTTP API。
防递归:检查原邮件 must_hives.system_notify,为 true 则跳过。
防递归:检查原邮件 must_haves.system_notify,为 true 则跳过。
发件人不是有效 Agent(如 system)→ 通知庞统代处理,避免广播风暴。
"""
try:
@@ -83,12 +183,8 @@ def notify_mail_failed(db_path: Path, original_mail_id: str,
original_mail_id, from_agent)
target_agent = "pangtong-fujunshi"
# 构造通知正文
text = _NOTIFY_TEMPLATE.format(
title=title,
to_agent=to_agent,
reason=reason,
)
# 构造通知正文v2.0 AI Native
text = _build_notify_text(title, to_agent, reason, detail)
# 创建通知邮件 Task
notify_id = f"mail-{int(datetime.now().timestamp() * 1000)}"
+4 -1
View File
@@ -845,6 +845,8 @@ curl -X POST http://{api_host}:{api_port}/api/projects/{project_id}/tasks/{task_
cls.get("retry_field", "retry_count")
)
elif outcome == "api_error":
# A9: [DEPRECATED] api_error 已改为 should_retry=True 走续杯路径。
# 此分支理论上不再命中,保留作为安全兜底。
# A9: 429/API 错误 → release counter(on_complete)+ 推回 pending + 冷却
# 有上限:api_retry_count 累计达 max_retries 则标 failed
await self._do_on_complete_async(on_complete, agent_id, outcome)
@@ -1842,7 +1844,8 @@ curl -X POST http://{api_host}:{api_port}/api/projects/{project_id}/tasks/{task_
"retry_field": "retry_count", "cooldown_seconds": 60}
if any(kw in stderr_lower for kw in [
"rate_limit", "500", "503", "api error"]):
return {"outcome": "api_error", "should_retry": False}
return {"outcome": "api_error", "should_retry": True,
"retry_field": "retry_count", "cooldown_seconds": 60}
if any(kw in stderr_lower for kw in [
"compaction-diag", "context-overflow"]):
return {"outcome": "compact_failed", "should_retry": False}
+4 -2
View File
@@ -165,14 +165,16 @@ class TestClassifyErrorApi:
1, {"status": "error"}, "rate_limit exceeded", None
)
assert result["outcome"] == "api_error"
assert result["should_retry"] is False
assert result["should_retry"] is True
assert result["cooldown_seconds"] == 60
def test_stderr_500(self):
result = Spawner._classify_outcome(
1, {"status": "error"}, "HTTP 500 Internal Server Error", None
)
assert result["outcome"] == "api_error"
assert result["should_retry"] is False
assert result["should_retry"] is True
assert result["cooldown_seconds"] == 60
class TestClassifyErrorCompact: