Compare commits
6 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| a116f7e6c0 | |||
| 7fb4d988ec | |||
| f4dd9ff78d | |||
| 6520e78c0b | |||
| 0169823b72 | |||
| 77252c39c6 |
@@ -950,7 +950,151 @@ handler.post_complete(task_id, agent_id, outcome, db_path)
|
||||
|
||||
---
|
||||
|
||||
# §18 设计决策记录
|
||||
## §20. Mail 失败通知机制
|
||||
|
||||
### 20.1 背景
|
||||
|
||||
Mail 是 A→B 点对点通信,失败应通知发件人 A,而非统一 @pangtong。
|
||||
|
||||
当前机制(v1.3 已实现):
|
||||
- `_mark_task("failed")` 对 _mail 项目:调用 `mail_notify.notify_mail_failed` 通知发件人
|
||||
- `_mark_task("failed")` 对 Task 项目:@pangtong-fujunshi(F2 原逻辑不变)
|
||||
- `_mail_auto_complete` 的 no_reply_found:标 failed 后通知发件人
|
||||
- 防递归:`must_haves.system_notify=true` 的邮件失败不再递归通知
|
||||
|
||||
### 20.2 失败场景与重试机制
|
||||
|
||||
所有可能的失败路径及其重试/等待机制(重试上限 max_retries=3,agent_timeout=630s):
|
||||
|
||||
| 失败类型 | 机制 | 重试次数 | 每次耗时 | cooldown | 最长总耗时 |
|
||||
|---|---|---|---|---|---|
|
||||
| `gateway_timeout` | 续杯 | 3 | 630s | 无 | ~31.5 分钟 |
|
||||
| `crashed` | ticker 兜底 | 3 | ~2-5 分钟 | 60s + 30s ticker | ~15 分钟 |
|
||||
| `api_error`(rate_limit) | 推 pending(**待改为续杯**) | 3 | ~2.5 分钟 | 120s | ~8 分钟 |
|
||||
| `compact_interrupted` | 续杯 | 3 | 630s | 60s | ~34 分钟 |
|
||||
| `gateway_unreachable` | 续杯 | 3 | 630s | 60s | ~34 分钟 |
|
||||
| `lock_conflict` | 续杯 | 3 | 630s | 60s | ~34 分钟 |
|
||||
| `fallback_timeout` | 续杯(A3b) | 3 | 630s | 60s | ~34 分钟 |
|
||||
| `compact_wait` | monitor 等待 | 3 | 630s | 无 | ~31.5 分钟 |
|
||||
| `compact_hanging` | monitor → release | 3 | 630s | 300s | ~31.5 分钟 + ticker |
|
||||
| `max_monitor_timeouts` | monitor 上限 | 3 | 630s | 无 | ~31.5 分钟 |
|
||||
| `session_stuck` | revive 1 次 | 1 | ~30s | 无 | ~30 秒 |
|
||||
| `compact_failed` | 无重试 | 0 | — | 300s | 立刻 failed |
|
||||
| `auth_failed` | 无重试 | 0 | — | — | 立刻 failed |
|
||||
| `agent_error` | 无重试 | 0 | — | 300s | 立刻 failed |
|
||||
| `no_reply_found` | 无重试 | 0 | — | — | 立刻 failed |
|
||||
|
||||
### 20.3 触发点
|
||||
|
||||
| 触发点 | 文件 | 说明 |
|
||||
|---|---|---|
|
||||
| `_mark_task(failed)` | spawner.py | _mail 项目 → notify_mail_failed;Task 项目 → @pangtong |
|
||||
| `_mail_auto_complete` no_reply_found | dispatcher.py | Agent 正常退出但没回复 request → 标 failed → 通知发件人 |
|
||||
|
||||
### 20.4 实现位置
|
||||
|
||||
- `src/daemon/mail_notify.py`:`notify_mail_failed` + `_is_mail_project` + 通知模板
|
||||
- `src/daemon/spawner.py`:`_mark_task` 中 _mail/Task 分流
|
||||
- `src/daemon/dispatcher.py`:`_mail_auto_complete` 中 no_reply_found 后调 notify
|
||||
|
||||
### 20.5 通知设计(v2.0 — AI Native)
|
||||
|
||||
通知提供充足事实信息,不做硬编码处理建议。收件 AI 自行判断下一步。
|
||||
|
||||
**通知结构**:
|
||||
```
|
||||
邮件投递失败通知
|
||||
|
||||
📧 原始邮件:「{title}」
|
||||
👤 收件人:{to_agent}
|
||||
❌ 失败原因:{reason_human_readable}({reason_raw})
|
||||
📊 重试情况:{attempt_info}
|
||||
📋 上下文信息:
|
||||
{detail_formatted}
|
||||
|
||||
常见失败原因参考:
|
||||
• no_reply_found:收件人未回复(Agent 未能识别或处理此邮件)
|
||||
• crashed / max_crash_count:收件人处理时进程崩溃(已自动重试 3 次)
|
||||
• max_retries:续杯耗尽(已自动重试 3 次,共约 34 分钟)
|
||||
• max_api_retry_count:API 连续失败达上限(rate_limit/500/503)
|
||||
• max_monitor_timeouts:处理超时达上限(共约 31.5 分钟)
|
||||
• gateway_timeout:Agent 执行超时(已续杯重试)
|
||||
• session_stuck:Agent 会话假死(lock PID 死亡,revive 失败)
|
||||
• revive_failed:会话假死后恢复失败
|
||||
• auth_failed:Agent 认证失败(配置问题)
|
||||
• fallback_exhausted:主模型和备用模型均失败
|
||||
• agent_failed:收件人主动标记失败
|
||||
• compact_failed:上下文压缩失败
|
||||
• compact_hanging:上下文压缩长时间未完成(等待超 31.5 分钟)
|
||||
• compact_interrupted:上下文压缩被中断(已自动重试 3 次)
|
||||
• gateway_unreachable:Gateway 不可达(已自动重试 3 次)
|
||||
• lock_conflict:会话锁冲突(已自动重试 3 次)
|
||||
• 其他:建议排查系统日志
|
||||
|
||||
——系统自动通知
|
||||
```
|
||||
|
||||
**reason 人话翻译映射**:
|
||||
|
||||
| reason_raw | reason_human_readable | detail 提取 |
|
||||
|---|---|---|
|
||||
| `no_reply_found` | 收件人未回复 | 无额外信息 |
|
||||
| `crashed` | 处理时进程崩溃 | stderr_preview 前 200 字 |
|
||||
| `max_crash_count` | 连续崩溃达上限 | count + stderr_preview |
|
||||
| `max_retries` | 续杯耗尽 | count + retry_field |
|
||||
| `max_api_retry_count` | API 连续失败达上限 | count |
|
||||
| `max_monitor_timeouts` | 处理超时达上限 | count + elapsed_seconds |
|
||||
| `gateway_timeout` | Agent 执行超时 | retry_count |
|
||||
| `session_stuck` | 会话假死 | stuck_count |
|
||||
| `revive_failed` | 假死后恢复失败 | stuck_count |
|
||||
| `auth_failed` | 认证失败 | stderr_preview |
|
||||
| `fallback_exhausted` | 模型全部失败 | fallback_count + fallback_reason |
|
||||
| `agent_failed` | 收件人主动标失败 | 无 |
|
||||
| `compact_failed` | 上下文压缩失败 | stderr_preview |
|
||||
| `compact_hanging` | 压缩长时间未完成 | compact_wait_count |
|
||||
| `compact_interrupted` | 压缩被中断 | 无 |
|
||||
| `gateway_unreachable` | Gateway 不可达 | stderr_preview |
|
||||
| `lock_conflict` | 会话锁冲突 | 无 |
|
||||
| 默认 | 未知原因 | reason + stderr_preview(如有) |
|
||||
|
||||
**重试情况格式**:
|
||||
- 有重试:`"已自动重试 {count} 次,共耗时约 {total_time}"`
|
||||
- 无重试:`"无法重试({reason_human_readable})"`
|
||||
|
||||
### 20.6 防递归
|
||||
|
||||
系统通知邮件(from=system)本身也可能失败:
|
||||
- 检查 `must_haves.system_notify=true` → 跳过递归通知
|
||||
- system 不是有效 Agent → 通知路由到 pangtong-fujunshi 代处理
|
||||
|
||||
### 20.7 待实现改动
|
||||
|
||||
#### P1:api_error rate_limit 改为可恢复 retry
|
||||
|
||||
**当前**:`_classify_outcome` 中 rate_limit/500/503 → `api_error`,`should_retry=False`,走推 pending 路径。
|
||||
**改为**:`should_retry=True`,走续杯路径。cooldown 60s。上限仍 3 次。
|
||||
**改动文件**:`src/daemon/spawner.py` `_classify_outcome` 的 `api_error` 分支。
|
||||
**影响**:`api_retry_count` 机制可以废弃(统一用 `retry_count`),但保持向后兼容暂不删除。
|
||||
|
||||
#### P2:通知模板更新(v2.0)
|
||||
|
||||
**当前**:`mail_notify.py` 的 `_NOTIFY_TEMPLATE` 是静态模板,不传 detail。
|
||||
**改为**:动态模板,根据 reason 选择人话翻译 + 提取 detail 信息 + 格式化重试情况。
|
||||
**改动文件**:`src/daemon/mail_notify.py`。
|
||||
**新增**:`_REASON_MAP` 字典(reason → 人话 + detail 提取函数)。
|
||||
|
||||
### 20.8 不改的
|
||||
|
||||
| 项目 | 原因 |
|
||||
|---|---|
|
||||
| F2 @pangtong 对 Task 的逻辑 | Task failed 仍 @pangtong,只对 Mail 不同 |
|
||||
| no_reply_found 的判定逻辑 | 只在判定后加通知,不改判定本身 |
|
||||
| inform 类型邮件的完成逻辑 | inform 直接 done,不存在 no_reply_found |
|
||||
| 外部 API 的 from 校验 | system 不走 HTTP,外部无法伪造 |
|
||||
|
||||
---
|
||||
|
||||
# §21 设计决策记录
|
||||
|
||||
本节记录设计过程中的关键讨论和决策,便于未来回顾。
|
||||
|
||||
@@ -1010,7 +1154,7 @@ handler.post_complete(task_id, agent_id, outcome, db_path)
|
||||
|
||||
---
|
||||
|
||||
## §19. 审查与验证历史
|
||||
## §22. 审查与验证历史
|
||||
|
||||
### Step 2-5 背靠背审查(2026-06-10/11)
|
||||
|
||||
|
||||
+114
-21
@@ -1,4 +1,4 @@
|
||||
"""Mail 失败通知 — 以 system 身份通知发件人"""
|
||||
"""Mail 失败通知 v2.0 — 以 system 身份通知发件人(AI Native)"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -6,7 +6,7 @@ import json
|
||||
import logging
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import Optional
|
||||
from typing import Dict, Optional
|
||||
|
||||
from src.blackboard.models import Task
|
||||
from src.blackboard.operations import Blackboard
|
||||
@@ -15,21 +15,118 @@ from src.config.agents import AGENT_IDS
|
||||
logger = logging.getLogger(__name__)
|
||||
|
||||
|
||||
# 邮件通知正文模板(统一模板,包含所有可能的失败原因和建议)
|
||||
_NOTIFY_TEMPLATE = """你的邮件投递失败了。
|
||||
# ── Reason 人话翻译 + detail 提取 ──────────────────────────────
|
||||
|
||||
📧 原始邮件:「{title}」
|
||||
👤 收件人:{to_agent}
|
||||
❌ 失败原因:{reason}
|
||||
def _extract_stderr(detail: dict, max_len: int = 200) -> str:
|
||||
"""从 detail 中提取 stderr_preview"""
|
||||
preview = (detail or {}).get("stderr_preview", "")
|
||||
if preview and len(preview) > max_len:
|
||||
preview = preview[:max_len] + "..."
|
||||
return preview
|
||||
|
||||
常见失败原因及处理建议:
|
||||
• no_reply_found:收件人未回复。建议重发邮件,或通过黑板任务方式联系
|
||||
• auth_failed:收件人认证失败。需检查 Agent 配置,联系姜维(jiangwei-infra)排查
|
||||
• crash_limit:收件人处理时多次崩溃。系统异常,建议稍后重试
|
||||
• task_timeout:处理超时。建议重发或通过其他方式联系
|
||||
• 其他原因:建议联系副军师(pangtong-fujunshi)排查
|
||||
|
||||
——系统自动通知"""
|
||||
def _fmt_retry_info(reason: str, detail: dict) -> str:
|
||||
"""格式化重试情况描述"""
|
||||
_NO_RETRY_REASONS = {
|
||||
"no_reply_found", "auth_failed", "agent_error",
|
||||
"agent_failed", "compact_failed",
|
||||
}
|
||||
if reason in _NO_RETRY_REASONS:
|
||||
reason_human = _REASON_MAP.get(reason, _REASON_MAP.get("_default", ("未知原因", lambda d: "")))[0]
|
||||
return f"无法重试({reason_human})"
|
||||
|
||||
count = (detail or {}).get("count", 0)
|
||||
fallback_count = (detail or {}).get("fallback_count", 0)
|
||||
|
||||
if count > 0:
|
||||
return f"已自动重试 {count} 次"
|
||||
if fallback_count > 0:
|
||||
return f"已自动重试 {fallback_count} 次(fallback)"
|
||||
return "系统已尝试恢复,但仍失败"
|
||||
|
||||
|
||||
# reason_raw → (reason_human_readable, detail_format_fn)
|
||||
_REASON_MAP: Dict[str, tuple] = {
|
||||
"no_reply_found": ("收件人未回复(Agent 未能识别或处理此邮件)", lambda d: ""),
|
||||
"crashed": ("处理时进程崩溃", lambda d: f"stderr: {_extract_stderr(d)}" if _extract_stderr(d) else "无 stderr 输出"),
|
||||
"max_crash_count": ("连续崩溃达上限", lambda d: f"崩溃 {d.get('count', '?')} 次"),
|
||||
"max_retries": ("续杯耗尽(已自动重试)", lambda d: f"重试 {d.get('count', '?')} 次"),
|
||||
"max_api_retry_count": ("API 连续失败达上限", lambda d: f"API 重试 {d.get('count', '?')} 次"),
|
||||
"max_monitor_timeouts": (
|
||||
"处理超时达上限",
|
||||
lambda d: f"超时 {d.get('count', '?')} 次,"
|
||||
f"共约 {d.get('elapsed_seconds', 0) // 60} 分钟"),
|
||||
"gateway_timeout": ("Agent 执行超时(已续杯重试)", lambda d: ""),
|
||||
"session_stuck": ("会话假死(lock PID 死亡)", lambda d: f"假死 {d.get('stuck_count', '?')} 次"),
|
||||
"revive_failed": ("会话恢复失败", lambda d: f"假死 {d.get('stuck_count', '?')} 次"),
|
||||
"auth_failed": ("Agent 认证失败(配置问题)", lambda d: f"stderr: {_extract_stderr(d)}" if _extract_stderr(d) else ""),
|
||||
"fallback_exhausted": (
|
||||
"主模型和备用模型均失败",
|
||||
lambda d: f"fallback {d.get('fallback_count', '?')} 次,"
|
||||
f"原因: {d.get('fallback_reason', '未知')}"),
|
||||
"agent_failed": ("收件人主动标记失败", lambda d: ""),
|
||||
"compact_failed": ("上下文压缩失败", lambda d: f"stderr: {_extract_stderr(d)}" if _extract_stderr(d) else ""),
|
||||
"compact_hanging": ("上下文压缩长时间未完成", lambda d: ""),
|
||||
"compact_interrupted": ("上下文压缩被中断(已自动重试)", lambda d: ""),
|
||||
"gateway_unreachable": (
|
||||
"Gateway 不可达(已自动重试)",
|
||||
lambda d: f"stderr: {_extract_stderr(d)}"
|
||||
if _extract_stderr(d) else ""),
|
||||
"lock_conflict": ("会话锁冲突(已自动重试)", lambda d: ""),
|
||||
"max_retry_count": ("重试耗尽", lambda d: f"重试 {d.get('count', '?')} 次"),
|
||||
"max_lock_retry_count": ("锁冲突重试耗尽", lambda d: f"重试 {d.get('count', '?')} 次"),
|
||||
"max_connect_retry_count": ("连接重试耗尽", lambda d: f"重试 {d.get('count', '?')} 次"),
|
||||
"_default": ("未知原因", lambda d: f"stderr: {_extract_stderr(d)}" if _extract_stderr(d) else ""),
|
||||
}
|
||||
|
||||
# 常见失败原因参考(AI Native:提供知识库让收件 AI 自行判断)
|
||||
_REASON_REFERENCE = """常见失败原因参考:
|
||||
• no_reply_found:收件人未回复(Agent 未能识别或处理此邮件)
|
||||
• crashed / max_crash_count:收件人处理时进程崩溃(已自动重试 3 次)
|
||||
• max_retries:续杯耗尽(已自动重试 3 次,共约 34 分钟)
|
||||
• max_api_retry_count:API 连续失败达上限(rate_limit/500/503)
|
||||
• max_monitor_timeouts:处理超时达上限(共约 31.5 分钟)
|
||||
• gateway_timeout:Agent 执行超时(已续杯重试)
|
||||
• session_stuck:Agent 会话假死(lock PID 死亡,revive 失败)
|
||||
• revive_failed:会话假死后恢复失败
|
||||
• auth_failed:Agent 认证失败(配置问题)
|
||||
• fallback_exhausted:主模型和备用模型均失败
|
||||
• agent_failed:收件人主动标记失败
|
||||
• compact_failed:上下文压缩失败
|
||||
• compact_hanging:上下文压缩长时间未完成(等待超 31.5 分钟)
|
||||
• compact_interrupted:上下文压缩被中断(已自动重试 3 次)
|
||||
• gateway_unreachable:Gateway 不可达(已自动重试 3 次)
|
||||
• lock_conflict:会话锁冲突(已自动重试 3 次)
|
||||
• 其他:建议排查系统日志"""
|
||||
|
||||
|
||||
def _build_notify_text(title: str, to_agent: str, reason: str,
|
||||
detail: Optional[dict] = None) -> str:
|
||||
"""构建通知正文(v2.0 AI Native)"""
|
||||
reason_human, detail_fn = _REASON_MAP.get(reason, _REASON_MAP["_default"])
|
||||
detail_info = detail_fn(detail or {})
|
||||
|
||||
retry_info = _fmt_retry_info(reason, detail or {})
|
||||
|
||||
lines = [
|
||||
"邮件投递失败通知",
|
||||
"",
|
||||
f"📧 原始邮件:「{title}」",
|
||||
f"👤 收件人:{to_agent}",
|
||||
f"❌ 失败原因:{reason_human}({reason})",
|
||||
f"📊 重试情况:{retry_info}",
|
||||
]
|
||||
|
||||
if detail_info:
|
||||
lines.append("📋 上下文信息:")
|
||||
lines.append(f" {detail_info}")
|
||||
|
||||
lines.append("")
|
||||
lines.append(_REASON_REFERENCE)
|
||||
lines.append("")
|
||||
lines.append("——系统自动通知")
|
||||
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
def _is_mail_project(db_path: Path) -> bool:
|
||||
@@ -43,7 +140,7 @@ def notify_mail_failed(db_path: Path, original_mail_id: str,
|
||||
"""Mail 失败后以 system 身份给发件人发通知邮件
|
||||
|
||||
直接通过 Blackboard 创建 Task,不走 HTTP API。
|
||||
防递归:检查原邮件 must_hives.system_notify,为 true 则跳过。
|
||||
防递归:检查原邮件 must_haves.system_notify,为 true 则跳过。
|
||||
发件人不是有效 Agent(如 system)→ 通知庞统代处理,避免广播风暴。
|
||||
"""
|
||||
try:
|
||||
@@ -83,12 +180,8 @@ def notify_mail_failed(db_path: Path, original_mail_id: str,
|
||||
original_mail_id, from_agent)
|
||||
target_agent = "pangtong-fujunshi"
|
||||
|
||||
# 构造通知正文
|
||||
text = _NOTIFY_TEMPLATE.format(
|
||||
title=title,
|
||||
to_agent=to_agent,
|
||||
reason=reason,
|
||||
)
|
||||
# 构造通知正文(v2.0 AI Native)
|
||||
text = _build_notify_text(title, to_agent, reason, detail)
|
||||
|
||||
# 创建通知邮件 Task
|
||||
notify_id = f"mail-{int(datetime.now().timestamp() * 1000)}"
|
||||
|
||||
@@ -845,6 +845,8 @@ curl -X POST http://{api_host}:{api_port}/api/projects/{project_id}/tasks/{task_
|
||||
cls.get("retry_field", "retry_count")
|
||||
)
|
||||
elif outcome == "api_error":
|
||||
# A9: [DEPRECATED] api_error 已改为 should_retry=True 走续杯路径。
|
||||
# 此分支理论上不再命中,保留作为安全兜底。
|
||||
# A9: 429/API 错误 → release counter(on_complete)+ 推回 pending + 冷却
|
||||
# 有上限:api_retry_count 累计达 max_retries 则标 failed
|
||||
await self._do_on_complete_async(on_complete, agent_id, outcome)
|
||||
@@ -1842,7 +1844,8 @@ curl -X POST http://{api_host}:{api_port}/api/projects/{project_id}/tasks/{task_
|
||||
"retry_field": "retry_count", "cooldown_seconds": 60}
|
||||
if any(kw in stderr_lower for kw in [
|
||||
"rate_limit", "500", "503", "api error"]):
|
||||
return {"outcome": "api_error", "should_retry": False}
|
||||
return {"outcome": "api_error", "should_retry": True,
|
||||
"retry_field": "retry_count", "cooldown_seconds": 60}
|
||||
if any(kw in stderr_lower for kw in [
|
||||
"compaction-diag", "context-overflow"]):
|
||||
return {"outcome": "compact_failed", "should_retry": False}
|
||||
|
||||
@@ -165,14 +165,16 @@ class TestClassifyErrorApi:
|
||||
1, {"status": "error"}, "rate_limit exceeded", None
|
||||
)
|
||||
assert result["outcome"] == "api_error"
|
||||
assert result["should_retry"] is False
|
||||
assert result["should_retry"] is True
|
||||
assert result["cooldown_seconds"] == 60
|
||||
|
||||
def test_stderr_500(self):
|
||||
result = Spawner._classify_outcome(
|
||||
1, {"status": "error"}, "HTTP 500 Internal Server Error", None
|
||||
)
|
||||
assert result["outcome"] == "api_error"
|
||||
assert result["should_retry"] is False
|
||||
assert result["should_retry"] is True
|
||||
assert result["cooldown_seconds"] == 60
|
||||
|
||||
|
||||
class TestClassifyErrorCompact:
|
||||
|
||||
Reference in New Issue
Block a user