feat(review): agent 侧虚拟工具 inspect_region —— 按可读分辨率放大指定区域
解决三个缺口:① 放大没复用标定/瓦片数学(LLM 一次 zoom 只出一张,宽区域仍读不清);
②「看清没」由 VLM 自评(易过度自信/幻觉);③ 只能按 handle / 手算 bbox 指认,
图上看到但没 handle 的东西指认不了。
核心思路:把「放大」升级成一个语义化工具——LLM 只指认「要看哪里」,由 agent 复用
tiling.py 的标定数学算出该切几张、每张覆盖多大,用构造保证可读。
新增 dwg_review/inspect.py(agent 侧,不编 DLL):
- TOOL_SCHEMA:inspect_region(target, purpose);target 三形态 {handle} / {window} /
{label,cx,cy,span},purpose = text(8px) / line(6px)。
- InspectState:单图框内「截图标签 → 世界可见范围」登记表,支持逐级下钻
(返回的 inspect-N 可继续作为 label 指认)。
- resolve_target / _point_box:label+cx+cy → WCS。cy 是图像坐标(上→下),
世界 y 必须翻转(y = y2 - cy*H)——单测专门钉住。
- plan_tiles:复用 world_per_px + tiles_for_box + subdivide;超 INSPECT_MAX_TILES(4)
则硬失败并给出「缩小 target / 分多次请求」的指引(不静默降级);拿不到比例/像素
则退回单张并如实告知。
关键几何:CAD 的 _.ZOOM _W 是等比适配(不拉伸),故「截图上的归一化坐标」必须按
max(W/ew, H/eh) 反算真实可见范围(新增 tiling.visible_box)——直接用送进去的窗口会偏。
zoom_to_window 返回的 view 是「加过 margin 的窗口」,不是可见范围,不能混用。
证据层配套:evidence_model.make_image(..., window=) 记录每张图的可见世界范围;
capture_images 用 view_px(overall 图原始像素)+ visible_box 逐张登记。
子循环接入:_run_readonly_tool 在只读白名单校验之前拦截 inspect_region(它不在
READ_ONLY_TOOLS 里,内部只调白名单内的只读 MCP 工具);_review_tools 启用时用
inspect_region 取代三个裸视口工具;inspect_region 计入 _VIEW_TOOLS → 结束复位视口。
配置:INSPECT_ENABLED(True) / INSPECT_MAX_TILES(4)。
端到端实测(真实图纸 1 图框):取证与审查两阶段均在最大化态、结束精确还原(4 项 PASS);
LLM 的 11 次取证调用里 2 次是 inspect_region,且都显式给 window + purpose:"text"
(x[20000,115000] y[-6500,3000],即 A0 图框底部横条=标题栏所在)。
验收口径达成:报告报出标题栏乱码 ???????????????,而 get_text_content 对全图返回 0 条文字
(本图文字全在块/自定义实体内)→ 块内文字是被视觉放大读出来的。
pytest 343 passed。
This commit is contained in:
1 parent
3bc64b6c3e
commit
b757892c67
18 files changed
+1003
-45
No files matched your search
+15
-7
@@ -53,13 +53,21 @@ def build_system_prompt(checklist, *, allow_tools=True):
|
||||
|
||||
if allow_tools:
|
||||
lines += [
|
||||
"你可以调用**只读**取证工具进一步确认细节;**不要凭猜测下结论**:",
|
||||
" - **截图里的文字/标注/细节看不清时**:用 `zoom_to_window` 放大该区域"
|
||||
"(窗口坐标用图框 bbox 或实体 bbox 计算,可适当留边距),"
|
||||
"再 `get_viewport_screenshot` 复核;已知 handle 时也可直接 `zoom_to_entity` 放大该实体。",
|
||||
" - 结构化数据不足以判断时:用 `get_entity_detail` / `find_duplicate_entities` /"
|
||||
" `find_zero_length` / `get_layers` 等查证后再下结论。",
|
||||
" - 放大截图是**复核手段**:确认清楚后即写入结论,不要为看图而反复放大。",
|
||||
"你可以调用**只读**取证工具进一步确认细节;**不要凭猜测下结论**。",
|
||||
"",
|
||||
"**看不清某处时,调用 `inspect_region` 指认位置**,它会自动按可读分辨率放大并返回清晰截图。",
|
||||
"`target` 三种指认方式任选其一:",
|
||||
' - `{"handle": "3A4"}` —— 已知实体句柄;',
|
||||
' - `{"label": "tile-3", "cx": 0.4, "cy": 0.6, "span": 0.2}` —— 某张已给截图上的一个位置',
|
||||
" (`cx` 从左到右、`cy` **从上到下**,均为 0~1 归一化;`span` 为要看的范围占该截图的比例,默认 0.25);",
|
||||
' - `{"window": {"min_x": …, "min_y": …, "max_x": …, "max_y": …}}` —— 直接给世界坐标框。',
|
||||
'`purpose` 选 `"text"`(要读文字/标注,默认)或 `"line"`(只看走向/连接,覆盖更广)。',
|
||||
"返回的截图会带 `inspect-N` 标签,可继续作为 `label` 指认更细的位置(逐级下钻)。",
|
||||
"**你不需要判断截图够不够清楚**——`inspect_region` 会保证返回的图达到可读分辨率;",
|
||||
"若它提示区域过大,请缩小 `target`(或用 `span` 缩小范围)后重试。",
|
||||
"",
|
||||
"结构化数据不足以判断时:用 `get_entity_detail` / `find_duplicate_entities` /",
|
||||
"`find_zero_length` / `get_layers` 等查证后再下结论。",
|
||||
"",
|
||||
]
|
||||
|
||||
|
||||
@@ -94,6 +94,11 @@ REVIEW_MAX_ITERS = int(os.environ.get("REVIEW_MAX_ITERS", "4"))
|
||||
# 实测:提示词鼓励「看不清→放大重看」后,模型一轮内会发多次 zoom+截图,
|
||||
# 上限过小会让它白拍(拍了却看不到)。6 可覆盖多数复核需求。
|
||||
REVIEW_OBS_IMAGES = int(os.environ.get("REVIEW_OBS_IMAGES", "6"))
|
||||
# 是否启用 agent 侧虚拟工具 inspect_region(按可读分辨率放大指定区域)。
|
||||
# 关闭则退回「LLM 自己组合 zoom_to_window + get_viewport_screenshot」的旧行为。
|
||||
INSPECT_ENABLED = _read_bool("INSPECT_ENABLED", True)
|
||||
# inspect_region 单次请求最多返回几张瓦片(超出则中止并要求缩小 target)
|
||||
INSPECT_MAX_TILES = int(os.environ.get("INSPECT_MAX_TILES", "4"))
|
||||
# 报告输出目录
|
||||
REPORT_DIR = _resolve_path(os.environ.get("REPORT_DIR", "reports"))
|
||||
# 批注回写使用的图层
|
||||
|
||||
@@ -16,9 +16,16 @@ SOURCE_IMAGE = "image"
|
||||
STRUCTURED_SOURCES = ("entities", "texts", "dimensions", "blocks", "polylines")
|
||||
|
||||
|
||||
def make_image(b64, mime="image/png", label=None):
|
||||
"""构造 images 列表元素。"""
|
||||
return {"b64": b64, "mime": mime or "image/png", "label": label}
|
||||
def make_image(b64, mime="image/png", label=None, window=None):
|
||||
"""构造 images 列表元素。
|
||||
|
||||
`window` 为该截图**实际可见的世界范围** `(min_x,min_y,max_x,max_y)`(可选)。
|
||||
审查阶段的 `inspect_region` 靠它把「截图上的归一化坐标」映射回 WCS(逐级下钻)。
|
||||
"""
|
||||
img = {"b64": b64, "mime": mime or "image/png", "label": label}
|
||||
if window is not None:
|
||||
img["window"] = window
|
||||
return img
|
||||
|
||||
|
||||
def source_type_of(container):
|
||||
@@ -37,7 +44,7 @@ def evidence_images(evidence):
|
||||
"""规范化证据里的图片列表(单点归一)。
|
||||
|
||||
优先读 `images`;否则回退旧的 `image_b64` / `image_mime` 单图字段。
|
||||
统一返回 `[{"b64","mime","label"}]`,过滤掉空 b64。
|
||||
统一返回 `[{"b64","mime","label","window"}]`,过滤掉空 b64。
|
||||
"""
|
||||
if not isinstance(evidence, dict):
|
||||
return []
|
||||
@@ -55,6 +62,7 @@ def evidence_images(evidence):
|
||||
"b64": b64,
|
||||
"mime": item.get("mime") or "image/png",
|
||||
"label": item.get("label"),
|
||||
"window": item.get("window"),
|
||||
}
|
||||
)
|
||||
if images:
|
||||
@@ -67,6 +75,7 @@ def evidence_images(evidence):
|
||||
"b64": b64,
|
||||
"mime": evidence.get("image_mime") or "image/png",
|
||||
"label": None,
|
||||
"window": None,
|
||||
}
|
||||
]
|
||||
return []
|
||||
|
||||
@@ -0,0 +1,288 @@
|
||||
"""agent 侧虚拟工具 `inspect_region`:按**可读分辨率**放大指定区域。
|
||||
|
||||
动机(A 阶段收尾):
|
||||
让 LLM 自己组合 `zoom_to_window` + `get_viewport_screenshot` 时——
|
||||
① 单张截图可能仍读不清(宽区域下采样后字太小);
|
||||
② 「看清没」只能由 VLM 自评(易过度自信 / 幻觉)。
|
||||
本模块把「放大」升级为**一个语义化工具**:LLM 只指认「要看哪里」,由 agent 复用
|
||||
`tiling.py` 的标定数学算出该切几张、每张覆盖多大,**用构造保证可读**。
|
||||
|
||||
不变量:
|
||||
- 只调用**白名单内**的只读 MCP 工具(`zoom_to_window` / `get_viewport_screenshot`),
|
||||
不写库、不改图元、不动图层;
|
||||
- 会改动视口 → 调用方(`phases/review.py`)负责在该图框审查结束后复位。
|
||||
|
||||
坐标约定:
|
||||
- 世界坐标:x 向右、y **向上**;
|
||||
- 截图内归一化坐标:cx 左→右、cy **上→下**(图像坐标,左上原点)——
|
||||
`_point_box` 会翻转 y,这是最容易写错的一处。
|
||||
"""
|
||||
|
||||
from . import config, evidence_model, tiling
|
||||
from .llm_client import LlmClient, downsample_image_b64, image_size_b64
|
||||
from .phases.evidence import zoom_and_shot
|
||||
|
||||
TOOL_NAME = "inspect_region"
|
||||
|
||||
# zoom 时四周留边比例(与 `evidence.zoom_and_shot` 的默认 margin 一致)
|
||||
_ZOOM_MARGIN = 0.02
|
||||
|
||||
# 送进瓦片规划前先把目标框外扩一点,避免把边界上的字切掉
|
||||
_EDGE_PAD = 0.05
|
||||
|
||||
TOOL_SCHEMA = {
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": TOOL_NAME,
|
||||
"description": (
|
||||
"放大并返回指定区域的**可读分辨率**截图。当你看不清已给截图里的文字/标注/细节时调用;"
|
||||
"会自动计算需要几张、每张覆盖多大,**无需你判断清晰度**。"
|
||||
"target 三选一:handle(实体句柄)/ window(世界坐标框)/ "
|
||||
"label+cx+cy(某张已给截图上的归一化位置)。"
|
||||
),
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {
|
||||
"target": {
|
||||
"type": "object",
|
||||
"description": "要放大的位置(三种形态任选其一)",
|
||||
"properties": {
|
||||
"handle": {
|
||||
"type": "string",
|
||||
"description": '实体句柄,如 "3A4"',
|
||||
},
|
||||
"window": {
|
||||
"type": "object",
|
||||
"description": "世界坐标框",
|
||||
"properties": {
|
||||
"min_x": {"type": "number"},
|
||||
"min_y": {"type": "number"},
|
||||
"max_x": {"type": "number"},
|
||||
"max_y": {"type": "number"},
|
||||
},
|
||||
"required": ["min_x", "min_y", "max_x", "max_y"],
|
||||
},
|
||||
"label": {
|
||||
"type": "string",
|
||||
"description": '已给截图的标签,如 "tile-3" 或之前返回的 "inspect-1"',
|
||||
},
|
||||
"cx": {
|
||||
"type": "number",
|
||||
"description": "该截图内的归一化横坐标 0~1(左→右)",
|
||||
},
|
||||
"cy": {
|
||||
"type": "number",
|
||||
"description": "该截图内的归一化纵坐标 0~1(上→下)",
|
||||
},
|
||||
"span": {
|
||||
"type": "number",
|
||||
"description": "以该点为中心的视野跨度(占该截图宽/高的比例),默认 0.25",
|
||||
},
|
||||
},
|
||||
},
|
||||
"purpose": {
|
||||
"type": "string",
|
||||
"enum": ["text", "line"],
|
||||
"description": "放大目的:text=读文字/标注(默认,更细);line=看走向/连接(更粗)",
|
||||
},
|
||||
},
|
||||
"required": ["target"],
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
class InspectState:
|
||||
"""单图框内的「截图标签 → 世界可见范围」登记表。
|
||||
|
||||
初始登记取证阶段给的 `overall` / `tile-*`(它们的 `window` 字段),
|
||||
之后每次 `inspect_region` 的新截图再追加 `inspect-1`、`inspect-2`…,
|
||||
于是 LLM 可以用**上一轮返回的标签**继续下钻。
|
||||
"""
|
||||
|
||||
def __init__(self, evidence):
|
||||
self.labels = {}
|
||||
for img in evidence_model.evidence_images(evidence):
|
||||
label = img.get("label")
|
||||
box = tiling.norm_box(img.get("window"))
|
||||
if label and box is not None:
|
||||
self.labels[label] = box
|
||||
self.seq = 0
|
||||
|
||||
def register(self, box):
|
||||
"""登记一张新截图,返回它的标签。"""
|
||||
self.seq += 1
|
||||
label = f"inspect-{self.seq}"
|
||||
self.labels[label] = box
|
||||
return label
|
||||
|
||||
|
||||
def _text(s):
|
||||
return {"type": "text", "text": s}
|
||||
|
||||
|
||||
def readable_px(purpose):
|
||||
"""按用途取 VLM 可读下限(像素)。"""
|
||||
return config.TEXT_READABLE_PX if purpose == "text" else config.LINE_READABLE_PX
|
||||
|
||||
|
||||
def _view_px(evidence):
|
||||
"""绘图区**原始像素**(取 overall 图;无图/无 Pillow → None)。"""
|
||||
img = evidence_model.primary_image(evidence)
|
||||
if not img:
|
||||
return None
|
||||
return image_size_b64(img.get("b64"))
|
||||
|
||||
|
||||
def _bbox_for_handle(evidence, handle):
|
||||
"""在已采集的结构化证据里按 handle 找 bbox;找不到返回 None。"""
|
||||
want = handle.strip().upper()
|
||||
for key in evidence_model.STRUCTURED_SOURCES:
|
||||
env = evidence.get(key)
|
||||
items = env.get("items") if isinstance(env, dict) else None
|
||||
for item in items or []:
|
||||
if not isinstance(item, dict):
|
||||
continue
|
||||
got = item.get("handle")
|
||||
if isinstance(got, str) and got.strip().upper() == want:
|
||||
box = tiling.norm_box(item.get("bbox"))
|
||||
if box is not None:
|
||||
return box
|
||||
return None
|
||||
|
||||
|
||||
def _point_box(base, cx, cy, span):
|
||||
"""截图内归一化点 → 世界框。
|
||||
|
||||
`cx` 左→右;`cy` **上→下**(图像坐标),故世界 y 需翻转:`y = y2 - cy * H`。
|
||||
"""
|
||||
x1, y1, x2, y2 = base
|
||||
w, h = x2 - x1, y2 - y1
|
||||
cx = min(max(float(cx), 0.0), 1.0)
|
||||
cy = min(max(float(cy), 0.0), 1.0)
|
||||
span = min(max(float(span), 0.02), 1.0)
|
||||
px = x1 + cx * w
|
||||
py = y2 - cy * h
|
||||
hw, hh = w * span / 2.0, h * span / 2.0
|
||||
return (px - hw, py - hh, px + hw, py + hh)
|
||||
|
||||
|
||||
def resolve_target(evidence, target, state):
|
||||
"""解析 `target` → `(世界框, 人类可读说明)`;失败返回 `(None, 错误说明)`。"""
|
||||
if not isinstance(target, dict):
|
||||
return None, "target 必须是对象"
|
||||
|
||||
handle = target.get("handle")
|
||||
if isinstance(handle, str) and handle.strip():
|
||||
box = _bbox_for_handle(evidence, handle)
|
||||
if box is None:
|
||||
return None, (
|
||||
f"未在已采集证据里找到 handle {handle} 的包围盒;"
|
||||
f"请改用 window 或 label 指认位置"
|
||||
)
|
||||
return box, f"实体 {handle} 的包围盒"
|
||||
|
||||
if target.get("window") is not None:
|
||||
box = tiling.window_to_box(target.get("window"))
|
||||
if box is None:
|
||||
return None, "window 非法:需要 min_x/min_y/max_x/max_y 四个数值"
|
||||
return box, "给定的世界坐标框"
|
||||
|
||||
label = target.get("label")
|
||||
if isinstance(label, str) and label.strip():
|
||||
label = label.strip()
|
||||
base = state.labels.get(label)
|
||||
if base is None:
|
||||
known = "、".join(sorted(state.labels)) or "(无)"
|
||||
return None, f"未知截图标签 {label};当前可用:{known}"
|
||||
try:
|
||||
cx = float(target.get("cx", 0.5))
|
||||
cy = float(target.get("cy", 0.5))
|
||||
span = float(target.get("span", 0.25))
|
||||
except (TypeError, ValueError):
|
||||
return None, "cx / cy / span 必须是数值"
|
||||
return (
|
||||
_point_box(base, cx, cy, span),
|
||||
f"截图 {label} 上 ({cx:.2f}, {cy:.2f}) 附近",
|
||||
)
|
||||
|
||||
return None, "target 需给出 handle / window / label 之一"
|
||||
|
||||
|
||||
def plan_tiles(box, scale, purpose, view_px, *, max_tiles, overlap, margin):
|
||||
"""把目标框规划成若干 zoom 窗口。
|
||||
|
||||
返回 `(windows, note)`;`windows is None` 表示**硬失败**(区域过大,已中止)。
|
||||
拿不到比例 / 绘图区像素时**退回单张**并如实告知(无法保证可读)。
|
||||
"""
|
||||
s = tiling.world_per_px(scale, config.TEXT_HEIGHT_MM, readable_px(purpose))
|
||||
eff = tiling.effective_px(view_px, config.IMAGE_MAX_EDGE)
|
||||
padded = tiling.expand_box(box, _EDGE_PAD)
|
||||
if padded is None or padded[2] <= padded[0] or padded[3] <= padded[1]:
|
||||
return None, "目标区域退化(宽或高为 0),无法放大"
|
||||
|
||||
if s <= 0 or eff[0] <= 0 or eff[1] <= 0:
|
||||
win = tiling.box_to_window(padded)
|
||||
return [win], "(拿不到出图比例或绘图区像素,已退回单张;文字可能仍读不清)"
|
||||
|
||||
cols, rows = tiling.tiles_for_box(padded, s * eff[0], s * eff[1], overlap)
|
||||
need = cols * rows
|
||||
if need > max_tiles:
|
||||
return None, (
|
||||
f"该区域按可读分辨率需 {need} 张截图,超过单次上限 {max_tiles};"
|
||||
f"请缩小 target(例如用 label + cx + cy + span 指定更小范围)或分多次请求"
|
||||
)
|
||||
wins = tiling.subdivide(padded, cols, rows, overlap, frame_win=None)
|
||||
if not wins:
|
||||
return None, "目标区域退化,无法放大"
|
||||
return wins, f"按 {purpose} 判据切为 {cols}×{rows} = {need} 张"
|
||||
|
||||
|
||||
def inspect_region(ctx, evidence, args, image_budget, state):
|
||||
"""执行一次放大取证,返回 `(content, image_budget)`。"""
|
||||
args = args if isinstance(args, dict) else {}
|
||||
purpose = str(args.get("purpose") or "text").strip().lower()
|
||||
if purpose not in ("text", "line"):
|
||||
purpose = "text"
|
||||
|
||||
box, note = resolve_target(evidence, args.get("target"), state)
|
||||
if box is None:
|
||||
return [_text(f"inspect_region 未执行:{note}")], image_budget
|
||||
|
||||
view_px = _view_px(evidence)
|
||||
scale = tiling.frame_scale(evidence.get("frame"))
|
||||
wins, plan_note = plan_tiles(
|
||||
box, scale, purpose, view_px,
|
||||
max_tiles=config.INSPECT_MAX_TILES,
|
||||
overlap=config.TILE_OVERLAP,
|
||||
margin=_ZOOM_MARGIN,
|
||||
)
|
||||
if wins is None:
|
||||
return [_text(f"inspect_region 未执行:{plan_note}")], image_budget
|
||||
|
||||
content = [_text(f"inspect_region:{note};{plan_note}。")]
|
||||
taken = 0
|
||||
for i, win in enumerate(wins, 1):
|
||||
if image_budget[0] <= 0:
|
||||
content.append(_text(
|
||||
f"(截图预算已用完,剩余 {len(wins) - i + 1} 张未拍;"
|
||||
f"请基于已有信息给出结论,不要再请求截图。)"
|
||||
))
|
||||
break
|
||||
shot = zoom_and_shot(ctx, win, margin=_ZOOM_MARGIN)
|
||||
if not shot:
|
||||
continue
|
||||
image_budget[0] -= 1
|
||||
visible = tiling.visible_box(tiling.window_to_box(win), view_px, _ZOOM_MARGIN)
|
||||
label = state.register(visible)
|
||||
content.append(_text(
|
||||
f"(放大截图 {label}:可见世界范围 "
|
||||
f"x[{visible[0]:.1f}, {visible[2]:.1f}] y[{visible[1]:.1f}, {visible[3]:.1f}])"
|
||||
))
|
||||
b64, mime = downsample_image_b64(shot[0], shot[1])
|
||||
content.append(LlmClient.image_content(b64, mime))
|
||||
taken += 1
|
||||
|
||||
if taken == 0:
|
||||
content.append(_text("(未能取得截图。)"))
|
||||
return content, image_budget
|
||||
@@ -20,6 +20,9 @@ _LABEL_OVERALL = "overall"
|
||||
# 绘图区最大化用的工具名(旧版插件没有此工具 → 静默降级)
|
||||
_TOOL_DRAWING_AREA = "set_drawing_area"
|
||||
|
||||
# zoom 时四周留边比例(与 `zoom_to_window` 的默认 margin 一致)
|
||||
_ZOOM_MARGIN = 0.02
|
||||
|
||||
|
||||
@contextlib.contextmanager
|
||||
def maximized_drawing_area(ctx):
|
||||
@@ -95,7 +98,7 @@ def sub_windows(window, tiles):
|
||||
return out
|
||||
|
||||
|
||||
def _zoom_and_shot(ctx, window, margin=0.02):
|
||||
def zoom_and_shot(ctx, window, margin=_ZOOM_MARGIN):
|
||||
"""缩放到窗口、等待渲染、截一张图,返回 (b64, mime) 或 None。"""
|
||||
ctx.tools.call("zoom_to_window", dict(window, margin=margin))
|
||||
time.sleep(config.RENDER_WAIT)
|
||||
@@ -209,6 +212,8 @@ def capture_images(ctx, window, *, tiles=0, max_images=None, frame=None,
|
||||
- 局部窗口优先由 `_plan_tiles` 按**实体分布**给出(空白区不出图);
|
||||
不可用时退回 `sub_windows(window, tiles)`,**同样按内容过滤**空白象限
|
||||
(过滤后为空则退回未过滤集合,保证不漏拍)。
|
||||
- 每张图登记其**实际可见世界范围**(`window` 字段)——审查阶段的 `inspect_region`
|
||||
靠它把「截图上的归一化坐标」映射回 WCS(逐级下钻)。
|
||||
- 结束时**复位到整体视口**,保证后续流程从整幅视图起手。
|
||||
- 受 max_images(默认 config.EVIDENCE_MAX_IMAGES)截断。
|
||||
"""
|
||||
@@ -216,14 +221,23 @@ def capture_images(ctx, window, *, tiles=0, max_images=None, frame=None,
|
||||
max_images = config.EVIDENCE_MAX_IMAGES
|
||||
images = []
|
||||
|
||||
def _add(shot, label):
|
||||
if shot and len(images) < max_images:
|
||||
images.append(evidence_model.make_image(shot[0], shot[1], label))
|
||||
|
||||
# 整体图:既是证据,也用于测量绘图区**实际像素**(决定单张截图的世界覆盖范围)
|
||||
overall = _zoom_and_shot(ctx, window)
|
||||
overall = zoom_and_shot(ctx, window)
|
||||
view_px = image_size_b64(overall[0]) if overall else None
|
||||
_add(overall, _LABEL_OVERALL)
|
||||
|
||||
def _add(shot, label, win):
|
||||
"""登记截图;`win` 为送给 CAD 的窗口,用于反算实际可见世界范围。"""
|
||||
if shot and len(images) < max_images:
|
||||
images.append(
|
||||
evidence_model.make_image(
|
||||
shot[0], shot[1], label,
|
||||
window=tiling.visible_box(
|
||||
tiling.window_to_box(win), view_px, _ZOOM_MARGIN
|
||||
),
|
||||
)
|
||||
)
|
||||
|
||||
_add(overall, _LABEL_OVERALL, window)
|
||||
|
||||
# 内容过滤:空白区不出图。按**实体 bbox** 判定,规划与回退两条路径共用同一谓词;
|
||||
# 实体数据被截断时不可信,放弃过滤(宁可多拍也不漏)。
|
||||
@@ -237,10 +251,10 @@ def capture_images(ctx, window, *, tiles=0, max_images=None, frame=None,
|
||||
for i, sub in enumerate(regions, 1):
|
||||
if len(images) >= max_images:
|
||||
break
|
||||
_add(_zoom_and_shot(ctx, sub), f"tile-{i}")
|
||||
_add(zoom_and_shot(ctx, sub), f"tile-{i}", sub)
|
||||
|
||||
# 复位整体视口(不截图)
|
||||
ctx.tools.call("zoom_to_window", dict(window, margin=0.02))
|
||||
ctx.tools.call("zoom_to_window", dict(window, margin=_ZOOM_MARGIN))
|
||||
return images
|
||||
|
||||
|
||||
|
||||
@@ -14,7 +14,7 @@
|
||||
|
||||
import json
|
||||
|
||||
from .. import config, evidence_model
|
||||
from .. import config, evidence_model, inspect
|
||||
from ..cad_tools import READ_ONLY_TOOLS, is_write_tool
|
||||
from ..findings import parse_conclusion
|
||||
from ..llm_client import LlmClient, downsample_image_b64
|
||||
@@ -24,7 +24,13 @@ from ..mcp_client import McpError, McpToolError, result_images, result_texts
|
||||
_MAX_OBS_IMAGES = config.REVIEW_OBS_IMAGES
|
||||
|
||||
# 会改动视口的只读取证工具:用过之后须把视口复位,别把用户的 CAD 留在放大处
|
||||
_VIEW_TOOLS = frozenset({"zoom_to_window", "zoom_to_entity"})
|
||||
_VIEW_TOOLS = frozenset({"zoom_to_window", "zoom_to_entity", inspect.TOOL_NAME})
|
||||
|
||||
# 启用 inspect_region 时,这三个裸视口工具不再直接提供给 LLM(由 inspect_region 统一代理),
|
||||
# 但仍留在只读白名单里,作为兜底仍然可被调用。
|
||||
_RAW_VIEW_TOOLS = frozenset(
|
||||
{"zoom_to_window", "zoom_to_entity", "get_viewport_screenshot"}
|
||||
)
|
||||
|
||||
|
||||
def evidence_payload(evidence):
|
||||
@@ -45,8 +51,17 @@ def evidence_payload(evidence):
|
||||
}
|
||||
|
||||
|
||||
def _run_readonly_tool(ctx, name, args, image_budget):
|
||||
"""执行一个只读取证工具,返回回灌给 LLM 的 content 列表。"""
|
||||
def _run_readonly_tool(ctx, name, args, image_budget, *, evidence=None, state=None):
|
||||
"""执行一个只读取证工具,返回回灌给 LLM 的 content 列表。
|
||||
|
||||
`inspect_region` 是 **agent 侧虚拟工具**(不走 MCP),在只读白名单校验**之前**拦截:
|
||||
它内部只调白名单内的只读 MCP 工具。
|
||||
"""
|
||||
if name == inspect.TOOL_NAME:
|
||||
if not config.INSPECT_ENABLED or evidence is None or state is None:
|
||||
return [{"type": "text", "text": "inspect_region 当前不可用。"}], image_budget
|
||||
return inspect.inspect_region(ctx, evidence, args, image_budget, state)
|
||||
|
||||
if name not in READ_ONLY_TOOLS or is_write_tool(name):
|
||||
return [{"type": "text", "text": f"工具 {name} 不在只读取证白名单内,已拒绝。"}], image_budget
|
||||
|
||||
@@ -75,6 +90,19 @@ def _run_readonly_tool(ctx, name, args, image_budget):
|
||||
return content, image_budget
|
||||
|
||||
|
||||
def _review_tools(ctx):
|
||||
"""审查子循环提供给 LLM 的工具 schema。
|
||||
|
||||
启用 `inspect_region` 时,用它**取代**三个裸视口工具(`zoom_to_window` /
|
||||
`zoom_to_entity` / `get_viewport_screenshot`)——一个语义化入口比三个裸工具更好用,
|
||||
且可读性由 agent 保证;关闭时维持原样(退回旧行为)。
|
||||
"""
|
||||
if config.INSPECT_ENABLED:
|
||||
names = READ_ONLY_TOOLS - _RAW_VIEW_TOOLS
|
||||
return ctx.tools.openai_tools(names=names) + [inspect.TOOL_SCHEMA]
|
||||
return ctx.tools.openai_tools(names=READ_ONLY_TOOLS)
|
||||
|
||||
|
||||
def _reset_viewport(ctx, evidence):
|
||||
"""把视口复位到该图框的整幅视图(审查中的放大不应残留给用户)。"""
|
||||
win = evidence.get("window")
|
||||
@@ -121,8 +149,9 @@ def _review_frame(ctx, evidence, system_prompt, log, view_changed):
|
||||
{"role": "user", "content": user_content},
|
||||
]
|
||||
|
||||
tools_schema = ctx.tools.openai_tools(names=READ_ONLY_TOOLS)
|
||||
tools_schema = _review_tools(ctx)
|
||||
image_budget = [_MAX_OBS_IMAGES]
|
||||
state = inspect.InspectState(evidence)
|
||||
|
||||
for _ in range(max(0, ctx.max_iters)):
|
||||
resp = ctx.llm.chat(messages, tools=tools_schema)
|
||||
@@ -142,7 +171,9 @@ def _review_frame(ctx, evidence, system_prompt, log, view_changed):
|
||||
log(f" [取证] {name} {args if args else ''}")
|
||||
if name in _VIEW_TOOLS:
|
||||
view_changed[0] = True
|
||||
content, image_budget = _run_readonly_tool(ctx, name, args, image_budget)
|
||||
content, image_budget = _run_readonly_tool(
|
||||
ctx, name, args, image_budget, evidence=evidence, state=state
|
||||
)
|
||||
messages.append(
|
||||
{"role": "tool", "tool_call_id": tool_call.id, "content": content}
|
||||
)
|
||||
|
||||
@@ -163,6 +163,66 @@ def boxes_intersect(a, b, eps=0.0):
|
||||
)
|
||||
|
||||
|
||||
def box_to_window(box):
|
||||
"""bbox 元组 → window 对象(`zoom_to_window` 的入参);非法返回 None。"""
|
||||
b = norm_box(box)
|
||||
if b is None:
|
||||
return None
|
||||
return {"min_x": b[0], "min_y": b[1], "max_x": b[2], "max_y": b[3]}
|
||||
|
||||
|
||||
def window_to_box(win):
|
||||
"""window 对象 → bbox 元组;非法返回 None。"""
|
||||
if not isinstance(win, dict):
|
||||
return None
|
||||
return norm_box(
|
||||
[win.get("min_x"), win.get("min_y"), win.get("max_x"), win.get("max_y")]
|
||||
)
|
||||
|
||||
|
||||
def expand_box(box, frac):
|
||||
"""把 bbox 四周按 `frac` 比例外扩;非法输入返回 None。"""
|
||||
b = norm_box(box)
|
||||
if b is None:
|
||||
return None
|
||||
try:
|
||||
f = float(frac or 0.0)
|
||||
except (TypeError, ValueError):
|
||||
f = 0.0
|
||||
mx, my = (b[2] - b[0]) * f, (b[3] - b[1]) * f
|
||||
return (b[0] - mx, b[1] - my, b[2] + mx, b[3] + my)
|
||||
|
||||
|
||||
def visible_box(box, view_px, margin=0.0):
|
||||
"""`zoom_to_window(box, margin)` 之后**实际可见**的世界范围。
|
||||
|
||||
CAD 的 `_.ZOOM _W` 是**等比适配**(不拉伸):指定窗口按视口宽高比放大到刚好装下,
|
||||
多余的一维露出更多内容。故可见范围 = 以窗口中心为中心、按 `max(W/ew, H/eh)`
|
||||
放大后的矩形。`view_px` 为截图的**原始像素**(未下采样);拿不到时退回
|
||||
「窗口外扩 margin」的近似值。
|
||||
|
||||
用途:审查阶段把「截图上的归一化坐标」准确映射回 WCS(逐级下钻)。
|
||||
"""
|
||||
b = expand_box(box, margin)
|
||||
if b is None:
|
||||
return None
|
||||
if not view_px or len(view_px) != 2:
|
||||
return b
|
||||
try:
|
||||
ew, eh = int(view_px[0]), int(view_px[1])
|
||||
except (TypeError, ValueError):
|
||||
return b
|
||||
if ew <= 0 or eh <= 0:
|
||||
return b
|
||||
w, h = b[2] - b[0], b[3] - b[1]
|
||||
if w <= 0 or h <= 0:
|
||||
return b
|
||||
k = max(w / ew, h / eh)
|
||||
vw, vh = ew * k, eh * k
|
||||
cx, cy = (b[0] + b[2]) / 2.0, (b[1] + b[3]) / 2.0
|
||||
return (cx - vw / 2.0, cy - vh / 2.0, cx + vw / 2.0, cy + vh / 2.0)
|
||||
|
||||
|
||||
def cluster_text_flags(clusters, text_boxes):
|
||||
"""标记每个簇是否含文字类实体(bbox 相交即算,保守)。
|
||||
|
||||
|
||||
Reference in new issue
Block a user