1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
|
# POC Evaluation Sheet (SigCLIP Baseline)
Use `poc_eval_sheet_template.csv` as the single source for decision testing.
## Test setup
- Target size: `N = 30` taps across varied scenes/materials.
- Baseline model first: `SigCLIP` in `model_name`.
- One row = one tapped object attempt.
- Keep `status` as `ok` or `fail` for every row.
## Required columns
- Identity: `sample_id`, `image_id`, `image_path`
- Interaction: `tap_x`, `tap_y`
- Ground truth: `ground_truth_object`, `ground_truth_material`
- Model output: `pred_object`, `pred_material_top1`, `pred_material_top3`, `pred_material_conf_top1`, `model_name`
- Segmentation quality: `segmentation_accepted` (`1`/`0`)
- Human correction: `user_edited_material` (`1`/`0`), `user_final_material`, `edit_reason`
- Runtime: `end_to_end_time_sec`, `segmentation_time_sec`, `classification_time_sec`
- Reliability: `status`, `error_type`
- Artifacts: `mask_area_px`, `bbox_xywh`, `passport_json_path`
## Decision metrics and thresholds
- Segmentation acceptance rate: `>= 85%`
- Material top-1 accuracy: `>= 70%`
- Material top-3 hit rate: `>= 90%`
- Manual edit rate: `<= 35%`
- Median end-to-end time: `<= 12 sec`
- Passport schema validity: `100%` (all `status = ok`, all JSON files valid)
## Google Sheets formulas
Assume headers are in row `1` and data rows are `2:1000`.
- Segmentation acceptance rate
`=AVERAGE(N2:N1000)`
- Top-1 accuracy (add helper col `Y`: top1_correct)
`Y2: =IF(H2="",,IF(H2=G2,1,0))` then fill down
Metric: `=AVERAGE(Y2:Y1000)`
- Top-3 hit rate (add helper col `Z`: top3_hit; top-3 delimiter is `|`)
`Z2: =IF(I2="",,IF(ISNUMBER(SEARCH(G2,I2)),1,0))` then fill down
Metric: `=AVERAGE(Z2:Z1000)`
- Manual edit rate
`=AVERAGE(O2:O1000)`
- Median end-to-end time
`=MEDIAN(Q2:Q1000)`
- Failure rate (`status != ok`)
`=COUNTIF(T2:T1000,"<>ok")/COUNTA(T2:T1000)`
## Colab/Pandas metric cell
```python
import pandas as pd
df = pd.read_csv("/content/poc_eval_sheet_template.csv")
df = df[df["status"].notna()]
df["top1_correct"] = (df["pred_material_top1"] == df["ground_truth_material"]).astype(int)
df["top3_hit"] = df.apply(
lambda r: int(str(r["ground_truth_material"]) in str(r["pred_material_top3"]).split("|")),
axis=1,
)
metrics = {
"n_samples": len(df),
"segmentation_acceptance_rate": df["segmentation_accepted"].mean(),
"top1_accuracy": df["top1_correct"].mean(),
"top3_hit_rate": df["top3_hit"].mean(),
"manual_edit_rate": df["user_edited_material"].mean(),
"median_e2e_time_sec": df["end_to_end_time_sec"].median(),
"failure_rate": (df["status"] != "ok").mean(),
}
thresholds = {
"segmentation_acceptance_rate": 0.85,
"top1_accuracy": 0.70,
"top3_hit_rate": 0.90,
"manual_edit_rate_max": 0.35,
"median_e2e_time_sec_max": 12.0,
}
print(metrics)
print("go_no_go:", (
metrics["segmentation_acceptance_rate"] >= thresholds["segmentation_acceptance_rate"]
and metrics["top1_accuracy"] >= thresholds["top1_accuracy"]
and metrics["top3_hit_rate"] >= thresholds["top3_hit_rate"]
and metrics["manual_edit_rate"] <= thresholds["manual_edit_rate_max"]
and metrics["median_e2e_time_sec"] <= thresholds["median_e2e_time_sec_max"]
and metrics["failure_rate"] == 0
))
```
## SigCLIP vs DINOv3 promotion rule
Promote `DINOv3` only if, on the same dataset, it delivers either:
- `+5` percentage points top-1 accuracy, or
- `-10` percentage points manual edit rate,
while keeping median end-to-end time increase within `+20%`.
|
