# POC Evaluation Sheet (SigCLIP Baseline)

Use `poc_eval_sheet_template.csv` as the single source for decision testing.

## Test setup

- Target size: `N = 30` taps across varied scenes/materials.
- Baseline model first: `SigCLIP` in `model_name`.
- One row = one tapped object attempt.
- Keep `status` as `ok` or `fail` for every row.

## Required columns

- Identity: `sample_id`, `image_id`, `image_path`
- Interaction: `tap_x`, `tap_y`
- Ground truth: `ground_truth_object`, `ground_truth_material`
- Model output: `pred_object`, `pred_material_top1`, `pred_material_top3`, `pred_material_conf_top1`, `model_name`
- Segmentation quality: `segmentation_accepted` (`1`/`0`)
- Human correction: `user_edited_material` (`1`/`0`), `user_final_material`, `edit_reason`
- Runtime: `end_to_end_time_sec`, `segmentation_time_sec`, `classification_time_sec`
- Reliability: `status`, `error_type`
- Artifacts: `mask_area_px`, `bbox_xywh`, `passport_json_path`

## Decision metrics and thresholds

- Segmentation acceptance rate: `>= 85%`
- Material top-1 accuracy: `>= 70%`
- Material top-3 hit rate: `>= 90%`
- Manual edit rate: `<= 35%`
- Median end-to-end time: `<= 12 sec`
- Passport schema validity: `100%` (all `status = ok`, all JSON files valid)

## Google Sheets formulas

Assume headers are in row `1` and data rows are `2:1000`.

- Segmentation acceptance rate  
`=AVERAGE(N2:N1000)`

- Top-1 accuracy (add helper col `Y`: top1_correct)  
`Y2: =IF(H2="",,IF(H2=G2,1,0))` then fill down  
Metric: `=AVERAGE(Y2:Y1000)`

- Top-3 hit rate (add helper col `Z`: top3_hit; top-3 delimiter is `|`)  
`Z2: =IF(I2="",,IF(ISNUMBER(SEARCH(G2,I2)),1,0))` then fill down  
Metric: `=AVERAGE(Z2:Z1000)`

- Manual edit rate  
`=AVERAGE(O2:O1000)`

- Median end-to-end time  
`=MEDIAN(Q2:Q1000)`

- Failure rate (`status != ok`)  
`=COUNTIF(T2:T1000,"<>ok")/COUNTA(T2:T1000)`

## Colab/Pandas metric cell

```python
import pandas as pd

df = pd.read_csv("/content/poc_eval_sheet_template.csv")
df = df[df["status"].notna()]

df["top1_correct"] = (df["pred_material_top1"] == df["ground_truth_material"]).astype(int)
df["top3_hit"] = df.apply(
    lambda r: int(str(r["ground_truth_material"]) in str(r["pred_material_top3"]).split("|")),
    axis=1,
)

metrics = {
    "n_samples": len(df),
    "segmentation_acceptance_rate": df["segmentation_accepted"].mean(),
    "top1_accuracy": df["top1_correct"].mean(),
    "top3_hit_rate": df["top3_hit"].mean(),
    "manual_edit_rate": df["user_edited_material"].mean(),
    "median_e2e_time_sec": df["end_to_end_time_sec"].median(),
    "failure_rate": (df["status"] != "ok").mean(),
}

thresholds = {
    "segmentation_acceptance_rate": 0.85,
    "top1_accuracy": 0.70,
    "top3_hit_rate": 0.90,
    "manual_edit_rate_max": 0.35,
    "median_e2e_time_sec_max": 12.0,
}

print(metrics)
print("go_no_go:", (
    metrics["segmentation_acceptance_rate"] >= thresholds["segmentation_acceptance_rate"]
    and metrics["top1_accuracy"] >= thresholds["top1_accuracy"]
    and metrics["top3_hit_rate"] >= thresholds["top3_hit_rate"]
    and metrics["manual_edit_rate"] <= thresholds["manual_edit_rate_max"]
    and metrics["median_e2e_time_sec"] <= thresholds["median_e2e_time_sec_max"]
    and metrics["failure_rate"] == 0
))
```

## SigCLIP vs DINOv3 promotion rule

Promote `DINOv3` only if, on the same dataset, it delivers either:

- `+5` percentage points top-1 accuracy, or
- `-10` percentage points manual edit rate,

while keeping median end-to-end time increase within `+20%`.