initial commit

1 files changed, 107 insertions, 0 deletions

diff --git a/poc_eval_sheet_guide.md b/poc_eval_sheet_guide.md
new file mode 100644
index 0000000..6e8df5a
--- /dev/null
+++ b/poc_eval_sheet_guide.md
@@ -0,0 +1,107 @@
+# POC Evaluation Sheet (SigCLIP Baseline)
+
+Use `poc_eval_sheet_template.csv` as the single source for decision testing.
+
+## Test setup
+
+- Target size: `N = 30` taps across varied scenes/materials.
+- Baseline model first: `SigCLIP` in `model_name`.
+- One row = one tapped object attempt.
+- Keep `status` as `ok` or `fail` for every row.
+
+## Required columns
+
+- Identity: `sample_id`, `image_id`, `image_path`
+- Interaction: `tap_x`, `tap_y`
+- Ground truth: `ground_truth_object`, `ground_truth_material`
+- Model output: `pred_object`, `pred_material_top1`, `pred_material_top3`, `pred_material_conf_top1`, `model_name`
+- Segmentation quality: `segmentation_accepted` (`1`/`0`)
+- Human correction: `user_edited_material` (`1`/`0`), `user_final_material`, `edit_reason`
+- Runtime: `end_to_end_time_sec`, `segmentation_time_sec`, `classification_time_sec`
+- Reliability: `status`, `error_type`
+- Artifacts: `mask_area_px`, `bbox_xywh`, `passport_json_path`
+
+## Decision metrics and thresholds
+
+- Segmentation acceptance rate: `>= 85%`
+- Material top-1 accuracy: `>= 70%`
+- Material top-3 hit rate: `>= 90%`
+- Manual edit rate: `<= 35%`
+- Median end-to-end time: `<= 12 sec`
+- Passport schema validity: `100%` (all `status = ok`, all JSON files valid)
+
+## Google Sheets formulas
+
+Assume headers are in row `1` and data rows are `2:1000`.
+
+- Segmentation acceptance rate  
+`=AVERAGE(N2:N1000)`
+
+- Top-1 accuracy (add helper col `Y`: top1_correct)  
+`Y2: =IF(H2="",,IF(H2=G2,1,0))` then fill down  
+Metric: `=AVERAGE(Y2:Y1000)`
+
+- Top-3 hit rate (add helper col `Z`: top3_hit; top-3 delimiter is `|`)  
+`Z2: =IF(I2="",,IF(ISNUMBER(SEARCH(G2,I2)),1,0))` then fill down  
+Metric: `=AVERAGE(Z2:Z1000)`
+
+- Manual edit rate  
+`=AVERAGE(O2:O1000)`
+
+- Median end-to-end time  
+`=MEDIAN(Q2:Q1000)`
+
+- Failure rate (`status != ok`)  
+`=COUNTIF(T2:T1000,"<>ok")/COUNTA(T2:T1000)`
+
+## Colab/Pandas metric cell
+
+```python
+import pandas as pd
+
+df = pd.read_csv("/content/poc_eval_sheet_template.csv")
+df = df[df["status"].notna()]
+
+df["top1_correct"] = (df["pred_material_top1"] == df["ground_truth_material"]).astype(int)
+df["top3_hit"] = df.apply(
+    lambda r: int(str(r["ground_truth_material"]) in str(r["pred_material_top3"]).split("|")),
+    axis=1,
+)
+
+metrics = {
+    "n_samples": len(df),
+    "segmentation_acceptance_rate": df["segmentation_accepted"].mean(),
+    "top1_accuracy": df["top1_correct"].mean(),
+    "top3_hit_rate": df["top3_hit"].mean(),
+    "manual_edit_rate": df["user_edited_material"].mean(),
+    "median_e2e_time_sec": df["end_to_end_time_sec"].median(),
+    "failure_rate": (df["status"] != "ok").mean(),
+}
+
+thresholds = {
+    "segmentation_acceptance_rate": 0.85,
+    "top1_accuracy": 0.70,
+    "top3_hit_rate": 0.90,
+    "manual_edit_rate_max": 0.35,
+    "median_e2e_time_sec_max": 12.0,
+}
+
+print(metrics)
+print("go_no_go:", (
+    metrics["segmentation_acceptance_rate"] >= thresholds["segmentation_acceptance_rate"]
+    and metrics["top1_accuracy"] >= thresholds["top1_accuracy"]
+    and metrics["top3_hit_rate"] >= thresholds["top3_hit_rate"]
+    and metrics["manual_edit_rate"] <= thresholds["manual_edit_rate_max"]
+    and metrics["median_e2e_time_sec"] <= thresholds["median_e2e_time_sec_max"]
+    and metrics["failure_rate"] == 0
+))
+```
+
+## SigCLIP vs DINOv3 promotion rule
+
+Promote `DINOv3` only if, on the same dataset, it delivers either:
+
+- `+5` percentage points top-1 accuracy, or
+- `-10` percentage points manual edit rate,
+
+while keeping median end-to-end time increase within `+20%`.