LLM Evaluations

Vision agent golden dataset

Proof-of-concept golden dataset of three mobile-app frames. Vision runs are captured offline (one case at a time, up to 10 runs each), human-reviewed, and locked in evals/golden/*/expected.json. CI replays committed outputs with no live Gemini calls. Scores measure current repeatability — not production readiness at scale.

Golden cases

mobile-app profile

Runs per case

captured offline, one at a time

Overall pass rate

77%

3/3 cases locked

Honest scope — not production-ready at scale

These evals prove we can measure vision-agent behavior and show it works well on focused defects (typos, hierarchy clashes). They also expose real variance on complex frames — the Order Details Modal case reaches only 40% full-match. That is expected for a capstone POC, not a failure to document.

Known limitations

3 cases, mobile-app profile only
Non-deterministic model — same input, different findings
Quota failures excluded from pass-rate denominator

Improvements needed

Larger golden set and layout-profile coverage
Consensus voting and per-finding confidence
Region-based inspection for large modals

Capture workflow

pnpm eval:setup — copy fixtures from example.json + figma-output
Start dev server with EVAL_ALLOW_LOCAL_IMAGES=true
pnpm eval:capture --case vaxin-1-4 --run 1 … repeat through run 10
pnpm eval:lock --case vaxin-1-4 --write after review
Repeat for the next case only after the current case is locked

vaxin — Google Pixel 2 - 1

kvT3qcauDE67CW76Kb56Qw · node 1:4

Locked

Profile: mobile-app
Nodes source: example.json
Image source: https://figma-alpha-api.s3.us-west-2.amazonaws.com/images/33f30ad2-8de0-4e29-a22e-d47ecf272e67
Pass rate: 100%
Expected findings: 2

Golden labels

hierarchy_clash · 1:4
typography_anomaly · 2:3

Recent captured runs

run 08 · verified=0 · error
run 09 · verified=0 · error
run 10 · verified=0 · error

vaxin — Google Pixel 2 - 4

kvT3qcauDE67CW76Kb56Qw · node 20:0

Locked

Profile: mobile-app
Nodes source: figma-output/kvT3qcauDE67CW76Kb56Qw_2026-06-10T08-28-44-322Z
Image source: https://figma-alpha-api.s3.us-west-2.amazonaws.com/images/d01b4569-0e5b-4dca-b13b-bc1f7ec66b22
Pass rate: 90%
Expected findings: 1

Golden labels

typography_anomaly · 23:24

Recent captured runs

run 08 · verified=2
run 09 · verified=2
run 10 · verified=2

Bittersweet — Order Details Modal

Ii83n99A2f6VqzHNerla4d · node 9:153

Locked

Profile: mobile-app
Nodes source: figma-output/Ii83n99A2f6VqzHNerla4d_2026-06-10T08-27-21-282Z
Image source: https://figma-alpha-api.s3.us-west-2.amazonaws.com/images/1153dc24-7b28-4fc2-9cef-29e7ee03d43b
Pass rate: 40%
Expected findings: 2

Golden labels

typography_anomaly · I9:160;2:23
typography_anomaly · I9:160;2:24

Recent captured runs

run 08 · verified=0
run 09 · verified=2
run 10 · verified=4