check grounding --mark

A checkout button an agent would miss

For computer-use agents: an action map in screenshot pixels, and for each target whether a click there actually reaches it. --mark draws the numbered overlay an agent would be handed.

Open the page under test · fixtures/grounding/partly-covered.html

Run it

vlmkit check grounding fixtures/grounding/partly-covered.html --resolution 1280x720 --mark {out}/mark.png

The numbered overlay, red where the coordinate carries a risk. `--resolution 1280x720` keeps the frame at full size for this page; by default the map is given in the downscaled frame a model is sent (640x360 here).

What to look at

A transparent veil covers the button's centre. The map moves t1's click point to a spot that does reach it, and still reports the defect, because anything aiming at the centre misses.

The fix

Remove the veil, or give it pointer-events: none.

What vlmkit printed

vlmkit check grounding fixtures/grounding/partly-covered.html --resolution 1280x720 --mark {out}/mark.png · exit 1 — the command fails, as a gate in CI would

vlmkit check grounding
source: fixtures/grounding/partly-covered.html (1280x720)

status: suspect
frame: 1280x720 screenshot px — cap 1280x720, scale 1.00 (every coordinate below is in these 1280x720 pixels)
screen: as first loaded — --after "click x,y" measures the screen an action leaves
targets: 2 actionable in frame (0 disabled or out of the frame)

Action map (click point in screenshot px):
  t1 button "Proceed to checkout" @ (208,58) 200x40 #cta [occluded-target]
  t2 button "Save" @ (100,124) 120x40 #save

Issues:
  x t1 occluded-target: #cta (button "Proceed to checkout") is covered at its own centre (140,60): a click there goes to #veil. This map aims at (208,58) instead, which does reach it, with 18px of room before the nearest thing that is not this element — still a defect, because anything aiming at the centre misses.