InsightUX — Webcam Eye-Tracking UX Research Browser
A webcam-based gaze tracking system that turns any consumer laptop into a UX research tool. Browse a website, and get back a heatmap, an attention timeline, and a ranked list of which page elements actually held your gaze.
No specialist hardware — just a webcam.
This is a local desktop app, not a hosted demo. It needs a physical webcam, a native window, and full-screen capture — none of which exist on a server, so it cannot run as a Hugging Face Space. Clone it and run it on your own machine.
Requirements before you start
- Windows (uses WebView2 via pywebview's WinForms backend)
- Python 3.11 exactly — mediapipe's legacy face-mesh API this project depends on does not exist on Python 3.12+
- A working webcam
How to run — step by step
1. Clone the repo
git clone https://huggingface.co/<your-username>/insightux
cd insightux
2. Create and activate a virtual environment
python -m venv venv
venv\Scripts\activate
3. Install dependencies — use the pinned versions
pip install -r requirements.txt
Do not pip install --upgrade pywebview or pythonnet. requirements.txt
pins pywebview==4.4.1 and pythonnet==3.0.3 deliberately — newer versions
have a bug in their Windows backend that freezes the app window and floods
the console with:
AccessibilityObject.Bounds.Empty.Empty.Empty.Empty...
If you already have a newer version installed globally, this install step will replace it with the working one.
4. Calibrate — required, per person, per setup
python calibrate.py
Sit normally at your usual distance from the screen, look at each of the 16 dots as they appear. Takes about a minute. This is per-person and per-setup — your eyes, your camera, your screen size. Everyone using this must run it themselves; it is not something you can copy from someone else.
Re-run it if your lighting, seating position, or camera position changes noticeably.
At the end it prints an honest quality readout, including whether the model can actually see where you're looking on each axis:
HORIZONTAL yaw vs screen-X : r = +0.995
VERTICAL pitch vs screen-Y: r = +0.883
If either number is low, the report will say so plainly and explain why — that means the model isn't seeing that axis, and no amount of recalibrating will fix it.
5. (Optional) Check your accuracy
python validate.py
Flashes 9 test dots and reports your real error in pixels. Good for knowing what to expect before relying on a session.
6. Run the browser
python browser_session.py
- Opens on an InsightUX-branded search page. Type a search term (goes to real Google results) or a URL (goes straight there).
- Land on the page you want to study, click on blank space on the page (not a text field), then press S to start eye-tracking.
- Press E to stop. A full report — heatmap over real screenshots, ranked attention list, dwell timeline — generates and opens automatically.
What's in this repo
| File | Purpose |
|---|---|
models/gaze_cnn_v4.onnx (+ .onnx.data) | Binocular gaze CNN (EfficientNet-B0 backbone, dual eye patches + head pose) |
models/model_v4.py | Model architecture definition, used during calibration fine-tuning |
checkpoints/best_model_v4.pt | PyTorch checkpoint for fine-tuning |
calibrate.py | Per-user calibration — 16-point, live blink/lighting rejection, honest quality report |
validate.py | Measures real accuracy in pixels after calibration |
browser_session.py | The research browser — search, track, auto-report |
analysis.py | Builds the session report (heatmaps over real screenshots) |
inference_pipeline.py | ONNX inference + RBF gaze→screen calibration mapping |
preprocessing/preprocessing_pipeline.py | Eye patch normalization, head pose estimation, illumination correction |
Calibration is per-person and per-setup — not included in this repo
calibration.pkl (generated by calibrate.py) encodes your eye geometry,
your camera characteristics, and your screen size. It is deliberately
not in this repo — it would be useless to anyone else and it's personal
data. Run calibrate.py yourself; it takes about a minute.
Accuracy — honest numbers
This is a webcam system, not a Tobii. Expect roughly 5–10% of screen diagonal mean error after a good calibration. That's enough for coarse AOI attribution (navbar vs hero vs footer) and heatmaps. It is not enough for reading-level analysis (which word you're on).
Vertical accuracy is typically a bit looser than horizontal — looking down partially occludes the iris under the eyelid, a physical limit of webcam gaze estimation, not a bug. Calibration automatically checks whether eye aperture (eyelid closing as you look down) tracks vertical position better than the model's raw output, and uses whichever signal is actually stronger.
Known limitations
- Windows-only (pywebview + WebView2 backend). Other platforms untested.
- Head movement during a session degrades accuracy — the calibration assumes a roughly stable head pose.
- No lens distortion correction —
solvePnPassumes zero distortion, which costs some accuracy near frame edges.
Troubleshooting
Window freezes with AccessibilityObject.Bounds.Empty.Empty.Empty...
spamming the console:
Confirm you actually have the pinned versions installed, not newer ones:
pip uninstall pywebview pythonnet -y
pip install pywebview==4.4.1 pythonnet==3.0.3
If it persists after that, turn off Xbox Game Bar (Settings → Gaming →
Xbox Game Bar) and any overlay software (Discord overlay, GeForce
Experience, OBS), then restart your PC — those hooks stay loaded until a
real reboot. Last resort: install PyQt5 + PyQtWebEngine (see
requirements.txt) and switch the GUI backend as described there.
"No calibration.pkl found" when pressing S:
Run python calibrate.py first — it must exist before browser_session.py
can track anything.
Can't type in the search box: Click directly into the search field first — this can happen if the window just opened and hasn't fully grabbed keyboard focus yet.
License
MIT.