crowd-nav — human-response models for the Foresight controller

Trained weights for Foresight, a convex receding-horizon controller that predicts how each nearby pedestrian will respond to the robot's candidate plan, and then chooses that plan by exact convex optimization. These are the response models the controller calls; the planner is plain code (no learned policy, no RL).

Every file, and what we use it for

Two stacks live here and they never touch each other. The controller models (repo root) predict how a person responds to the robot's plan — they are what the robot thinks with. The crowd models (crowd_models/) predict where pedestrians go — we use them to be the crowd the robot has to get through, so the robot is never evaluated against the same model family it plans with.

Which one should I use?

linres_head.pt unless you have a reason not to. It is a few centimetres less accurate as a pure forecaster, but its prediction is exactly affine in the robot's planned displacement, so the planner gets an exact constant Jacobian and the safety constraints stay linear — the whole plan selection becomes a convex QP that solves in ~17 ms with a checkable feasibility certificate. That trade wins where it counts: it leads our benchmark, and it is the only variant that crossed a 90-pedestrian oncoming corridor with zero contacts.

Measured results

Isaac Sim, 16 scenarios × 10 paired seeds (160 episodes per method), identical hardware:

Stress test — a 4.5 m × 105 m corridor with 90 pedestrians all walking against the robot and a continuous inflow so it never empties: LinRes reached the goal in 98.5 s with zero contacts; against pedestrian models it was never tuned on (SRFM, social-force) it reached in 85.0 s and 91.7 s, also contact-free.

Prediction accuracy on the held-out real split (factual ADE, metres — lower is better):

Architecture

Both models are a physics prior plus a learned correction — small by design, because the real moving-robot data is scarce.

prior (no parameters, per pedestrian per 0.25 s step)
  f_goal  = (1.3·(g−p)/‖g−p‖ − v)/0.5
  f_ped   = Σ 2.0·exp(−d/0.4)·d̂      (d < 2.0 m)
  f_robot = k·exp(−d/0.5)·d̂          (d < 3.0 m)
  v ← clip(v + f·Δt, 1.8);  p ← p + v·Δt

residual_predictor_k1.pt     ŷ = prior(k=1) + f_θ(x)
  f_θ : MLP 22 → 128 → 128 → 16   (21,520 params)

linres_head.pt               ŷ = prior(k=0) + f_θ(x) + G_φ(x)·ΔR
  f_θ : MLP 22 → 128 → 128 → 16   (21,520 params)
  G_φ : MLP 22 → 64 → 96          (7,712 params) ⇒ rank-3 16×16 operator
        G = Σ_{r=1..3} a_r b_rᵀ,  ΔR = (R − 1⊗r₀)/5.0
        ⇒ ∂ŷ/∂R = G/5.0  — exact, constant, no differentiation needed

Input x (22-d, pedestrian local frame): 1 s of past positions (4 @ 0.25 s), velocity, goal direction, robot relative position, a robot-nearby flag, the two nearest neighbours, and a 3-way condition one-hot. Output: 8 × 2 future displacements — 2.0 s at 0.25 s.

Training: supervised (Adam, lr 1e-3, batch 128, ~25 epochs), synthetic pre-train then fine-tune on the real PeRoI robot–pedestrian recordings. No reinforcement learning anywhere in this stack. Both run in well under 1 ms per call on CPU.

Load

pip install huggingface_hub torch
hf download elmoghany/crowd-nav linres_head.pt residual_predictor_k1.pt --local-dir .

import torch
ck = torch.load("linres_head.pt", map_location="cpu")
ck["config"]
# {'in_dim': 22, 'horizon_dim': 16, 'rank': 3, 'hidden': 64, 'dr_scale': 5.0,
#  'sfm_k_robot': 0.0, 'convention': 'y = sfm(k=0) + f_resid(x_ctx) + head(x_ctx, dR)'}
ck["f_resid"], ck["head"]   # two state_dicts

The config block is authoritative: sfm_k_robot must match the robot-force strength used in your prior at deployment. Mismatching them is a real bug we shipped once — the residual then corrects an error that is not there. residual_predictor.pt is the artifact of that mistake and is kept only for reproducibility.

Running linres_head.pt with no repository

The checkpoint holds two state dicts and nothing else, so the model classes fit here. Copy this block and it runs — no clone, no install beyond torch and huggingface_hub.

import torch, torch.nn as nn
from huggingface_hub import hf_hub_download

class ResidualMLP(nn.Module):                       # f_resid: 22 -> 128 -> 128 -> 16
    def __init__(self, in_dim=22, out_dim=16, hidden=128):
        super().__init__()
        self.net = nn.Sequential(nn.Linear(in_dim, hidden), nn.ReLU(),
                                 nn.Linear(hidden, hidden), nn.ReLU(),
                                 nn.Linear(hidden, out_dim))
    def forward(self, x): return self.net(x)

class LinearResponseHead(nn.Module):                # G(x) = sum_r a_r b_r^T, rank 3
    def __init__(self, in_dim=22, horizon_dim=16, rank=3, hidden=64):
        super().__init__()
        self.horizon_dim, self.rank = horizon_dim, rank
        self.net = nn.Sequential(nn.Linear(in_dim, hidden), nn.ReLU(),
                                 nn.Linear(hidden, 2 * rank * horizon_dim))
    def _factors(self, x):
        ab = self.net(x).view(-1, 2, self.rank, self.horizon_dim)
        return ab[:, 0], ab[:, 1]
    def forward(self, x, dR):                       # exactly linear in dR
        a, b = self._factors(x)
        return torch.einsum('brh,br->bh', a, torch.einsum('brj,bj->br', b, dR))
    def operator(self, x):                          # G, the exact Jacobian d(response)/d(dR)
        a, b = self._factors(x)
        return torch.einsum('bri,brj->bij', a, b)

ck = torch.load(hf_hub_download("elmoghany/crowd-nav", "linres_head.pt"), map_location="cpu")
f_resid, head = ResidualMLP(), LinearResponseHead()
f_resid.load_state_dict(ck["f_resid"]); head.load_state_dict(ck["head"])
f_resid.eval(); head.eval()

# response to a planned robot displacement dR = (robot_future - robot_now) / 5.0
x, dR = torch.randn(4, 22), torch.randn(4, 16)
with torch.no_grad():
    y = f_resid(x) + head(x, dR)                    # add your sfm(k=0) prior to get the forecast
    G = head.operator(x)                            # (4, 16, 16); divide by 5.0 for metres
    assert head(x, torch.zeros(4, 16)).abs().max() == 0          # no robot => no response
    assert (torch.einsum('bij,bj->bi', G, dR) - head(x, dR)).abs().max() < 1e-5

The two assertions are the properties the convex planner is built on: a zero action produces an exactly zero response (so the no-robot counterfactual is free), and operator() is the Jacobian rather than an approximation of it — no autodiff at plan time.

Where the code lives

Note the gap this leaves: the public package ships the velocity-grid planner, so it loads residual_predictor_k1.pt and not linres_head.pt. LinRes's advantage is that it makes plan selection an exact convex QP, and that only pays off with the QP planner. Use linres_head.pt as a response model anywhere; to reproduce the closed-loop numbers above you need the research stack.

Crowd models (crowd_models/) — the pedestrians, not the robot

Four architectures behind one interface, so they are directly comparable: each observes the target's heading-normalized step velocities plus its 8 nearest neighbours (the robot included, as one more neighbour) and predicts a residual on constant velocity. Rotation into the target's frame makes the error rotation-invariant, so canonical error equals world error.

slstm and sgan are our re-implementations of the published architectures against this interface, not the authors' released weights.

JRDB — trained and tested on real robot-in-crowd data

humanmodel/train_jrdb.py, JRDB alone (43,750 train / 6,923 test windows), 80 epochs AdamW. Metres, lower is better:

Read this honestly: 9 % over constant velocity is the whole prize, the 518 k-parameter transformer does not beat the 36 k-parameter one, and the train/test gap grows with capacity — the ordering on test is close to the reverse of the ordering on train. Pretrained Social-Transmotion scores 0.40 on the same split with its own evaluation script. This is the evidence behind the "real-data forecasting is a tie" limitation below, and the reason our contribution is a controller rather than a forecaster.

ETH/UCY — the crowd drivers

humanmodel/train_cb_models.py, all five ETH/UCY scenes pooled (36,197 windows), 60 epochs. Recorded train-ADE: phys 0.418, slstm 0.273, sgan 0.404.

Those are in-sample numbers and are not accuracy claims. Pooling every scene was deliberate — these weights exist to drive a crowd, and a model that has seen all five scenes makes a better crowd — but it leaves no held-out set, so nothing here should be quoted as ETH/UCY performance. Use the JRDB table above for that. (If you do benchmark on ETH/UCY, note that Social-GAN's <scene>/train directory already contains the other four scenes; reading it as "train" while testing on the held-out scene leaks the test fold and produces impossibly low error.)

Used by humanmodel/crowd_rollout.run_learned_model: an 8-frame history per pedestrian, a model step every 0.4 s, the residual un-rotated into the world, and the crowd walks. The robot in those runs is a scripted wall-bouncing zigzag — we model the humans, not the robot.

Why they are here. Learned crowds are the honest test of a controller that plans against its own learned response model: against pedestrians driven by these, the robot cannot be right by construction. The result that made us keep them is a negative one. Counting contacts with an oncoming robot over one long corridor traverse:

Every trajectory predictor here walks into the robot roughly ten times as often as a model that represents the robot explicitly — and the best forecaster is not the best avoider. Predicting where people walk is a different job from getting out of the way, which is the whole reason the controller stack models the response to the robot rather than the trajectory.

Honest limitations

• Real-data forecasting is a tie. On four real robot–pedestrian datasets (including JRDB), every learned model here ties the analytic baselines on factual accuracy. Our measured explanation: the counterfactual proxy those benchmarks rely on (constant-velocity extrapolation) correlates only r ≈ 0.30 with exact simulated counterfactuals — the target is mostly noise. The gains above are closed-loop gains, not forecasting gains.
• Compliance bias. The training recordings contain cooperative pedestrians. On deliberately stubborn crowds the k1 residual is the weakest learned variant; compliance-randomized training data is the known fix, not yet applied.
• Cross-dataset transfer. Moving residual_predictor_k1.pt to JRDB over-predicts robot-induced deviation by ~6× (its k=1 prior is calibrated to our robot). It makes a controller conservative, not unsafe — it still threaded real JRDB crowds contact-free — but retune sfm_k_robot for a different platform.