Coop-SimLingo
Coop-SimLingo is the hint-enabled SimLingo baseline released with CoDrive, a closed-loop benchmark for temporally grounded cooperative vision-language driving.
Files
coop_simlingo_fp32.pt: model-only FP32 PyTorch state dictionary.coop_simlingo_inference.yaml: frozen inference architecture.model_metadata.json: format and SHA256 metadata.SHA256SUMS: integrity checksums.
Usage
Clone CoDrive, create the pinned environment, and download this model through its helper:
git clone https://github.com/YongshuoLiu/CoDrive.git
cd CoDrive
conda env create -f environment-simlingo.yaml
conda activate simlingo
python -m pip install -e . --no-deps
bash codrive.sh setup
For one-image inference without CARLA:
bash codrive.sh predict \
--image front_1024x512.jpg \
--speed 6.2 \
--target 15.0,0.2 \
--next-target 30.0,0.4 \
--scenario S1 \
--front-car-pos 18.0,-1.2
A complete, runnable S1 record with the front image, numeric inputs, inference
script, and best-005 reference output is available in the
CoDrive example directory:
bash codrive.sh example --device cuda:0
The public Python API is codrive.CoopSimLingoRuntime. S1-S5 use the
<FRONT_CAR_POS> token and its numeric ego-frame position. The canonical S6
cross-traffic warning is non-positional.
Inputs and cooperative hints
The model requires a rectified 1024x512 front image with 110-degree
horizontal FoV, current speed in m/s, and two ego-frame navigation target
points. The optional canonical hints are:
| ID | Exact hint |
|---|---|
| S1 | The pedestrian at position <FRONT_CAR_POS> is about to enter the ego lane. |
| S2 | The vehicle at position <FRONT_CAR_POS> will cut into the ego lane soon. |
| S3 | The static object at position <FRONT_CAR_POS> is blocking the ego lane. |
| S4 | A wrong-way vehicle at position <FRONT_CAR_POS> will cross the ego left-turn path. Yield before turning left. |
| S5 | A wrong-way vehicle at position <FRONT_CAR_POS> will cross the ego right-turn path. Yield before turning right. |
| S6 | A vehicle approaching from the left will cross the ego right-turn path. Yield before turning right. |
S1-S5 additionally require the referenced [forward, right] ego-frame
position in meters. Positive values mean forward/right; negative lateral
values mean left. For example, [18.0, -3.5] is 18 m ahead and 3.5 m left.
The value is not a pixel, GPS/world coordinate, or distance-bearing pair.
<FRONT_CAR_POS> is a hazard-anchor token and may refer to an actor or
expected conflict point, not necessarily a front car. The special token must
not be replaced with coordinate text; the numeric value is supplied separately
through the CoDrive API.
Current speed is a non-negative scalar in meters per second, not km/h. The
camera, speed, navigation targets, warning, and hazard position must describe
the same timestamp. Full details and command examples are provided in
docs/inference_interface.md in the CoDrive repository.
Scope
The model predicts future waypoints for recorded data, vehicle integration,
and CARLA evaluation. It is not a physical-vehicle controller. Deployment
requires independent localization, camera calibration, V2X message handling,
waypoint control, collision monitoring, watchdogs, and emergency stopping.
Follow docs/real_vehicle.md in the CoDrive repository.
Base model
The architecture uses SimLingo with OpenGVLab/InternVL2-1B. Users must comply
with the licenses and terms of the upstream SimLingo and InternVL2 components.