Railway-adapted LISA 7B — semantic + reasoning

This is the merged BF16 checkpoint for the Rail-finetuned LISA — semantic + reasoning row in Table 1 of the Railway Perception Foundation Model repository. It combines railway semantic supervision with language-guided ego-route segmentation and structured switch-reasoning supervision.

This checkpoint requires the project repository and container. It is not a drop-in model for the generic Hugging Face pipeline() or AutoModel APIs.

Reported results

Scores are percentages on the 2,000-image held-out evaluation described in the repository. CIoU pools intersection and union over each subset; GIoU is mean image-level IoU.

Prompts:

• Reasoning-oriented: By examining rail continuity and switch geometry, segment the active ego-route the train is following in this image.
• Generic: Segment the track bed in this image.

Files

The root contains the merged LISA model, tokenizer, configuration, and weight shards. The tuned CLIP model and processor are stored separately under vision_tower/; the entire folder is required for inference.

Inference

Use chat_batch.py through the supplied demo_LISA.sbatch. The launcher accepts either one image or a flat image directory through TEST_IMAGE_SUBDIR.

Download the complete snapshot into a checkpoint directory whose model-folder name contains clip, then run:

# Replace the repository ID and host paths first.
hf download m-vakili75/railway-lisa-7b-semantic-reasoning-clip \
  --local-dir /path/to/checkpoints/railway-lisa-7b-semantic-reasoning-clip

PROC_CKPT=/path/to/checkpoints \
PROC_CODE=/path/to/Railway_Perception_FoundationModel \
PROC_DATA=/path/to/data \
PROC_OUTPUT=/path/to/outputs \
MODEL_NAME=railway-lisa-7b-semantic-reasoning-clip \
TEST_IMAGE_SUBDIR=test/images \
LISA_PROMPT='By examining rail continuity and switch geometry, segment the active ego-route the train is following in this image.' \
sbatch demo_LISA.sbatch

The launcher explicitly supplies both the model root and its sibling vision_tower/. Masks are saved under the selected output run; generated text is written to the Slurm log.

Compatibility notes

• The current local vision-tower builder accepts a saved directory only when its full path contains clip. Keep clip in the downloaded model path and use demo_LISA.sbatch, which passes that path explicitly.
• The exported generation_config.json contains legacy BOS/EOS metadata. The supported chat_batch.py path replaces it at runtime from the bundled tokenizer under Transformers 4.31, yielding BOS 1, EOS 2, and PAD 0. Direct generation through a different loader or Transformers version is not validated.
• The model was exported in BF16 and the supported inference path requires an NVIDIA GPU. Quantized and CPU inference are not validated.
• A generated mask requires the model output to contain [SEG]. If no mask is written, inspect the generated text in the job log and confirm that the complete checkpoint, tokenizer, and saved vision tower were loaded together.

Intended use and limitations

The checkpoint is a research artifact for railway ego-route segmentation and switch-scene reasoning. Generated masks and rationales may be incorrect or mutually inconsistent, and rationales should not be treated as verified causal explanations. Performance may degrade under domain shift, poor visibility, unusual track layouts, occlusion, or ambiguous switch geometry. Do not use its outputs for safety-critical railway control or operational decision-making.

Model provenance

This railway checkpoint was fine-tuned from LISA-7B-v1. The LISA documentation identifies LLaVA-Lightning-7B-v1-1 as the base used for its 7B model. That LLaVA model belongs to the original LLaMA/Vicuna/LLaVA lineage: its upstream model card describes a LLaMA/Vicuna-based model whose delta weights must be applied to the original LLaMA weights. LISA uses Segment Anything (SAM) as its segmentation component and OpenAI CLIP ViT-L/14 as its vision encoder.

Training data and RailSem19

RailSem19 imagery and semantic annotations were used as research/training data for the railway adaptation; project-created reasoning masks and structured switch rationales were additionally used for this joint checkpoint. This Hugging Face model repository contains trained checkpoint files and reported model results only. It does not redistribute RailSem19 images, annotations, sparse metadata, dense metadata, or meta-images. This model release does not claim ownership of, or relicense, any RailSem19 material. RailSem19 remains governed by its official license agreement.

Upstream models and licenses

• LISA / LISA-7B-v1: the starting checkpoint for this railway fine-tune. LISA source code is distributed under the Apache License 2.0, but that code license does not remove restrictions inherited by its model weights.
• LLaVA / LLaVA-Lightning-7B-v1-1: the LLaVA base identified by LISA for its 7B model; it derives from the original LLaMA/Vicuna lineage and remains subject to those upstream terms. LLaVA source code is Apache-2.0; see the LLaVA repository and LLaVA-Lightning-7B-v1-1 model card.
• Original LLaMA/Vicuna lineage: the language-model lineage inherited through LLaVA and LISA. The original LLaMA license permits use solely for non-commercial research and is reproduced in LICENSES/LLaMA-LICENSE.txt. Its restrictions continue to apply to this derivative checkpoint.
• Segment Anything (SAM): the upstream SAM implementation is distributed under the Apache License 2.0. SAM's Apache-2.0 license applies to that upstream component and does not make this entire checkpoint Apache-2.0.
• OpenAI CLIP: LISA uses CLIP ViT-L/14 as its upstream vision encoder. The OpenAI CLIP implementation is distributed under the MIT License; the CLIP component remains subject to its applicable upstream terms, and its license does not relicense the complete checkpoint.

See THIRD_PARTY_NOTICES.md for component attribution, exact source links, and the required original LLaMA notice.

License and use restriction

This checkpoint is released for non-commercial research use only and remains subject to the applicable licenses and terms of its upstream components, including the original LLaMA/Vicuna/LLaVA lineage used by LISA-7B-v1. The checkpoint itself is not licensed under Apache-2.0. Commercial use is not granted by this model card.

Where stated, the Railway Perception Foundation Model project licenses its source code under Apache License 2.0. That code license applies only to source code covered by it; it does not license these model weights as Apache-2.0 and does not override upstream model or dataset terms. Users are responsible for complying with all applicable licenses and terms when using or redistributing this checkpoint.

Disclaimer

This model card summarizes the known upstream licensing and provenance of the checkpoint and is not legal advice. Users are responsible for ensuring that their use and redistribution comply with all applicable upstream licenses and terms.

Citation

@article{ghorbanalivakili2026reasoning,
  author  = {Ghorbanalivakili, Mohammadjavad and Varghese, Ashley and Sohn, Gunho},
  title   = {Reasoning-guided Ego-path Segmentation for Autonomous Trains using Vision-language Models},
  journal = {The International Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences},
  volume  = {XLIX-B3-2026},
  pages   = {89--96},
  year    = {2026},
  doi     = {10.5194/isprs-archives-XLIX-B3-2026-89-2026}
}