FALCON
Functional Assembly and Language for Compositional Reasoning in X-ray.
Paper · Project page · Code
This repository contains the completed Stage-3 model, including Vicuna-7B-v1.5, DINOv2-L/14, the trained RF-DETR segmentation detector, multimodal adapters, and unmerged LoRA weights.
Usage
Install the pinned dependencies from the code repository:
pip install 'falcon-x[train] @ git+https://github.com/yonathan-kiflom/FALCON.git'
Use JonathanJMK/FALCON or a local downloaded model directory. Authenticate
with huggingface-cli login when accessing the private repository. Review the
custom code before trusting it; pin a Hub commit with revision= for reproducible use.
from transformers import AutoModel
model = AutoModel.from_pretrained(
"JonathanJMK/FALCON", trust_remote_code=True,
).to("cuda").eval()
result, masks = model.predict(
image="image.png", prompt="Describe the image.", max_new_tokens=256,
)
print(result["answer"])
Use predict_segmentation for grounding prompts and predict_panoptic for
panoptic prompts. Tokenization, image preprocessing, and prompt formatting are
included. The model supports one device; automatic multi-device dispatch and
quantized loading are not supported. The saved per-component precision is
preserved; do not cast the entire model to half precision.
Evaluate using the same model:
python -m falcon evaluate run --model JonathanJMK/FALCON \
--dataset /path/to/falcon-x --split test --tasks all \
--run-dir runs/falcon-evaluation --device cuda
Scope and limitations
Trained on falcon-x for X-ray descriptions, questions, component
presence/completeness, instance grounding, and segmentation. Generated answers
and masks can be wrong; this is a research model, not a certified screening
system. Available structured heads are declared in config.json; untrained
risk and physical-link heads are not presented as predictions. Counterfactual
completeness does not establish real-world danger or physical connectivity.
The package preserves the trained architecture and LoRA for further research. It is not an optimizer-state checkpoint, and loading with Transformers alone does not make the repository's stage-training CLI a general fine-tuning tool. Task metrics and evaluation limitations are documented in the code repository.
Licenses
Vicuna is derived from Llama 2 and retains the Llama 2 Community License and Acceptable Use Policy. Llama 2 is licensed under the LLAMA 2 Community License, Copyright (c) Meta Platforms, Inc. All Rights Reserved.
DINOv2 and
RF-DETR segmentation 1.5.2
use Apache-2.0. FALCON code is Apache-2.0 (LICENSE-code); this does not
relicense upstream model weights. Retain LICENSE-Llama-2, LICENSE-code and
Notice when redistributing the full package.