RefControl Pose LoRA — FLUX.2‑klein‑4B
A RefControl LoRA for FLUX.2‑klein‑base‑4B that re‑poses a character: give it a pose skeleton and a reference image of a subject, and it renders that subject in the skeleton's pose. Pose control and identity/reference are carried by the two control inputs of klein's edit path — this is not a separate ControlNet, just a low‑rank adapter on the DiT.
A generated character (T‑pose reference, image 2) re‑posed by a skeleton (image 1). Fully synthetic — no training data shown.
- image 1 — an OpenPose‑style skeleton (the target pose)
- image 2 — a reference image of the subject (identity / appearance)
- prompt —
apply pose from image 1 with reference from image 2 - trigger word —
refcontrol
Files
| File | Notes |
|---|---|
refcontrol-pose-klein-4b.safetensors | rank‑32 LoRA, ~88 MB, 160 tensors. BFL‑fused key layout (diffusion_model.double_blocks.N.*, single_blocks.*), 60 double + 40 single target modules. Loads at bf16 and int4. |
Usage
Swift (MLX, Apple Silicon)
Via the open klein package xocialize/flux2-klein-swift (v0.6.0+):
klein-cli \
--snapshot <FLUX.2-klein-4B> \
--edit skeleton.png,reference.png \
--lora refcontrol-pose-klein-4b.safetensors \
--prompt "apply pose from image 1 with reference from image 2" \
--size 1024 --steps 4 --seed 42 --out posed.png
Or through MLXEngine by pointing KleinConfiguration.loraPath at this file.
Tips
- Control order is load‑bearing: image 1 = skeleton, image 2 = reference. Swapping them swaps the roles.
- The skeleton must be an OpenPose‑style COCO‑18 render (colored limb sticks + joint disks on black). A self‑consistent extractor/renderer (Apple Vision → OpenPose) is available in the training tooling.
- A clean, plain‑background reference yields a clean output — the model preserves the reference's context, so a busy background carries through. For character sheets, use a white‑background reference.
Training
- Base:
black-forest-labs/FLUX.2-klein-base-4B(Apache‑2.0),flux2_klein_4barch, qfloat8. - Recipe: ai‑toolkit, LoRA rank 32 / alpha 32, 4000 steps, lr 1e‑4, AdamW8bit, flowmatch, resolutions 512/768/1024. This checkpoint = the best of the run by a held‑out‑identity A/B (step 3500; later steps regressed slightly).
- Data: ~1,800 (skeleton, reference, target) triples across ~110 distinct human‑motion identities and ~25 activity types (dance, martial arts, gymnastics, fitness, everyday movement). Frames sourced from Wikimedia Commons (CC‑BY / CC‑BY‑SA) and Pexels (Pexels License). Real‑human data; generalizes to stylized subjects via the base model.
Attribution
Training frames are derived from Wikimedia Commons contributors (CC‑BY / CC‑BY‑SA) and Pexels videographers. The dataset itself (the source frames) is not redistributed here — only the trained LoRA weights. Thanks to Black Forest Labs for FLUX.2‑klein and to ostris/ai‑toolkit.
License
Apache‑2.0, matching the FLUX.2‑klein‑base‑4B base model. The weights are a derivative of the Apache base; the training media are credited above and not included.