Lucy-2
A 112.5M-parameter text-to-image diffusion transformer, pretrained from scratch at 128x128 on a single RTX 3060.
Not a fine-tune of Stable Diffusion or anything else — the transformer weights start from random init. Only the VAE and the CLIP text encoder are pretrained off-the-shelf components, both frozen throughout.
Usage
You must pass the negative prompt. It is not optional.
python inference.py --prompt "a portrait of a woman smiling" --n 4
# the important part
NEGATIVE = "watermark, text overlay"
Recommended settings: cfg_scale 4.5, steps 50.
CC3M is largely stock photography, so the model learned to draw caption bars.
Training tagged those captions with watermark, text overlay so the sampler can
steer away from the concept. Without the negative prompt you get watermarks
and noticeably worse faces — watermarked stock imagery correlates with the
waxy portraits in the dataset, so steering away from it lifts quality generally.
Architecture
| Parameters | 112.5M (transformer only) |
| Resolution | 128x128 |
| Latent | 16x16x4, stabilityai/sd-vae-ft-mse (f=8), frozen |
| Text encoder | openai/clip-vit-large-patch14, frozen |
| Backbone | DiT, dim 640, depth 12, 10 heads, patch 1 (256 tokens) |
| Conditioning | adaLN-Zero on timestep + pooled text; cross-attention to text tokens |
| Objective | Rectified flow, logit-normal timestep sampling |
| Sampler | Euler ODE with classifier-free guidance |
Training
Roughly 44 GPU-hours total on one RTX 3060 (12 GB), in stages:
| stage | data | time |
|---|---|---|
| Pretrain | CC3M, 3.02M pairs, 2.86 epochs, 8.6M samples | 20 h |
| Fine-tune | watermark-filtered subset | 4 h |
| Fine-tune | watermark-tagged pool (4% marked) + 10% CelebA face crops | 12 h |
The face crops matter: in a scene a face occupies maybe 5x5 of a 16x16 latent, but a portrait crop that fills the frame gets the whole budget. Mixing them in fixed faces within 2500 steps after they had failed for the entire pretrain.
Limitations
This is a small model on a small compute budget — about 2 H100-hours equivalent, against the ~150,000 A100-hours behind Stable Diffusion. Calibrate accordingly.
Works: scenes, composition, colour, material, lighting. Landscapes, interiors, food, vehicles, single animals. Close-up portraits.
Fails:
- Multi-subject prompts. "Two cats sitting on a sofa" produces uncanny hybrid creatures. Counting is unreliable. This never improved at any stage.
- Relational clauses. "Strawberries next to a glass of milk" drops the milk.
- Structured objects. "A blue bicycle" gives a blue object, not a bicycle — thin spoked geometry is beyond a 16x16 latent.
- Text rendering. No.
- Proper nouns. CC3M hypernymises named entities ("person", "actor"), so they were never learnable.
- Faces in wide scenes remain poor; only close-ups improved.
Training data and licensing
Trained on CC3M (via
dimzhead/cc3m-256) and CelebA
(via tglcourse/CelebA-faces-cropped-128).
CelebA is licensed for non-commercial research use only. Since those images were used in training, this model is released for research use only. Do not use it commercially. CC3M images are likewise not owned by the dataset authors and carry their own terms.
The training data is web-scraped and unfiltered beyond the watermark heuristic described above. It carries the biases of that corpus, including in what kinds of people and scenes it represents well. The CelebA mixture skews the portrait distribution toward that dataset's demographics. No safety filtering of any kind was applied to outputs.