Fabric 1.5-0.7B-Base
A 742.5M-parameter base language model with chunked long-range memory and a native 32K context window, developed by Fabric AI.
Overview
Fabric 1.5-0.7B-Base is a 742.5M-parameter pretrained causal language model developed by Fabric AI.
It introduces the custom Fabric Memory Block, which combines local causal attention with compressed summaries of earlier text chunks. This gives the model access to long-range context without applying full quadratic attention across the entire sequence.
This repository contains the FP16 model-only base checkpoint. It is intended for text completion, continued pretraining, fine-tuning, architecture research, and long-context experiments.
Model Details
| Property | Value |
|---|---|
| Parameters | 742,528,520 |
| Context window | 32,768 tokens |
| Pretraining tokens | 23,000,514,560 |
| Weight precision | FP16 |
| Checkpoint size | 1.38 GiB |
| Hidden dimension | 1,536 |
| Layers | 24 |
| Query heads | 24 |
| KV heads | 6 |
| Vocabulary size | 65,536 |
| Local attention window | 2,048 tokens |
| Memory chunk size | 512 tokens |
| Summaries per chunk | 4 |
| Maximum memory summaries | 256 |
| Training hardware | 1× NVIDIA DGX H100 (8-GPU) |
| License | Apache-2.0 |
Architecture
Fabric 1.5 uses:
- 16 local-attention transformer blocks.
- 8 Fabric Memory Blocks.
- One Fabric Memory Block every third layer.
- Grouped-query attention with 24 query heads and 6 KV heads.
- SwiGLU feed-forward layers.
- RMSNorm and rotary position embeddings.
Each Fabric Memory Block combines:
- Local causal attention over the latest 2,048 tokens.
- Learned summaries of earlier completed 512-token chunks.
- A learned gate that blends local and long-range memory outputs.
At full context, the model uses 256 summary vectors to represent earlier parts of the sequence.
Pretraining
Fabric 1.5-0.7B-Base was pretrained on a single NVIDIA DGX H100 node with 8× H100 80GB GPUs.
| Dataset | Approximate weight |
|---|---|
| FineWeb-Edu | 60% |
| DCLM Baseline | 25% |
| OpenWebMath | 10% |
| Cosmopedia v2 | 5% |
Total processed training tokens:
23,000,514,560
Base-Model Inference
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_id = "FabricAI/Fabric1.5-0.7B-Base"
tokenizer = AutoTokenizer.from_pretrained(
model_id,
trust_remote_code=True,
)
model = AutoModelForCausalLM.from_pretrained(
model_id,
trust_remote_code=True,
torch_dtype=torch.float16,
device_map="auto",
attention_backend="auto",
)
inputs = tokenizer(
"The capital of France is",
return_tensors="pt",
).to(model.device)
with torch.no_grad():
output_ids = model.generate(
**inputs,
use_cache=False,
max_new_tokens=64,
do_sample=False,
pad_token_id=tokenizer.pad_token_id,
eos_token_id=tokenizer.eos_token_id,
)
print(tokenizer.decode(output_ids[0], skip_special_tokens=False))
Fabric 1.5-0.7B-Base is a raw text-completion model. Prompt it with text that the model should continue.
Limitations
Fabric 1.5-0.7B-Base is a small experimental research model. It may hallucinate, repeat text, lose coherence, produce outdated information, or generate incorrect outputs.
Important outputs should be verified independently.
Checkpoint Integrity
SHA256: 62a50732d1b1f4f94ba3c4c9838b42777f773fe038126cb6307b235bdfc9878a
License
Fabric 1.5-0.7B-Base is released under the Apache License 2.0.
Citation
@misc{fabric15base,
title = {Fabric 1.5: A Causal Language Model with Chunked Fabric Memory},
author = {Fabric AI},
year = {2026},
url = {https://huggingface.co/FabricAI/Fabric1.5-0.7B-Base}
}