Myosotis-1.5-base (Preview)

Myosotis-1.5-base is the second flagship release from us, built on the FWKV architecture.

Myosotis-1.5 is engineered to never truly forget—using a mathematically clamped exponential decay that guarantees an infinite effective context window while maintaining blazing-fast inference on consumer hardware.

Architecture at a Glance

Core Technical Innovations

1. The FWKV Recurrent Core

Instead of pairwise attention, Myosotis uses a fixed-size state vector updated via a gated linear recurrence:

$$ S_t = S_{t-1} \odot W + k_t \odot v_t $$

• \( W = \text{clamp}(\sigma(w), 0.1) \) is a learned, constant-per-channel decay.
• By clamping the minimum decay to 0.1, the model guarantees that past information decays exponentially but never reaches zero.

2. Training via \( O(\log T) \) Parallel Scan

Because \( W \) is data-independent (LTI), the recurrence can be trained using a pure-PyTorch Hillis-Steele associative scan. This bypasses the sequential bottleneck of RNNs without requiring custom CUDA kernels, making the architecture trivially portable across NVIDIA, AMD, Apple Silicon, and standard CPUs.

3. Factorized Tied Embeddings

We employ a low-rank factorization for the embedding layer:

• Stored weight shape: \(V \times d_{emb}\) (vs. standard \(V \times d_{model}\)), with a learned linear projection mapping inputs up to the full hidden dimension \(d_{model}\) before the first block, and back down to \(d_{emb}\) for the final logits (so the internal representation remains \(d_{model}\)).
• This saves \(V \times (d_{model} - d_{emb})\) parameters (~15 million in our setup) while maintaining full representational capacity.
• The output head uses the transposed projection, enabling chunked cross-entropy loss to avoid materializing huge logit tensors during training.

Training Regimen

Myosotis-1 was trained in two distinct stages, leveraging the Chinchilla scaling law (20× parameter count in tokens) for optimal performance.

Stage 1: Pretraining (General Language)

• Data Mixture (SmolLM-Corpus style):
  • HuggingFaceTB/smollm-corpus
  • mlfoundations/dclm-baseline-1.0
  • HuggingFaceTB/smollm-corpus
  • HuggingFaceFW/finepdfs-edu
  • HuggingFaceTB/smollm-corpus
  • FlameF0X/arXiv-AI-ML
• Optimizer: AdamW (fused)
• Precision: BF16 Mixed Precision with Gradient Checkpointing

Performance & Speed

Myosotis-1 is optimized for hardware portability over pure theoretical FLOPs. It achieves this by avoiding data-dependent (selective) gating, allowing pure vectorized operations.

• Inference Style: True RNN. No growing KV cache. Constant memory usage \( L \times d_{model} \).
• CPU Speed : [...] tokens/second Decoding & [...] tokens/second Prefill (pure PyTorch, no custom kernels).
• GPU Speed : [...] tokens/second with BF16.

While we sacrifice the "selectivity" of models like Mamba, we gain the ability to run effectively on devices where CUDA is unavailable (edge devices, free-tier cloud notebooks, and standard servers).

Evaluation

Not the best results, we blame the cost of the training.

Usage Example

Myosotis-1 is fully compatible with the HuggingFace transformers library.

from transformers import AutoModelForCausalLM, AutoTokenizer

tok = AutoTokenizer.from_pretrained("FWKV/Myosotis-1.5-base-Preview", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("FWKV/Myosotis-1.5-base-Preview", trust_remote_code=True)

inputs = tok("The quick brown fox", return_tensors="pt")
out = model.generate(**inputs, max_new_tokens=150, use_cache=True)
print(tok.decode(out[0]))

Strengths & Limitations

Cited papers:

arXiv:2305.13048

Licensing & Acknowledgements

License: Apache 2.0

This model is dedicated to the open-source community. Built by Daniel B. (FlameF0X) using vast.ai, PyTorch, and the Hugging Face ecosystem.