MaleCNS x SpikeWhale anatomy LM

Research artifact. Undertrained. Not a usable language model.

This run of the project is done for now. I answered most of the questions I had. I may pick it up later. At the last saved step (14580) it can speak English fragments - not fluent, but not the the the either. The curiosity benchmark on the network is in the card now (ARC / PIQA / ArithMark / WikiText-2).

Read this before you ask for a scrambled graph

A scramble would ask whether junk wiring of the same size also drops language CE. That is one question. It is not a stamp you put on everything below. These measurements were already in this card. They are not waiting on a scramble.

If you still want a language-CE scramble, say so as that question. Do not use it to wave away PAM/PPL1, MBON dopamine, frozen synapses, sugar/bitter, or the 8-type fit. Those are in the logs.

Hard-LIF language SNN from scratch. Recurrent matrix is the Janelia / Google MaleCNS v1.0 fruit-fly CNS, not a dense 4x1488 SpikeWhale stack. LIF cell is the SpikeWhale primitive (lif.py: hard spikes, fast-sigmoid slope 20, soft reset, DERF clamp 30). Synapse values train with nnz-only autograd through the sparse CSR.

The trained fly as a connectome: exports/connectome-weights-male-cns-v1.0-fly14580.feather - step 14580 (frozen, 30 simulated days) written in the exact format of Janelia's connectome-weights-male-cns-v1.0-minconf-0.5.feather: three int64 columns body_pre, body_post, weight, same sort, same body IDs, so it drops straight into anything that reads the original. 23,970,332 connections (1,653,146 of the original's are gone, none added), 177.9M synapses against the original's 124.2M, 36.2% of connection counts changed, no sign violations (Dale holds, and signs live in the separate transmitter file as in the original). Written by analysis/export_malecns_native.py; exports/malecns_*.feather keep the exact fractional weights plus the original counts and signs.

Language still latches. Greedy at 2500 was the the the. At 2855 it was a longer repeated clause ("It is after the ...). Different failure. Not a fix. This is a measurement. Not SpikeWhale-216M quality.

Connectome export / Dale numbers: step 2948.

I put the eyes first because language barely uses them. Joint vision + language ran 2948 -> 14580 (Dale on). Visual tokens were the wrong cells until 10225. After VPN tokens, a decoder, dopamine "looking" reward, then SmolLM2 judge + looped decoder / HRM / JEPA / MTP, image_gain is +0.114. Captions still don't tell two photos apart. Sugar still works in the body. Bitter does not. Generation used to feed grey-field visual tokens on plain text; that was a generate-path bug, not training.

Summary: the MaleCNS SNN that learned broken English

I wanted to know if I could take MaleCNS v1.0 as Janelia / Google shipped it - no extra neurons, no invented wiring - and get a spiking net that learns language. And what that does to the fly.

It works, sort of. Broken English. Captions through the actual eyes. 167,565 neurons, 25.6M connections, all the fly's. Training only moved synapse strengths (37% of them by step 12040). Text goes in through non-visual sensory cells. A trained head reads it out. The language is in the activity: swap in another sentence's brain state and CE jumps ~3.2 -> ~8.6 (brain_gain ~5.7). Sugar -> proboscis still fires. Still a fly.

Once upon a time, there was a girl smiled .          (step 10225, greedy)
A dog runs in the skateboard .                       (photo on the eyes, sampled)
A dog runnings through a green grasss dog is playing (green image, step 12040, greedy)

Those samples were worse than the model. Text with no picture was being generated with "visual tokens" of the grey field, while training never gives text any visual tokens (only captions have images). Fixed; the same brains, run the way they were trained:

, there was a little girl who loved to play with her mommy and daddy. S   (step 14580, greedy)
, there was a little girl named Sally. He loved to play with her and wa   (step 14260, greedy)
 she was very happy. Then, she went down in the sky and started to go home
                                    ("...saw a big red ball on the grass and", 14260, greedy)
A girl of a yellow field .                              (green image, 14580, greedy caption)

Test-sentence CE dropped from 4.44 to 3.18 (step 12040) and from 4.16 to 2.80 (step 14260) with that one fix. Bug was in the generation path only. Training metrics were always the real ones.

Tokens seen (step 0 -> 14580):

Side branches that did not lead to this checkpoint add ~7.8M positions (the 2948 -> 3440 spike-cap leg, ~4.0M; the first vision leg, ~3.8M).

The Space serves the frozen flies of steps 14580, 14260 and 12040.

What I actually did, in order.

Hard-LIF SNN, sparse synapse training, language from scratch to CE ~4.8. Then Dale's law (keep transmitter signs), spike-cap schedules, a MaleCNS export that round-trips exactly. Then the eyes: photoreceptors placed from their own synapses, colour by opsin, both eyes, tonic lamina, joint caption + text.

A lot of it broke. Full-strength eye flooded the head and exploded gradients. A hinge loss killed caption learning. Visual tokens were read from the wrong cells. A prompt bug made generation look worse than the model.

What made the picture count: tokens from the optic lobe's output neurons, DINOv2 on the eye, Daisy-Base on language, a small decoder, and a dopamine reward for looking through the fly's own PAM / PPL1 cells. After that, image use stayed above zero and kept climbing.

Then I put the trained brain back in a whole-brain LIF fly model. Sugar still fires the proboscis. Bitter does not (0 Hz at 14580; original 21 Hz). 30 simulated days of homeostatic plasticity bring activity back. Freeze the trained synapses and the original biology adapts around them. Language and captions stay. Unbounded scaling wrecks the language even when activity looks right.

It is not fluent. Captions get dog / grass / snow. They do not describe two photos correctly. The fly model's neurons are simplified. This is a simulation of the released wiring. That's it.

Space: Quazim0t0/MaleCNS-Fly. Frozen flies at steps 14580, 14260 and 12040. Talk to one, show it pictures, sugar test, or let it live 1-30 days. Runs get saved to a dataset.

Vision (next phase): the fly's eyes

The visual system is ~103,000 of the 167,565 neurons (62% of the CNS): 89,392 optic-lobe intrinsic cells, 9,201 visual projection neurons (the optic lobes' outputs, 346 types), 4,115 photoreceptors, 563 centrifugal cells. Language training barely touches it: at step 3180 only 6.2% of optic-lobe cells and 1.4% of visual projection neurons had ever been trained, against 33% of the rest of the brain.

Phase 1 trace. Rows: ship, truck, cat. Left to right: 32x32 input; both-eye photoreceptor mosaic; R8y (green, Rh6); R8p (blue, Rh5); R7 (UV, blue proxy); firing rate per stage (photoreceptors, lamina, medulla, T4/T5, lobula / lobula plate, VPN output). Colour-calibrated, both eyes. File: vision/trace/phase1_photoreceptor_channels.png.

The eye, built from the connectome's own metadata (vision/eye.py)

• Retinotopy. assignedOlHex1/2 give each column a hex coordinate: 892 columns (right eye), 879 (left). One transform turns them into a true hexagonal lattice (88% of columns have 6 equidistant neighbours; the rest are the rim).
• Photoreceptors placed from their own synapses. 4,108 photoreceptors carry no soma side (their cell bodies are outside the reconstructed volume), so each is assigned an eye and a column from the cells it synapses onto (vision/photoreceptor_placement.csv). Right eye: R7 (UV) in 651 columns, R8 (blue/green) in 698. Their targets match known fly wiring: R7 -> Dm8a/b, Tm5a/b; R8 -> Mi4, Mi15, Tm20, Mi1; R1-R6 -> L1, L2, L3.
• Colour by opsin. Green -> R8y (Rh6); blue -> R8p (Rh5); green+blue -> R1-R6 (broadband Rh1); red -> nothing (flies are red-blind). RGB carries no ultraviolet, so R7 (UV) gets a blue proxy by default, or can be left silent. Dorsal-rim receptors (polarisation) are left undriven.
• Orientation from anatomy. The dorsal-rim receptors sit at the top edge of each eye (98 and 104 degrees from the eye centre, at 0.89 of its radius), so each lattice is rotated until they point up; the left eye is mirrored. Check: in all three test images the top of the image lands on the dorsal receptors (e.g. ship: top half 0.733 / dorsal receptors 0.747). An earlier mapping had up and down swapped; the Phase 0 numbers below were run with it (linear probes are orientation-blind, so they stand).
• Resolution without inventing neurons. Both eyes (1,771 columns) plus fixation scanning (a larger image shown as a sequence of windows, as a fly scans with saccades). No columns are interpolated.
• Tonic lamina. Every photoreceptor synapse here is histaminergic and inhibitory (R1-R6 -> L1/L2 mean weight -0.38/-0.39), so without a resting drive nothing downstream of the photoreceptors fires at all. Lamina monopolar cells get the tonic depolarisation they have in the fly; light then suppresses them. Calibrated on 6 images x 20 settings (vision/trace/calibration.json): tonic 0.7 with recurrent gain 10 gives L2 firing that tracks the light at its column with r = -0.19 (correct sign, but weak), outputs that differ ~22% between images, 33% of cells active. Higher gain saturates (61% active, images indistinguishable).

Signal flow. Activity moves through the hierarchy one stage per tick -- lamina (tick 1), medulla (3), T4/T5 (4), lobula / lobula plate and outputs (4-5) -- then settles. From the lamina, 98% of the visual projection neurons are 2 synapses away; from the photoreceptors, every Kenyon cell and the whole language readout is within 3. Light can reach what the language head reads inside one token's 4 ticks.

Phase 0: the untrained optic lobe as an encoder

Right eye, conversion weights, linear probe on the 4,612 output neurons. The twin keeps every cell's degrees and outgoing weights and signs, and shuffles only who connects to whom.

• General images: the fly loses to its own rewired twin by ~3 points, at both gains. The twin pools from random places across the eye -- effectively a random projection, which suits a linear MNIST probe; the real optic lobe stays local and specialised.
• Parallax: a small real positive. The fly decodes speed (standing in for nearness) at 0.42, above both pixel baselines (<= 0.35), and the motion-removed control falls to 0.31, so it comes from motion.
• Looming: nothing. The fly sits at chance; the task is linearly solvable across frames, so pixels win.
• Caveats. The twin's outputs are 2-4x more active in the depth tasks (92% vs 45% ever fire), so that comparison is not activity-matched. Every neuron shares one leak and threshold, so motion circuitry that relies on fast vs delayed pathways cannot behave as in the fly; only the wiring is biological. There is no stereo and no range sensing here -- flies have essentially neither. Stimuli were validated so no single frame gives the class away (an earlier looming version leaked it at 68% from the first frame and was rebuilt).

Text is currently fed into the eyes. The atlas "visual" text-entry group is 4,115 cells, 3,937 of which are the placed photoreceptors. Joint image + text training has to move text to the other four sensory groups and the hubs.

Joint vision + language (built; first run from step 2948, Dale on)

One brain, two streams, one readout. Images through both eyes' photoreceptors. Text through olfactory / mechanosensory / gustatory / other sensory groups and the hubs. Same Kenyon + hub next-token head. Every Kenyon cell and the whole readout is within 3 synapses of the photoreceptors, so an image can shape the next token inside one token's 4 ticks. Each step: caption micro-batch (Flickr8k, image held on the eyes for the whole caption) plus a text micro-batch (eyes on a grey field) so the language it already has is rehearsed. Text rehearsal from step 2949 on is 40% FineWeb-Edu / 40% TinyStories / 20% FinePhrase, not the language-only phase's FineWeb-Edu / DCLM / IFM mix.

Turning the eyes on at full strength wrecks the language model. Step-3440 weights, first step with the calibrated eye at full strength: logit_rms 11 -> 40, text CE 4.76 -> 12.3 after one update. Cause: 174 of the head's 256 hub readout cells are optic-lobe cells (CT1, Am1, Li, LPi, Pm, LT, ...). The head has been reading the optic lobe the whole time. Lighting the eye changes its input immediately, whatever the eye-to-brain connection does.

Fix is one dial, alpha, ramped 0 -> 1 over 2,000 steps (anchored to an absolute step, so it continues across relaunches instead of restarting). Cross-fades everything eye-related:

Check from step 3440 (unconstrained, cap 600; tiny batch, one step per alpha):

At alpha = 0 it sits in its own language regime. Caption CE is already well below chance (9.71). Jumping to alpha = 1 in two steps still breaks it. The real run moves alpha by 0.0005 per step.

Start point: step 2948, Dale constraint on. 2948 and 3440 are the same language model within noise (CE 4.875 vs 4.80), but making them Dale-compliant costs +0.22 at 2948 against +1.70 at 3440 (see Dale's law). Vision run starts from 2948: 1,090,918 sign-violating synapses zeroed at load, synapse signs held to the connectome's neurotransmitter signs after every update, central spike cap stays at 80 (what 2948 was trained at). Check from step 2948:

Text CE at alpha = 0 is 2948's 4.875 plus the Dale cost, as expected. logit_rms starts near 21 rather than 11 because zeroing a million synapses shifts the head's inputs. Guard: abort only if logit_rms stays above 35 for 10 consecutive steps (abrupt eye-opening above gave 33-44; real divergence has been 128-315). Saves skipped above that so a bad state never overwrites a good checkpoint. Step 3440 is still a valid start: resume from it with central cap 600, ramp anchored at 3441, Dale off unless you accept the +1.70 cost.

Vision training: what happened (steps 2949 - 10225)

Two vision legs from step 2948, Dale on. Each optimizer step: one caption micro-batch (Flickr8k, 16 x 48 tokens, image held on the eyes) plus one text-rehearsal micro-batch (16 x 48 tokens, eyes on a grey field). About 1,536 token positions per step. ~4.6M token positions for the second leg (2949 -> 5920), half captions. Text rehearsal (the mix for every vision leg, 2949 - 14580): FineWeb-Edu 40%, TinyStories 40%, FinePhrase (faq / tutorial / math / table) 5% each.

Leg 1: eyes opened to full strength -- failed

Alpha ramped 0 -> 1 over 2,000 steps. Up to alpha ~0.7 (step ~4360) it looked healthy: caption CE 5.5 -> 3.0, text CE ~4.7, logit_rms 10-13. Past alpha ~0.95 (step ~4880) it broke:

• Gradient explosions with a finite loss. grad_norm 10^11 - 10^14 and repeated non-finite gradients. Uncapped, 2x-gain optic lobe amplified the gradient tick after tick on the way back through time. I first blamed a spike-cap increase at step 5000. Logs: explosions started at cap 80, before that.
• Language collapsed to The / A line loops (step 5160 generations). Caption CE 3.0 -> 5.3.
• Images became worse than a blank field (caption CE 0.4-0.7 nats higher with the real image than with grey).
• Single rolling checkpoint. Good states between 3400 and 4800 got overwritten. Only 5060 and 5160 survive (both after the break). Later legs keep a numbered checkpoint every 100 steps.

Fixes that came out of it: per-tick cap on the membrane gradient (membrane_grad_clip 1.0: typical per-tick norm 0.07, worst normal tick 0.15-1.3), skip (don't crash on) a non-finite gradient step, linear spike-cap schedule instead of +100/+120 jumps, alpha capped at 0.5.

Leg 2: alpha 0.5, and what I tried

The wrong-image hinge was a design error. It pushed caption CE under a wrong image up, treating right-image CE as a constant. The model could not yet tell images apart, so raising wrong-image CE meant raising all caption CE. Weight 1 cancelled the caption loss. Caption learning stopped for ~1,700 steps.

Why the images are not used: optic-lobe outputs are silent

A trace of each visual stage for the same images (step 4300, and step 2948, at alpha 0.5 and 1.0):

Picture enters the eye and the lamina, then fades through the medulla. Neurons that carry vision to the central brain are nearly silent for every image. Some image information still reaches the language head through the 174 optic-lobe cells among its 256 hub readout cells. Linear probe for caption words (presence of dog, grass, snow, beach, ball, ...; held-out AUC, 0.5 = chance, shuffled-label controls 0.47 - 0.53), 640 Flickr8k images:

So the head receives scene-level information (as much as the pixels carry at this resolution). Opening the eye further adds noise, not information. What the head never does is use it: the image-use check stayed at zero throughout.

Image-use check (logged at every save): 32 held-out image / caption pairs scored with the right image, a wrong image, and a grey field. image_gain = CE(wrong) - CE(right). Over the whole of leg 2 it sat between -0.03 and +0.03 nats, including after attention, DINO, and Daisy. Greedy captions are the same for different images (A man girl., there was a time to the girk .).

Additions to the head and the training signal

• Head attention. Base head reads each token's brain state alone through an MLP. Two pre-norm residual blocks at the head's 128-d width, zero-initialised outputs (loaded model unchanged at step 0), 695k parameters: causal self-attention over earlier head states, and cross-attention from every caption token to 4 visual tokens (the head's input features after each of 4 image-only steps before the caption).
• DINOv2 teacher for the eye (facebook/dinov2-base, frozen, training only). Each caption image also goes through DINOv2 at 224 x 224. The fly's 9,201 visual projection neurons (tanh membrane, averaged over the image-only steps) pass through a trainable linear map. Loss is 1 - cosine to DINO's CLS embedding. Gradient runs back through the fly's own synapses. Match rose from 0 to 0.28 by step 5480, then held at 0.19 - 0.27: optic-lobe outputs now carry part of what DINO sees. The head still did not use it for words.
• Daisy-Base language teacher (DaisyChainAI/Daisy-Base, 106M SpikeWhale transformer, frozen). Tokenizer is byte-for-byte the SNN's (16,512 tokens), so distillation is on the full next-token distribution: on text batches, loss += 1.0 x T^2 x KL(teacher / T || SNN / T), T = 2. On the SNN's own token stream Daisy scores CE 2.57 against ~4.9 for step 2948. In 440 steps the KL gap fell 5.37 -> 2.9. Text CE (4.2 - 5.0, batch-dependent; teacher's own CE on the same batches swings 2.4 - 2.9) had not yet clearly moved when the leg ended at 5920.

State at step 5920

The SNN moved towards the language teacher (KL gap nearly halved). Text CE itself had not yet improved on step 2948 by the end of the leg. Vision side did not produce image-dependent captions. Caption CE fell by learning how Flickr captions are phrased, not what is in each picture.

Steps 5921 - 10225 (both teachers, alpha 0.5)

Same setup for 4,305 more steps (~6.6M token positions, half captions): DINOv2 eye teacher and Daisy-Base language teacher on, head attention on, central spike cap ramping 80 -> 400 by step ~7200 and held there. Stable: no skipped steps, grad_norm ~5, logit_rms ~15. Averages over each tenth of the run (logs/vision_v2_train_5921_10225.log):

Held-out caption CE (image-use check) 4.31 -> 2.87; image_gain stayed at +0.00 to +0.03 the whole run (last: right image 2.865, wrong image 2.873, grey 3.038). Text CE improved only 0.16 over the leg.

Generations at step 10225 (trained eye setup, spike cap 400, greedy; logs/generate_vision_step10225.log):

eyes: grey field     A man in a red shirt -> "y, the the snowy."
eyes: photo 1        A dogs -> " is a rock and the snowy in the air and a girl snow"
eyes: photo 2        A man in a red shirt -> " ack ack ack ack ack ack ack ack ack ack cle ack"
any                  Once upon a time -> "er ack ack ack ..." / "iting."

With a photo the words shift towards caption vocabulary (dog, rock, snow, air, girl). The two photos are not described differently. Greedy falls into ack ack ack loops. Plain-text prompts come out broken. Language is weaker than the training CE suggests.

Why the head does not use the image (diagnosed at step 10225)

Each link from the picture to the words, tested on 64 caption images (--diagnose, logs/vision_diagnose_step10225.json):

Cause: a design error in the visual tokens. The DINO teacher trained the optic lobe's output neurons (VPNs). They learned to carry the image. Visual tokens were built from the language head's input cells - a different set, almost the same for every image. Cross-attention learned to use those tokens as a constant (worth 0.2 nats) while the image sat in cells it never read.

Fix (visual_tokens_from: "vpn"): visual tokens are now the VPNs' tanh membrane, averaged per image-only step (4 tokens x 9,201 cells). In training the caption loss also flows back into the eye through them. Measured on the same step-10225 weights, before any training with it (32 images, logs/vision_diagnose_step10225_vpn_tokens.json): image-specific share of the token variance 0.19% -> 14% (~70x), mean similarity between two images' tokens 0.998 -> 0.858. Token input layer starts untrained, so caption CE does not depend on the image yet (right 2.652 / wrong 2.647). A training run has to learn to read them. Next leg started from step 10225 with this change.

Steps 10226 - 11920: eye-output tokens, a decoder head, and a dopamine reward

Same setup (DINOv2 eye teacher, Daisy-Base language teacher, Dale on), now with visual tokens from the optic-lobe outputs, plus three additions:

• Eye coupling 0.5 -> 0.85 over 1,500 steps from step 10226 (reached at 11726).
• Decoder head. Small modern transformer on top of the brain's read features: width 512, 4 layers, 8 heads, each with causal self-attention with RoPE, cross-attention to the visual tokens and an MLP, final norm; 32M parameters. Its logits add to the original head's through a zero-initialised output layer, so the loaded checkpoint is unchanged at the first step (text CE identical, 3.618 on the CPU check). It does not see the token embedding. Everything it knows comes through the brain. A brain check is logged at every save: same held-out captions scored with another sequence's brain state; brain_gain = CE(wrong brain state) - CE(right). It was 4.3 at 10225 and rose to 5.3 - 5.8 with the decoder: the head leans more on the brain, not less.
• Dopamine reward for looking (from step 10761). For every caption image: reward = CE(caption | wrong image) - CE(caption | right image); reward prediction error = (reward - running mean) / running std. Delivered through the fly's own dopamine neurons: a positive error drives the 316 PAM (reward) cells, a negative one the 16 PPL1 (punishment) cells, while the image is shown again. Their spikes reach 8,615 cells (4,061 of the 4,064 Kenyon cells) through the real 149,070 DAN synapses. A three-factor rule then changes the 2.6M synapses onto those cells: d|W| = lr x (dopamine at the post cell - its running baseline) x pre rate x post rate (PAM input +, PPL1 input -; magnitudes only, Dale-safe; lr 1e-4, capped at 1e-3 per step), applied next to the gradient update.

image_gain averaged over each period (a single save scores only 32 pairs and swings +-0.02 - 0.03 on its own, so averages are what matter):

By quarter of the dopamine period: +0.040, +0.045, +0.043, +0.058. Per-batch training reward stayed positive throughout (+0.040 -> +0.067). First period where image use was consistently above zero. Language improved at the same time: text CE 4.28 -> 4.07, gap to Daisy (KL) 2.30 -> 2.04; brain_gain held at 5.6. No skipped steps. That before/after is the result. You do not need a scrambled MaleCNS to believe it. The leftover control is not "delete the fly": eye coupling also rose 0.63 -> 0.85 in the same window, so a run at 0.85 with dopamine off would split dopamine from a stronger eye. grad_norm rose from 4 - 8 to 17 - 48 in this period (clipped, stable so far).

Generation fix. Generation code did not put the prompt into the head's attention history (training always sees every earlier token), so attention effectively forgot the prompt. With the prompt fed in, greedy text at 10225 went from Once upon a timeer ack ack ack ... to Once upon a time, there was a girl smiled ., and with a photo on the eyes A dog runs in the skateboard . (sampled). Generation now prints greedy and sampled text (temperature 0.7, top-k 40, repetition penalty 1.3).

Where the time goes (CPU profile of one step, full setup, logs/profile_cpu_step10225.json): backward pass 49.8%, brain ticks forward 40.6%, text data 4.1%, optimizer 2.8%, DINO 1.1%, Daisy 0.5%, head + attention + decoder 0.1%. On the logged runs a step went from 3.24 s to 3.99 s as the spike cap rose 200 -> 400. The brain simulation is the whole cost.

Steps 11921 - 14580: SmolLM2 judge, looped decoder, HRM, JEPA, MTP, and a cheaper step

Added on copies (vision/train_anatomy_vision_v3.py, _v4.py). Older scripts left alone. Each new part starts at exactly zero effect so a checkpoint resumes unchanged.

A second teacher that judges, and tastes. HuggingFaceTB/SmolLM2-360M scores the fly's own continuations (8 text prompts per step, 24 new tokens): reward = mean log p(continuation | prompt) + 2.0 x (share of distinct tokens - 1). The second term is because a judge finds ack ack ack very predictable. Reward prediction error (reward minus a running mean, over its running std) drives the fly's PAM dopamine cells when better than expected and PPL1 when worse, and at the same time the sugar-like taste neurons (reward) or bitter-like ones (punishment), while the fly's own words are replayed. Three-factor rule then changes the 2.6M synapses onto dopamine-receiving cells. Daisy-Base still teaches the full next-token distribution next to it (different tokenizer, so SmolLM2 can only judge whole sentences).

Byrne pieces in the head (from Byrne70M-GatedFlow-HRM-JEPA): a looped decoder (the 4 blocks run 3x with shared weights, per-pass embeddings, zero-init gate per extra pass), HRM refinement with deep supervision, MTP (predict the token after next), and JEPA on the fly's own brain state (predict the next token's read-cell activity).

Two of those were dead on arrival. I measured the gradients and fixed them:

Also added: document masking in the packed 256-token text (attention and JEPA stay inside one document; the brain itself runs on, as a fly's does) and 256-token text windows for the rehearsal half (captions stay at 48), so the head's attention sees multi-sentence context instead of 1-2 sentences.

Throughput. Benchmark (analysis/ + logs/profile_cpu_step10225.json): batch 16 -> 64 gives 1.75x more text per hour and saturates there (128 gives no more). Brain ticks are 72% of a step. Census of 40 real batches: 129,273 of 167,565 neurons (77%) never spiked, and 100,962 of them never rose above half of their threshold and are not read / visual-output / eye / Kenyon / dopamine / taste cells. Skip the synapses into those (10.1M of 25.6M kept, 39%): captions bit-identical (CE 2.93761 vs 2.93762), ~43% less brain work. Every 500 steps one batch runs through the FULL brain so anything that wakes up rejoins (530 rejoined in the first refresh). Batch 64 + the cut + background text loading: a step went 15.5 s -> 34.4 s for 4x the text. ~1.8x more tokens per hour.

DEER does not work here (analysis/deer_test.py). Parallel-in-time solving (Lim et al. 2023, quasi-DEER with per-neuron scans) reproduces the sequential trajectory exactly, but needs 57 iterations for 64 ticks. Spikes are discrete, so correct information spreads only ~1 tick per iteration. ~50x slower than just running the ticks. Parallel hardware does not rescue it: each iteration still has a full sequential scan.

Results over the leg (steps 11921 - 14580, ~2,700 steps):

grad_norm spikes to 200 - 1600 on some steps. Measured: w_grad ~1.0 on those steps (the synapses, not the new head parts). Caption half. Happens with the neuron cut ON and OFF alike (67.5 vs 64.6 on the same step). Particular caption batches, not the cut or the batch size. Clipped. No skipped steps.

Interruptions. v4 leg interrupted three times (restarts from 14260, 14452, 14580). Each lost up to 20 steps plus startup.

The newest fly: step 14580

Frozen the same way (30 simulated days as a fly, trained synapses frozen): brain responding to sugar 2.6% -> 4.1%, sugar -> MN9 217 -> 177 Hz, bitter 0 Hz throughout, untouched synapses changed 17.9%, and language exactly unchanged (CE 2.7627 before and after, identical text and captions). MaleCNS edge table round-trips exactly (max difference 0.0; 52.9% of edges changed from the original fly, 0 sign flips). checkpoints/vision_v4_step14580_frozen_day30.pt.

Put back in the body at 14580: sugar still works. Bitter is gone. Original fly: bitter-like taste -> MN9 21 Hz. This checkpoint: 0 Hz. Sugar still drives the proboscis. The aversive arm of the feeding circuit did not survive this far. (Bitter -> MN9 was already 0 Hz on every trained brain from 2948 on, including 12040. Original 21 Hz leak never came back.)

What training changed in the fly (analyses on steps 2948 / 3440)

Where the weights moved (analysis/delta_map.py, trained minus connectome, by presynaptic transmitter and pathway). Training hit the neuromodulators hardest:

Visual centrifugal -> optic lobe: 71% changed, 14.8% of all change. Kenyon -> Kenyon: 45.5% changed at 2948, 56.6% at 3440.

Could a local rule explain it? (analysis/local_rule_fit.py) Presynaptic firing rate alone predicts how much a synapse moved (held-out R^2 0.33, Spearman 0.65). Adding Hebbian co-firing barely helps (R^2 0.35). 61% of changed synapses grew. The pre factor is local. The post factor (the credit) is not Hebbian.

Could the modulatory neurons deliver the credit? (analysis/credit_signal_test.py) Cells that receive dopamine / octopamine / serotonin input hold 98.8% of all credit (dLoss / d membrane), so the coverage is there. Moment-to-moment modulatory input does not track it (within-cell r -0.06, vs -0.11 for matched cholinergic input and -0.03 for a time-shuffled null; signed r 0.001).

Mushroom body test (analysis/mushroom_body_test.py). Connectome mushroom body is complete: 4,064 Kenyon cells, 97 MBONs (37 types), 340 DANs (PAM / PPL1 / PPL2), 61,210 Kenyon -> MBON synapses, every MBON has a direct DAN input (textbook pairings recovered, e.g. PPL1-gamma1pedc -> MBON11). Only Kenyon -> MBON synapses learn. Odour-conditioning task (64 tokens presented alone as "odours", random valence; frozen step-2948 brain; 5 seeds; held-out accuracy, majority class 0.58):

Every dopamine rule learns, ordered by how precise the teaching signal is. None beats backprop. Adding dopamine to backprop made it worse. Cheaper, blurrier route to the same credit - useful for learning without backprop, not as an add-on to it. (Harder in-context version of the task: nothing, not even an unconstrained readout, beat the majority class.)

Back to the original connectome format

export_to_malecns.py inverts the MaleCNS -> SpikeWhale conversion (W = sign(transmitter) x synapse count x 0.01) and writes a flat edge table in MaleCNS terms: body_pre, body_post, weight (effective synapse count) plus the original count and both signs. Untouched artifact round-trips exactly (0 of 25,623,478 edges differ; 124,174,455 synapses). At step 5160: 34.9% of edges changed, 564,241 trained to zero, 0 sign flips (Dale on), ~143.7M effective synapses (+16%); counts are no longer whole numbers. Converting that table back into the SNN reproduces the step-5160 weights exactly (max difference 0.0) and identical generations. Neuron-class parameters (threshold, leak, input gain), the language head, and the eye settings have no MaleCNS equivalent. They stay in the training checkpoint.

The trained brain as a fly again (repair tests)

Once the connectome has learned language, does it still work as a fly? If it lives as a fly again, learning only with biological rules, can the untouched biology repair the trained parts?

Simulation. Whole-brain leaky integrate-and-fire in the style of the community models (Shiu et al. 2024, Nature): dt 1 ms, v_rest / v_reset -52 mV, threshold -45 mV, tau_m 20 ms, tau_syn 5 ms, 2 ms refractory, 0.275 mV per synapse (signed by transmitter), taste neurons driven with 150 Hz Poisson spikes. Every brain gets the same neuron model; only the synapses differ (no language head, no trained per-class gains, no spike cap). analysis/fly_circuit_test.py, analysis/fly_repair_sim.py.

Taste groups. MaleCNS labellar taste neurons carry no receptor labels (types LB1a - LB4b), so the groups come from the original wiring: labellar types that excite the proboscis motor neuron MN9 within three synapses are the appetitive (sugar-like) group, LB2b, LB3a-d, LB4b (86 cells); those that inhibit it are the aversive (bitter-like) group, LB1c, LB2a, LB2c (26 cells).

Stage 1: the feeding reflex after training

MN9 firing, 1 s, 4 trials (logs/fly_circuit_sugar_mn9.json):

The original validates the model: sugar-like taste drives the proboscis strongly, bitter-like barely. Sugar survives. Bitter does not. The small bitter leak into MN9 is gone in every trained brain (21 Hz -> 0 Hz). Taste responses spread through far less of the brain (13.8% -> 3 - 4% of cells): activity reaches its target without spilling, matching the training's biggest changes (to the widely broadcasting neuromodulator synapses).

Stage 2: 30 simulated days of repair

The step-2948 brain lives through "days": each day sugar-like taste, bitter-like taste, and two fixed sets of other sensory neurons (150 ms each). After each day every neuron applies homeostatic synaptic scaling: excitatory input synapses scale up if it fired below its set point, down if above. Signs never change. Set points are the original connectome's firing rates for the same experiences - the untouched biology is what pulls the brain back. Language CE is measured with the step-2948 head on 4 fixed FineWeb-Edu batches (original synapses 9.25, trained 4.99).

Controls: original brain under the unbounded rule stays stable for 30 days (12.1 - 13.1% active, MN9 187 - 207 Hz), so the rule itself does not damage a healthy brain. Trained brain without plasticity does not change.

• Activity comes back. The synapses do not. Under every rule the trained brain's response spread moves back towards the original's. Only ~7 - 10% of the trained synapses move back towards their original values. Neurons that training had quietened scale up their other inputs instead. The brain finds a different wiring with similar activity (degeneracy) rather than undoing the training.
• Unbounded, that compensation runs away and destroys both the reflex (MN9 67 Hz) and the language (CE 110.5, far worse than chance).
• Bounded, it settles (activity doubles, reflex intact) but still slowly erodes the language, because the language synapses are rewritten too.
• With the trained synapses frozen, the rest of the fly adapts around them. Activity recovers almost as much (3.4% -> 5.8%), the reflex holds, and language CE does not move at all (4.9852 on every day), although the untouched synapses changed by 15% on average. The synapses training never touched are exactly those whose presynaptic cells are silent during language (the trainer only changes a synapse when its presynaptic cell fires). Original biology can rewire all of them without touching what was learned.

Day 15 of the bounded run, converted to MaleCNS and back. export_to_malecns.py -> MaleCNS edge table (38.8% of edges changed from the original fly) -> rebuilt by body IDs -> identical to the day-15 brain (max difference 0.0). Its 1,090,918 sign flips are the ones step 2948 already had from before Dale's law was switched on. Generations at spike cap 80 (logs/generate_repair_bounded_day15_from_malecns.log; baseline logs/generate_step2948_baseline_for_repair.log):

                      DAY 15 (repaired, round-tripped)       STEP 2948 (no repair)
The                   The the the the the the the |          The to the |----------
Once upon a time      Once upon a time the |-------           Once upon a times.
The fruit fly brain   The fruit fly brains.                   The fruit fly brain.

Both are the weak early step-2948 language model. Day 15 is slightly more repetitive, in line with its higher CE (5.75 vs 4.99). The frozen run's language CE equals step 2948 exactly, so its text would be the same.

The same tests on step 12040 (vision + language + dopamine, Dale on)

Stage 1 (logs/fly_circuit_sugar_mn9_step12040.json): 37.2% of synapses changed; sugar -> MN9 235 Hz (original 222), bitter -> MN9 0 Hz, 2.7% of the brain responding (original 13.8%). Sugar still works after more training. Bitter is already gone.

Stage 2, 30 days (brain responding to sugar, day 0 -> 15 -> 30; sugar -> MN9 at day 30), with language and captions measured in the full vision model (decoder and eye included) on six fixed test sentences and a green test image:

• Unbounded repair now brings activity all the way back to the original fly's level (12.0%), but the reflex collapses and the language turns into letter loops.
• Bounded repair keeps the reflex (207 Hz) and moves the trained synapses little (1.06x), yet still breaks the language.
• Frozen: the fly's untouched 63% adapted for 30 days (mean change 17.6%) while language, the green-image caption (A dog runnings through a green grasss) and the reflex stayed exactly the same. It recovers less activity than at step 2948 (4.3% vs 5.8%) because 37% of the brain is now frozen instead of 22%. This frozen fly (checkpoints/vision_v2_step12040_frozen_day30.pt) is the one the Space runs.

The same tests on step 14580 (the newest fly)

Frozen and unbounded, the two ends of the range, on the newest checkpoint (logs/fly_repair_frozen_step14580.*, logs/fly_repair_unbounded_step14580.*, logs/fly_repair_unbounded_step14580_language.json). 39.6% of synapses changed by training; set point 17.2% of cells active.

• Unbounded repair brings the brain all the way back to the original fly's activity (12.0%, original ~12.5%), and at 14580 it does not wreck the reflex the way it did at 12040 (157 Hz here vs 63 Hz there) - the trained synapses drift only 1.18x from the original, against 26x on the step-2948 brain. More training left the brain closer to something the homeostatic rule is happy with.
• It still destroys the language: CE 2.76 -> 6.61, and greedy text falls apart (, there was a little girl who loved to play with her mommy and da -> 't wastret thikingstgle tot STNARERSTCO). Activity being right is not the same as the language being right.
• Frozen is still the only arm that keeps both. The untouched 60% of the fly adapts for 30 days (17.9% mean change), the reflex holds, and the language is bit-identical.

Limits. One circuit family (taste -> proboscis) measured closely. Taste groups inferred from wiring. Synapses only (no per-class neuron parameters). Simulated spike rates run high (compare brains, not absolute Hz). Homeostatic scaling is the only rule tried in stage 2 (a dopamine-reward version during the fly's life is not yet tested).

What went wrong

1. Opening the eye to full strength flooded the head (174 of its 256 hub cells are optic-lobe cells) and caused the gradient explosions.

2. A single rolling checkpoint lost the good vision states. Numbered checkpoints now.

3. Wrong-image hinge loss stopped caption learning for ~1,700 steps.

4. The fly's own vision pathway to the central brain is almost silent, so the image never reaches the language circuits through the route the fly uses. Neither attention nor a DINO teacher on those outputs made the head use the image in 900 steps.

5. A bfloat16 teacher cast broke Daisy-Base's n-gram hashing on the first step of one launch (fixed: the teacher stays float32).

6. Visual tokens for the head's cross-attention were read from the wrong cells (the head's own input cells, 99.8% identical across images) instead of the optic-lobe outputs the DINO teacher trained. ~4,700 steps of vision training could not use the image through them. Fixed: tokens now come from the visual projection neurons.

7. Greedy text at step 10225 loops (ack ack ack) and breaks on plain prompts, even though text CE with the Daisy teacher is 4.36 - 4.5. The teacher narrowed the distribution gap. It did not fix degenerate greedy decoding.

8. Much of the ack ack ack was a generation bug: the prompt was never put into the head's attention history. Fixed. With it, greedy text at 10225 reads Once upon a time, there was a girl smiled ..

9. Two of the Byrne additions were dead on arrival: the loop gates sat at +-0.006 (zero per-pass embeddings behind zero gates) and JEPA had loss 0.0065 (consecutive brain states are ~98% alike, so "predict the next state" was "copy this one"). Both only showed up when the gradients were measured, ~700 steps in.

10. The run was interrupted three times by the compute being revoked (restarts from 14260, 14452, 14580), each losing up to 20 steps plus startup. That is what ended the run at 14580 rather than a chosen stopping point.

How it ended. The dopamine "looking" reward was the first thing that kept image_gain consistently above zero. That is measured. A scramble is not the missing piece. The leftover split is eye coupling 0.85 with dopamine off, if anyone wants it. The SmolLM2 judge, the looped decoder and JEPA (after the zero-gradient traps were fixed) plus the 39% synapse cut took text CE to 3.30 and test-sentence CE to 2.76 by step 14580. The Daisy KL gap (1.92) was still closing when the run stopped.

Connectome (this graph)

Wiring is MaleCNS v1.0 from FlyEM (HHMI Janelia), the University of Cambridge, the MRC LMB, and Google Research. Hugging Face Papers only auto-tags arXiv IDs. This paper is in Cell, so the Hub will not grow an arxiv: chip.

Connectome data: CC-BY 4.0.

Architecture (language checkpoint)

token id
  -> embed (64)
  -> current on: 5 sensory-group winners + top-32 of 256 highest-out-degree cells (drive 0.8)
  -> 4 LIF ticks, hold current, rec = (W @ spikes) / sqrt(in_degree) * rec_scale(5)
  -> Kenyon spikes re-enter W (mask_kenyon=false); spike-mass cap 80
  -> tanh(pre-reset)[Kenyon U hubs] -> LayerNorm -> Linear 128 -> vocab 16512

• kc_drive = 0. NT gains frozen. Trainable: sparse W, embed, hub_in, head.
• Optimizer: MuonEq-R on 2D except out; AdamW on 1D / embed / out. Peak Adam 1e-3, Muon 5e-3.
• Data, language-only phase (steps 0 - 3440): 40% FineWeb-Edu / 40% DCLM / 20% IFM (pretraining_config.json; 9 streams - the IFM share is 7 behavior configs at 0.0286 each). From step 2949 (the vision run) the text mix changed to 40% FineWeb-Edu / 40% TinyStories / 20% FinePhrase (faq, tutorial, math, table at 5% each; vision/text_sources_vision.json), replacing DCLM and IFM: short narrative text to rehearse against Flickr8k captions. Everything from 2949 to 14580 used that mix.
• Tokenizer: SpikeWhale byte-level length-max, vocab 16512.
• code/sparse_mm_fire.py: fire-only forward (scatter firing cells); d_spikes = W^T @ gout via cuSPARSE (exact LIF). Config checkpoint_every: 0 = no activation checkpoint (full tape).

Language training status

Everything in this section is the language-only phase (steps 0 - 3440), before the eyes were opened. It ends at step 3440; the run that continues to 14580 starts from step 2948 and is described under Vision training above. The tables are kept because they are where the spike cap, coverage and Dale numbers were measured.

Step 3440 (last language-only step) / 3180 - spike-cap ramp

Seq 512 x batch 16 x accum 1 (8192 tokens/step, packed, FineWeb-Edu only), max_spikes ramped 400 -> 500 (step 3281) -> 600 (step 3381). 260 steps, ~2.13M tokens. Every log line now records coverage: which cells have ever fired, and therefore which synapses have ever been reachable by gradient (seeded from what earlier runs had already changed).

The ramp raised intensity, not reach. The same ~31k cells fired harder; almost no new cells were recruited (+0.08 points of synapse coverage in 260 steps). Text enters through a fixed set of entry points, and it does not reach most of the fly -- above all the visual system. No aborts, no skipped saves.

Step 2855 (seq-2048 continuation)

Nothing latched and nothing broke. 350 steps at 8x the tokens per step did not move CE out of the band that 1500-2500 already sat in.

Step 2500 (seq 256)

Kenyon/readout did not latch. Language did not leave the weak-unigram / broken-English band. 1.5k extra steps after 1000 moved CE 4.89 -> 4.78. That is not a ceiling proof; it is also not a generation win.

What actually trains

Gradient reaches a synapse only if its presynaptic cell fired (sparse_mm_fire.py computes grad_values for active presynaptic cells only). Measured against the conversion weights at step 2948:

~78% of the "trainable" synapses have never received a gradient. The edges that did move changed by more than their own original magnitude.

Instantaneous participation is far lower than cumulative. Counting spikes per neuron over a 128-token window (logs/participation_2500_vs_2855.json):

~1.5% of the connectome is active at any moment, and the spike mass concentrates further into the readout population as training goes on. The effective model is a ~2.5k-neuron subnetwork inside a 167,565-neuron graph.

This is not a reachability limit. From the sensory + hub injection sites, one synaptic hop already reaches 159,113 neurons (95%), and 165,378 (98.7%) are both drivable and able to influence the readout within the 4 ticks. The graph is small-world; the sparsity is dynamics, not wiring.

Spike-mass cap: what pins participation

GradThroughSNN.step rescales every spike by min(1, max_spikes / mass). With ~268 cells firing and max_spikes: 80, the scale is ~0.30 and total spike mass is pinned: recruitment is zero-sum, because a newly firing cell dilutes the ones already carrying signal. Ramping the cap (logs/spike_cap_experiment.json):

The cap was the binding constraint. 80 -> 400 tripled the number of firing neurons, raised whole-network activity 3.6x and doubled the gradient reaching W, at stable logit_rms. 2000 broke training outright: grad_norm 2.4 -> 48.3, CE above uniform. Note 0.461 is close to the 0.45 where failure mode 2 lives.

What it did not buy: CE. Over 150 steps at cap 400 the mean went 5.017 -> 4.939 (best single step 4.468) - back into the pre-cap plateau band, not below it. More of the network participates and more synapses receive gradient; that has not yet become better prediction.

Dale's law and the exported connectome

Training is free to flip a synapse's sign, which makes the graph no longer neurotransmitter-consistent. By step 2948, 1,090,918 synapses had flipped sign (19.3% of the edges that changed, 4.3% of all edges).

export_trained_brain.py recombines a trained checkpoint's synapse vector with the frozen artifact's wiring and writes a plain connectome SNN - same neurons, same edges, same identity matrix, trained synapses, no language-model parts - loadable by ConnectomeSNN.load():

python export_trained_brain.py \
  --ckpt <trained_checkpoint>.pt \
  --out male_cns_spikewhale_trained.pt          # add --dale for the constrained one

--dale clamps every synapse back onto the sign its neurotransmitter gave it (violating ones go to 0), so the exported brain stays Dale-compliant. The trainer also takes dale_constraint: true, which re-projects after every optimizer step. Language-only training had not used that yet. The vision legs from 2948 ran with it on.

The constrained brain fires more, because zeroing sign-violating synapses removes inhibition that training created by flipping excitatory synapses negative.

Both exports still drive language when paired with the step-2948 head. Greedy, 32 new tokens, same prompts, same decode:

                  AS-TRAINED (0 zeroed)                DALE-PROJECTED (1,090,918 zeroed)
The               The to the |-------------            The therect then them to bet is again,
                                                       them therect then, them to there is after

Once upon a time  Once upon a times.                   Once upon a times, therecting then then
                  (blank lines to the limit)           then thenks.

The fruit fly     The fruit fly brain.                 The fruit fly brain, then then them to
brain             (blank lines to the limit)           because the |------------------

Greedy on the as-trained brain collapses into whitespace or dashes within a few tokens on all three prompts; the Dale-projected one keeps emitting word-shaped tokens for the full 32. Neither is English.

That is not the constrained brain modelling language better. On identical batches (logs/dale_vs_astrained_eval.json):

Projection costs 0.22 nats of prediction. What improves is greedy degeneracy only - 1.8x more distinct tokens, far less filler - which is a decoding artifact, not accuracy. Under real language input the constrained brain fires less, not more, so the pulse test above does not explain the generations: an earlier "more excitable" reading of this was wrong.

The likelier mechanism: the head spent 2,948 steps fitting one feature distribution and learned a strong period-then-whitespace attractor; any large perturbation of W knocks the features off that memorised point, breaking the attractor while also hurting accuracy. The control that would settle it - zeroing 1,090,918 randomly chosen synapses of matched magnitude - has not been run.

Sampled (T=0.8, top-k 50) from the Dale-projected brain:

Once upon a time thems are really, and the reven anly to a back that the said, deal, a cleanks i
The fruit fly brain is then a lonk-sing but themss to bet it was bet is a site wasn in if you

The head was trained against unconstrained weights, so all of this measures how a head copes with a brain that shifted under it - not the quality of a model trained with the constraint on. The trainer supports that (dale_constraint); it has not been run. Full text: logs/generate_connectome_step2948.log.

What survives as a result: removing every Dale-violating learned update costs only 0.22 nats. The sign-illegal part of what training learned is real, but small.

At step 3440 that no longer holds. Same projection, same protocol (identical batches, each model at its own training cap -- 600 here), logs/dale_vs_astrained_eval_step3440.json:

After the spike-cap leg the model depends far more on its sign-flipped synapses. Two readings fit and this data cannot separate them: training at caps 400-600 built more dependence on those synapses, or the higher cap the model runs at amplifies the same perturbation. Step-3440 Dale-projected brain also generates visibly worse (next section). The less-degenerate greedy output at 2948 did not survive.

W ablation (steps 1000 and 2855)

Same 4 batches. Not a trained scramble control. Not re-run at 2500. File logs/eval_w_ablation_step2855.json is in the repo.

Trained synapses are used at eval.

Step 2855 (same protocol as step 1000: seq 256, batch 2, 4 batches; logs/eval_w_ablation_step2855.json):

Every corruption costs more than at step 1000 (shuffle 16.3 -> 27.0, signs 10.7 -> 21.8, conversion W 6.63 -> 21.53, rec_scale=0 33.7 -> 90.4). The head specialised to the trained synapses. All ablations also cut the spike rate ~7x (0.024 -> ~0.0035), so part of the damage is the network going quiet, not only lost information.

Generations (still not good)

Oldest first; the last block is the current fly. Everything up to step 3440 is the language-only phase, read with that phase's head.

Step 14580 (current, greedy)

Once upon a time, there was a little girl who loved to play with her mommy and daddy. S
The little girl saw a big red ball on the grass and it was very happy. She wanted to go home, but she didn'
green test image                        A girl of a yellow field .

Test-sentence CE 2.76. Still fragments, still confuses he / she, captions still do not tell two photos apart. Earlier claims of degenerate text at 12040 / 14260 came from a generate-path bug (grey-field visual tokens attached to plain text); the same brains, run the way they were trained, read as above.

Step 2855

Greedy no longer stutters single words. It repeats a clause instead. Two of the five prompts collapse straight into dash / asterisk runs. Sampled text is word-shaped fragments, not English.

PROMPT: The
GREEDY: The, then the "It is the "It is again, the "It is after the "It is after the
SAMPLE: The) and a looking, and graphructs
Socing. Opt-Put, and we'p the entibolor, somethip

PROMPT: Once upon a time
GREEDY: Once upon a times, there are the "It is after the "It is aftly the -derstand and there is a s

PROMPT: In 2020, scientists discovered
GREEDY: In 2020, scientists discovered, the |------------------------------

Against step 2500: the the the and there is there is are gone, and are now agrees after a plural - a longer repeated unit, not a working sentence. Full log: logs/generate_step2855.log.

Step 2500

Greedy still latches. Sampled is slightly more word-like than 1k, still not English.

PROMPT: Once upon a time
GREEDY: Once upon a times, there is the | |---...
SAMPLE: Once upon a time about Artue. It work. ...

PROMPT: The fruit fly brain
GREEDY: The fruit fly brain, there is there is the |---...

Full log: logs/generate_step2500.log. Compare logs/generate_step1000.log.

Step 3440, as-trained vs Dale-projected

Exported brains (export_trained_brain.py) driven by the step-3440 head at spike cap 600. Greedy, 32 new tokens:

                     AS-TRAINED                                  DALE-PROJECTED (1,115,852 zeroed)
The                  The the "The. / The. / The. / The. ...      The to bet thenk...........nnn, epts. �sed to
Once upon a time     Once upon a times, the "It is there are      Once upon a times, �sedn, �sed. �sed to
                     there are the "The. / The. ...              thenn, �sedn the
The fruit fly brain  The fruit fly brains, there are the "It is  The fruit fly brains, �sed. �sed...........
                     after the "It is there are there are there

As-trained still latches. The repeated clause of step 2855 is back, plus a new The. line loop. Dale-projected now emits broken-byte tokens and dot runs: worse, consistent with its +1.70 CE. Full text: logs/generate_connectome_step3440.log.

What this does show

It can learn something. CE 9.71 -> ~4.8 by 2.56M tokens. w_grad live. Rate in band. Step-1000 ablation: destroying W wrecks CE.

Raising the spike cap engages more of the graph but has not moved CE. Cap 400 triples the firing population and doubles w_grad; 150 steps there ended at CE 4.94, inside the same band. The 3180-3440 ramp raised intensity, not reach.

Then it stops. From step ~1500 through 2855 - across an 8x sequence-length change, an 8x tokens-per-step change, and a switch to padding-free packing - CE sits at 4.8-4.9. A fit over 1500-2500 gives -0.0009 CE per 100 steps: flat. Whatever caps this is not step count and not batch size.

That learning is weak language. Greedy at 2500 is still degenerate. Last-50 mean CE ~4.84. Best single batch 3.96 is noise, not a 4-break (need a 100-step mean). 2.56M tokens is a probe, not a pretrain.

Most of the visual system never sees language training. 62% of the CNS. ~6% of optic-lobe cells ever trained. Joint vision (2949-14580): Flickr phrasing first, then VPN tokens + decoder + dopamine, then SmolLM2 judge + looped decoder. image_gain +0.114 on v4, positive on 100% of saves. Still not two-photo-distinct captions. Generate-path used to attach grey-field visual tokens to plain text; that was a bug, not the model.

Sugar survives. Bitter does not. Sugar -> MN9 still fires after language and vision training. Put back in the body at 14580, bitter -> MN9 is 0 Hz (original 21 Hz). Freeze the trained synapses and 30 simulated days of homeostatic scaling recover activity without moving language CE.

• Shows: sparse MaleCNS + SpikeWhale LIF accepts gradient and beats chance. Untrained optic lobe has a small motion/parallax signal. Daisy KL gap can close. VPNs can match DINO (~0.37). PAM/PPL1 looking-reward is when image_gain left noise and stayed there (table: ~0 then +0.047 then +0.114). That is measured on this brain. A scramble is not required to read it. Skipping never-spiking cells (39% of synapses kept) leaves captions bit-identical. Sugar -> MN9 survives. Bitter -> MN9 does not (0 Hz at 14580 in the body; original 21 Hz). Frozen trained synapses + homeostatic rest-of-fly does not erase language. Step 14580 test-sentence CE 2.76.
• Does not show: more steps => fluent text, two-photo-distinct captions, that any random 316+16 cells would match PAM/PPL1, fly-as-attention helping a transformer, looming, unbounded scaling as a safe "repair", a preserved bitter/aversive feeding arm at 14580, or dopamine vs eye-coupling 0.85 with dopamine off (that split was not run).

Known failure modes (already hit)

1. 1 tick/token: sensory atlas does not reach Kenyon. Anatomy is multi-hop.
2. Seq 512 + strong rec: Kenyon clique latches.
3. 1.1 current on Kenyon: greedy the the the.
4. Muon 0.02 on out: logit blowup.
5. Fly graph as attention (prefix-sum + hub CSR instead of MHA): ~50M Byrne-looped GPT, ~120M tokens, CE stuck ~6.4. Not this checkpoint.
6. Triton per-neuron d_spikes: ~110 s/step vs ~75 s cuSPARSE. Reverted; fire-only is forward only.
7. Long windows with one document per window: at seq 2048+ the single-document blocks() pads most windows, and pad tokens cost full LIF compute before being masked out. Pack documents instead.
8. checkpoint_every: 0 at long seq: the full tape is fine at seq 256, but at seq 8192 the per-token boundary state alone is ~44 GB. Segment it (checkpoint_every: 8).

Files

Load / generate

PYTHONPATH=code python generate.py \
  --ckpt checkpoints/real_seq2048_b4_step3440.pt \
  --device cuda --n-new 64 --temperature 0.8 --top-k 50

generate.py expects male_cns_spikewhale.pt at ../../output/male_cns_spikewhale.pt relative to experiments/anatomy_lm. In this snapshot:

mkdir -p output experiments/anatomy_lm/output/real_seq256_hubs atlas
cp checkpoints/male_cns_spikewhale.pt output/
cp checkpoints/step2500.pt experiments/anatomy_lm/output/real_seq256_hubs/latest.pt
cp -r atlas experiments/anatomy_lm/
cp config.json experiments/anatomy_lm/

Resume training

Same run name loads latest.pt. Optimizer moments are not in the ckpt.

cp checkpoints/real_seq2048_b4_step3440.pt \
   experiments/anatomy_lm/output/real_seq2048_b4/latest.pt
python -u experiments/anatomy_lm/train_anatomy.py \
  --start-training --device cuda --condition real --run-name real_seq2048_b4
# expect {"resume_from": 3440, ...}

The seq-2048 leg ran with sequence_length: 2048, batch_size: 4, gradient_accumulation: 1, checkpoint_every: 8 and contiguous document packing. At seq 256 the old config.json still applies (checkpoint_every: 0, run name real_seq256_hubs, resume from step2500.pt).

Do not copy an older checkpoint over latest.pt unless you intend to rewind.

License

Code: Apache-2.0, consistent with SpikeWhale-SNN-216M.

MaleCNS data: CC-BY 4.0 (Janelia / Cambridge / MRC LMB / Google Research). Cite Berg et al. 2026 if you use the graph.

@article{berg2026malecns,
  title   = {Sexual dimorphism in the complete {Drosophila} male central nervous system connectome},
  author  = {Berg, Stuart and Beckett, Isabella R. and Costa, Marta and Schlegel, Philipp and Januszewski, Micha{\l} and Marin, Elizabeth C. and Nern, Aljoscha and others},
  journal = {Cell},
  volume  = {189},
  number  = {18},
  pages   = {5504--5526.e15},
  year    = {2026},
  doi     = {10.1016/j.cell.2026.08.015}
}

The eight neuron types the model always had

code/model.py gives every cell a threshold, a leak and an input gain derived from its transmitter through three 8-vectors (thr_gain, beta_gain, in_gain). AnatomyLM freezes them, and they had been zero since step 0 - so all 167,565 neurons shared one threshold (1.0), one leak (0.9) and one gain (1.0). Eight possible neuron types, one used. The comment next to the freeze says why: a raised threshold on the cells the sensory hash injects into can silence that hash, because the inject is 1.1.

Fitting those 24 numbers on the frozen step-14580 brain (analysis/fit_neuron_types.py; every synapse, the head, the decoder and the eye held fixed):

Nobody told it which transmitters are inhibitory. It found that anyway:

Damp the big excitatory population, make the inhibitory ones easier to fire, and make the dopamine cells the most excitable in the brain - an E/I balance, from 24 free parameters, with the wiring untouched. checkpoints/vision_v4_step14580_types.pt, and the Space serves it as "Step 14580 + 8 neuron types" (same synapses as the 14580 fly, different cells).

What it is worth, honestly. The text is different, not plainly better (logs/neuron_types_generations.json, 6 test sentences and 5 prompts, greedy):

Once upon a time     one type : , there was a little girl who loved to play with her mommy and daddy
                     8 types  : , the was a little girl who loved to play with her. She wanted to go
One day, a boy found one type : a little girl in the garden. She was so happy that she had to go back
                     8 types  : a little girl in the garden. She was so excited and she wanted to go

CE fell on 3 of 6 test sentences and all 5 generations changed. So: a real but small gain, bought with 24 numbers and fifteen minutes, against ~29M tokens of training.

Training them further did not help (yet). A leg with the types trainable (vision/train_anatomy_vision_v5.py, batch 32, save every 10, thresholds clamped under the 1.1 inject) ran 270 steps to 14850. Caption CE and the Daisy KL both improved during it (4.28 -> 3.15, 2.04 -> 1.77), but test-sentence CE came out worse: 2.9975, and the spread between types halved (std 0.108 -> 0.051) as every threshold slid downwards together while the firing rate stayed flat at 0.030. Under a fixed spike-mass cap, lowering all thresholds at once is free - the cap renormalises - so that drift may be a degeneracy rather than learning. That leg also changed the text blend at the same time (adding IFM reasoning at 0.10), so it cannot separate the two causes. The clean version - types trainable, blend unchanged, thresholds re-centred each step so the spread can grow without the whole set sliding - has not been run.

Benchmarks at the last step (14580), for curiosity

Scored the usual zero-shot way (analysis/benchmark_lm.py, logs/benchmarks.json): for multiple choice, pick the option with the highest log-likelihood of the continuation, reported raw (acc) and length-normalised per byte (acc_norm), as lm-evaluation-harness does; for WikiText-2, byte perplexity over 40 windows of 256 tokens. 200 items per task, seed 0. Daisy-Base - a conventional small transformer, this fly's language teacher - is the reference column, from its own card.

At chance on knowledge, a few points over chance elsewhere. ARC-Easy is grade-school science. This brain has seen ~29M tokens of children's stories and photo captions. It has no facts to retrieve. 25.5% is what it should score. PIQA acc_norm 53% and ArithMark 29% on 200 items: suggestive at best. Not a claim.

WikiText byte perplexity is the honest number. No knowledge, no multiple-choice. 7.27 against Daisy-Base's 2.55. Wikipedia is only partly in domain (rehearsal is 40% TinyStories, 20% FinePhrase). Its own test-sentence CE is 2.76; byte perplexity there would be far lower.

Expected shape for 29M tokens through a fly's wiring. Here for completeness. Not because these numbers were going to look good.

Conclusion

In order.

1. The graph is the fly. MaleCNS v1.0 as released (Janelia / Google). 167,565 neurons, 25.6M synapses. Hard-LIF SpikeWhale cells. Train synapse strengths only. Wiring stays the fly's.

2. Language from scratch. Seq 256 to CE ~4.8 by 2.56M tokens. Ablation at step 1000: destroy W and CE dies. Then it plateaus (4.8-4.9 through 2855). Greedy latches (the the the, then longer repeated clauses). Spike cap 80 pins participation; ramping it recruits more cells but does not break the CE floor. Dale's law: language-only training flipped ~1.1M signs; projecting them back costs +0.22 nats at 2948 and +1.70 at 3440.

3. The eyes. 62% of the CNS. Language barely trains them. Photoreceptors placed from their own synapses, colour by opsin, both eyes, tonic lamina. Untrained optic lobe: small parallax signal, no looming, loses MNIST to a rewired twin.

4. Joint vision + language. Start 2948, Dale on. Full-strength eyes explode gradients (174 of 256 head cells are optic lobe). Hinge loss kills caption learning for ~1,700 steps. Visual tokens were the wrong cells until 10225. VPN tokens + DINOv2 + Daisy + decoder still left image_gain at noise. PAM/PPL1 on at 10761: image_gain leaves noise. That is the proof for those cells in this brain. Not a missing scramble. SmolLM2 judge, looped decoder / HRM / JEPA / MTP. Skip never-spiking cells (39% of synapses kept, captions bit-identical). Generate-path used to attach grey-field visual tokens to plain text; that was a bug.

5. Last saved step (14580). English fragments. Test-sentence CE 2.76. Greedy: , there was a little girl who loved to play with her mommy and daddy. S. image_gain +0.114. Frozen fly: 30 simulated days, trained synapses frozen, language CE identical (2.7627). Native Janelia-format connectome: exports/connectome-weights-male-cns-v1.0-fly14580.feather. Space serves that.

6. Still a fly, mostly. Sugar -> MN9 still fires after all of this. Bitter does not. Put back in the body at step 14580, sugar still triggers the proboscis; bitter-like taste -> MN9 is 0 Hz (original fly 21 Hz). Freeze the trained synapses and the rest of the brain can adapt around them, with language bit-identical. Unbounded homeostatic scaling brings activity all the way back to the original fly's (12.0%) and, at 14580, mostly spares the reflex (157 Hz) - but it destroys the language every time (CE 2.76 -> 6.61). Right activity is not right language.

7. Benchmarked, for curiosity. ARC-Easy 25.5% (chance 25), PIQA 49.5% / acc_norm 53.0% (chance 50), ArithMark 3.0 29.0% (chance 25), WikiText-2 byte perplexity 7.27 against Daisy-Base's 2.55. At chance on knowledge, a few points over chance elsewhere. Expected at ~29M tokens.

8. This is paused here. I answered most of the questions I had, and the benchmark above was the last one on the list. May try more later. Not a chatbot. A simulation of the released wiring that learned broken English and still works as a fly.

Citation

If you use this project, the code, the checkpoints, or the write-up, please cite Dean Byrne (Quazim0t0) and the MaleCNS paper above.

@misc{byrne2026malecns_snn,
  title        = {MaleCNS x SpikeWhale: training language and vision on the Drosophila male CNS connectome},
  author       = {Byrne, Dean},
  year         = {2026},
  howpublished = {Hugging Face},
  url          = {https://huggingface.co/Quazim0t0/MaleCNS-SpikeWhale-Anatomy-LM},
  note         = {Quazim0t0 / Dean Byrne}
}

Space: Quazim0t0/MaleCNS-Fly.