"in 6–12 months such a swarm could be capable of taking over the entire internet with a persistent botnet" Maybe, maybe not. If it does, the internet (really: software and operating systems) obviously need some improvement and we should do so. If the bugs are there, they will be found, one way or another regardless of some LLM. What a great opportunity for a renaissance. Anyone else remember the doomerism around Y2K? Int overflows because of epoch seconds? This is that.

The push for regulatory capture and AI cabalism is accelerating. This, in large part, is why I'm building parallax. True P2P unstoppable, uncensorable AI systems aren't just neat, they're mandatory. This little single DGX spark "datacenter" in my forge with parallax can train (with other nodes) an 80b parameter model. - two solar blankets - ecoflow battery - lapdock for kvm - starlink internet - dgx spark Total input from solar ~600w (800w max) Total power draw from spark/starlink/etc. < 200w No datacenters, no water supplies, no external power grid, no multi gigabit link. We must design AI systems that are as resilient as the humble cockroach. Why do we all appreciate BTC? Because it's a mathematically certain sovereign ledger that can't be inflated away or centrally controlled. Current AI represent the greatest concentration of wealth, power, and information (IP theft? blackmail? or simply ads and behavior control?) the world has ever seen... As BTC did to finances, we must do to AI. Chutes + parallax is a step forward towards tearing down this moat.

Remember, the main point here is not even the decentralized training (even if it's what I'd consider a step change). The real magic is in the inference optimizations it enables. (sparse fp4 native compute for routed experts, fixed kv cache for most of the attention, sparse latent for the rest, tiny model weights, etc.) This is just from an unoptimized vllm branch on our model arch, not even really tuned yet. Maybe not an entirely fair comparison need a lot more models to compare against and some nuance and so on, but this is the whole point. Limitless tokens on cheap hardware.

And if you're wondering why I used 5090s for this, it's because they are the worst GPUs on earth for stability at this utilization and have like 50% failure rate in my experience thus far (at least 1 of 8 dropping off bus or producing NaNs randomly etc.). Stress test.

Dashboard is a quick work in progress, but for visibility into a run here ya go! This is around $11/b tokens, insane actually. MFU also insane. The whole thing, pretty legendary, and inference... Using a few nodes from @lium_io also!

Super excited to share the results of my deep, dark research cave over the past couple months building out parallax at @ExploitSummit Someone asked me if we had a breakthrough - couldn't even condense a response. The benchmaxxing era is (or should be) over, efficiency-maxxing is next.

Ternary and sparse FP4 are a match made in heaven. 2x the performance, cost is basically nothing in terms of quality since ternary already has natural sparsity. AND, you can pack those params 1.15 bits per param instead of 1.58. A petaflop... at 200 watts... Almost magic.

NVIDIA B300 servers retail around $350k min, crazy power/cooling requirements/etc. (bad maths) per GPU (8) is $43,750, if you can even find them (narrator: you can't) then pay for collocation etc... 9 PFLOPS dense FP4. DGX Spark: $4700, plug in anywhere, 1 PFLOP (fp4 sparse). B300 9ish PFLOP dense FP4 super expensive impossible to collocate anywhere etc. etc., $4861.11 per PFLOP. DGX Spark $4700 plug in any 15 amp outline in any house, $4700/PFLOP (but sparse). Memory is slower, so I guess use less of it/less manipulation of it... Yep, math checks out - pretraining on decentralized DGX Spark with native sparse FP4 (checks notes: yes we're using ternary weights which slot perfectly into sparse FP4 compute) should be a thing. Time to test it out...

So engy starts as explicitly non-TEE inference, but better for some reason because each instance requires 80 gpus that are flaky and slow over 25gbps interconnects. Then they realize TEE is necessary for any real use case, but TEE without open source/fixed/attested workloads is pointless because you could just change the stack to log whatever you want otherwise. So now it'll be... a chutes clone? But without any transparency, run by an anon and his many alts and lackies who are in a perpetual smear campaign against chutes? How many times to tao holders need to be rugged/deceived by anons with huge promises? Careful out there.

TL;DR: libp2p TCP transport with dual trainer/syncer roles = chef's kiss For model training, don't bottleneck and single-point-of -failure yourself with a blob store (S3/R2). Also, don't bash your forehead against a wall using listening sockets on nodes that may be behind firewalls/NAT/port mapped containers/etc. Just set up a separate backbone (sync only, non-GPU/training nodes) across the world with super fast WAN and use push only from training/GPU nodes to this layer. Easy peasy, works like a charm. And bonus, libp2p's TCP transports are reliably better across providers/networks/countries compared to default quic/udp. Skynet?

Napkin math on bits per byte vocab adjusted etc, this is also beating olmoe 7b 1ba at matched tokens (which is 30% more active params, 40% more total params). Maybe competitive with MobileMoE if we did a complete several trillion token severely overtrained run. Quantile balancing LatentMoE Hybrid GDN-2/SWAX/MSA Role differentiated mixed precision I think we have a winner. More napkin math, I think this 5b model would be able to do around 1200tps inference on 5090s. Kinda want to do the full 9t token run on some 5090s of this tiny one, for funsies. Parallax-5b-instant (or we could bump total params up to 10b for free if we don't change active params, or maybe higher)

167k tokens per second training throughput on a single 8x 5090 box with parallax (on a toy 5b 1b active test moe)... That's quite high. Every GPU you add reduces compute for the others so that number goes up > linearly. Exciting times. And look at the beautiful loss curve, beating DDP baseline at matched steps (identical warmup/lr/dataset/etc.) With 5.99 GFLOPS per token for this arch and BF16 denominator that's ~60% MFU equivalent. 3.068 nats so far at ~19b tokens (llama-3 tokenizer), beating the DDP baseline at matched token budget and getting there ~15% faster and 64% cheaper, and remember it's ternary routed experts... Best guess as to nats gap is separate expert muon optimizers with the surrogate feedback+fold normalization etc. Almost ready for the big runs...