I was really lucky to come across a very interesting phenomenon in complex neural networks, so I wanted to share it with you. It’s called the Phase Coherent Transformer. Under specific conditions, complex-valued neural networks (CvNNs) exhibit significantly better generalization and superior performance across multiple tasks. I’ve written a detailed experimental report in the paper.Until now, CvNNs were thought to be good mainly in a narrow range of tasks like signal analysis and phase information processing. After Transformers and attention mechanisms came along, the gap with real-valued networks only kept widening. In fact, in this experiment as well, turning a standard Transformer into a complex version mostly just led to performance degradation (except for phase-related tasks). But now, This transformer seems really suited for complex neural networks, unlike before, and it’s showing completely unexpected behaviors and impressive benchmark results. I suspected that “the softmax in Transformers might be causing the loss of phase information in complex weights across multiple layers.” So I looked for attention structures that can preserve phase information, and ran experiments. Approaches like Sigmoid attention and the recently announced Multiscreen have this kind of structure. After complexifying them and tuning for better multi-layer phase control, I named the resulting model the Phase Coherent Transformer (PCT).The experimental results showed that it greatly outperforms many real-valued NNs on tasks where complex NNs had traditionally struggled, while still retaining its advantage on phase-related tasks. Paper preprint: It’s not yet clear how this will impact AI overall, but I think it’s a pretty significant development for the field of complex neural networks at least.This naturally brings up one important question: Q: Should today’s AI stay based on real numbers? In my field, circuit-style cryptography (ZKP, WE), the algebraic structures used as function inputs have naturally expanded into extension fields when necessary. It wouldn’t be surprising if AI follows a similar natural path. Moreover, CvNNs can likely associate and infer from data (=responses and signals from circuits or functions) under the assumption that phase information is hidden inside them. This feels incredibly powerful, because it could allow much more sophisticated handling not only of circuit outputs but also of intermediate state traces.