Register and share your invite link to earn from video plays and referrals.

Muyu He
@HeMuyu0327
RL @ZyphraAI | Expert of mixtures, or 21st century schizoid man.
Joined September 2023
322 Following    4K Followers
As I take it to be a central conflict in transformers that value matrices want token embeddings but cannot get them, I want to see if an AttnRes style model can "get them" by learning per-layer weights. But still, token embeddings make no contribution to their value matrices. This reveals a fundamental constraint in transformer design -- that token embeddings are forced to have tiny norms. In the AttnRes model, we can see that although value matrices learn very large (in softmax attention like weights, that means close to 1) weights, their contribution to the value vector output norm is tiny. This means that even with the ability as well as the motivation to upweight token embeddings, they still cannot have an impact. If we plot the L2 norm of each layer output, it is pretty evident why. We do norm(emb) immediately before the first layer, so token embeddings will have constant L2 norm equal to hidden_size. By contrast, later layer outputs do not have normalized length, so they become unbounded and can surpass token embeddings' norms. In AttnRes-style model, both the paper and my experiment show that aside from the token embedding, other layers now have similar norms, instead of the exponential blowup in standard transformer. Yet because token embeddings are normalized, they will be instantly overwritten by any subsequent layers. It is therefore a good experiment to see -- if AttnRes style model learn unbounded scalar weights for token embeddings, will it be very large and how large will it be? It will be pretty cool if it does become large and solve our no-contribution problem. That will be the call to do some arch mod. Ongoing work. Blog:
Show more