Looped transformers are a popular architecture topic right now. This new technical report extends the loop across tokens. Recurrent Looped Transformer (RLT) makes the decoder recurrent over every token, prompt and response included. A causal encoder builds global KV memory. For each new token, the decoder combines the token's encoder representation with its own final hidden state from the previous token and a sliding-window cache of recent activations. With a 48-layer decoder, the computation path after t tokens runs through 48t decoder blocks, while each token still executes a fixed number of blocks. Depth grows with the sequence and per-token cost stays the same. The same state transition is used for pretraining, SFT, sampling and RL replay, and nothing resets at the prompt-response boundary. RL replay rebuilds states under the current weights instead of reusing stale rollout states. The report is a design proposal. The author states that reasoning gains, hardware speedups and RL scaling are goals that have not been measured yet. Paper: Chat with Paper:

Sentence Transformers v6 has been released, and it is centered around Multi-Vector Embedding Models, what does that mean? Usually, when you build a semantic search or RAG-based system, you've probably used the @OpenAI embedding API or similar, as it allows to quickly index and search through a set of documents. Such embedding models are called "bi-encoders", as they typically encode 2 things ("bi"): encode each document or chunk, encode the query, and then compute pairwise cosine similarity between each (document, query) pair to retrieve the top-k ones. This is known as vector search or semantic search, and typically achieves better results compared to traditional keyword search, which relies on the classic BM25 algorithm. In 2020, researchers at @Stanford came up with something better than bi-encoders, called ColBERT. The idea is to create many vectors for each document, and many vectors for each query. One creates a vector for each token (a word or part of a word) of the query, and a vector for each token of each document. Next, to find the best matches, one computes the so-called MaxSim similarity. This is illustrated in the animation below. Unlike traditional vector similarity metrics that operate on pairs of single vectors, MaxSim computes similarity between sequences of vectors. The key insight here: each query token finds its best match in the document, then we sum. This enables fine-grained semantic matching and avoids averaging. These "multi-vector" embedding models are also called "late-interaction" models, as they keep multiple vectors for each document and query, delaying the matching step until the very end. Over the last year, researchers at @LightOnIO trained some very impressive late interaction models, which they openly released on @huggingface. Today, the Sentence Transformers library, which is the go-to library for open-source embedding models, added first-class support for them, as well as for a variety of SOTA late interaction models trained by @mixedbreadai, @liquidai and more. This means that you can now also train or fine-tune your own late interaction models very easily. Various vector databases like @elastic and @qdrant_engine already include support for multi-vector embedding models. This integration will definitely boost adoption of multi-vector embedding models by the industry. Read more here: - ColBERT paper: - Blog:

Hasbro Transformers Generations Power of the Primes Throne of the Primes Optimal Optimus Reissue is up for preorder at Entertainment Earth ($99.99 + free shipping) - #ad#