my days are like `N` hours of working with frontier models, babysitting them in frustration and disbelief at the frequent errors and slop
interrupted by `M` min breaks of browsing X takes about how the same models (and _especially_ the next gen!) are superhuman at ~everything
Show more
to be clear: @LightOn's mLateOn, @LightOnIO GTE-ModernColbert, @LightOnIO Colbert-Zero, @LightOnIO LateOn.
We do like to train some late interaction models
After topping Korean MTEB without being trained on Korean ( mLateOn now tops another leaderboard that includes languages it has not been trained for and outperforms much larger models (and even rerankers)
mLateOn benches results are strong, but what's not evaluated by current benches makes it even stronger
Show more
In-context continual learning requires models to accumulate experience and reuse it later in the same sequence. But an RNN compresses an ever-growing history into a fixed-size state, where each token gets a single write into memory.
We study dynamic compression: letting the model revisit the past and reorganize its state as it discovers what needs to be reused.
Show more
We're composing one of the most strongest team focus on retrieval now, super excited to work with @antoine_chaffin on a daily baisis, a lot more coming soon!
one of the best minds in this space
i thought i knew just how much @antoine_chaffin and @LightOnIO have done in such little time, but wow - the list gave me a pause.
congrats @antoine_chaffin and SO excited for how this accelerates @perplexity_ai’s recent releases even further!
Show more
Personal update: I've left @LightOnIO to join @perplexity_ai
Personal update: I've left @LightOnIO to join @perplexity_ai
Intuition: Compaction is agentic recurrence (RNNs), whereas recursion (RLMs) is agentic attention.
Recurrence maintains a constant-size state at all times by compressing the past.
Attention keeps all context fully represented and then decides how to re-process it for each step.
Show more
Awesome blog! And they evaluated on the congress ToT split of OBLIQ-Bench
@NoahZiems We are happy users at Weaviate! 💚
Between BRIGHT and OBLIQ-Bench I think the retrieval community is "benchmark-rich" for now 😆
*I should caveat this by clarifying that we have just tested on the Congress subset so far.
Show more
@jeremyphoward @answerdotai He's right! 200k monthly downloads and not for nothing, here's the link for those who want to try it:
The amazingly fast (only 30M!) and extremely accurate @answerdotai ColBERT model is now supported in Sentence Transformers to make it easy to create and query embedding indexes locally directly in Python! 🥳
Show more
🚨I've just released Sentence Transformers v6.0!
MultiVectorEncoder joins the family: ColBERT-style late interaction models are now a first-class model type, for training, inference & interpretation, alongside dense, sparse & reranker models.
Big thread 🧵
Show more
Good news for the late interactors 🤓
Sentence Transformers is the best option to train a multi-vector model from now on ☺️
We choose to build PyLate on top of ST to benefit from all the hard work from @tomaarsen
Since our work has been very convincing and impactful, it ended up back in ST, which I'm very proud of
Show more
Using the best embedding/retriever models just became easier!!
And STv6 supports ColPali-like models out-of-the-box, with beloved features like token pooling and similarity maps for interpretability!
🚨I've just released Sentence Transformers v6.0!
MultiVectorEncoder joins the family: ColBERT-style late interaction models are now a first-class model type, for training, inference & interpretation, alongside dense, sparse & reranker models.
Big thread 🧵
Show more
@jacobli99 this is a great blog post
