We think it’s great that lots of people are reviewing and scrutinizing DeepSWE. It being our first benchmark, we wanted to invite public collaboration and discourse, which is why we shared all our tasks and grading results in the open.
Since then, we've made lots of improvements in our methods internally and I'm very excited about the problems we’re better able to solve as a result. Notably, we’re at a point again where no benchmarks accurately reflect the capabilities of current models, and our upcoming work will be addressing this.
Show more
