💻 Even top-tier coding AIs see their scores collapse by more than half when the user isn't a programmer. A new benchmark puts that uncomfortable fact on the table. Title: SWE-Journey: Towards More Realistic Evaluation of Coding Assistants through Long-Horizon, Multi-Turn Interaction URL: ❓ What was the problem? Existing coding benchmarks tend to use short tasks with the full spec handed over up front, which doesn't match how real users gradually reveal requirements and react to what the assistant does. ❓ What does SWE-Journey do differently? It auto-synthesizes long-horizon tasks (averaging 10.2 changed files and 768 lines) and has AI play four personas drawn from real Claude conversation logs - non-coder, product manager, new developer, and software architect - each chatting with the assistant over dozens of turns. ❓ What did they find? Even the best model only reaches 52.22% Final F2P (final feature-implementation success). It hits 78.50% with the software-architect persona but drops to just 23.00% with non-coders, a 55.50-point gap. ❓ Why such a big gap? Weaker models rarely ask clarifying questions when requirements are vague, stay overconfident about the wrong part of the code, and sometimes report a fix as "done" with no real evidence. How well a model adapts its questioning and verification style to the user turns out to matter as much as raw coding skill. #AICoding# #LLMEvaluation#