OpenAI research scientist @aidan_mclau warns the real risk isn't a mode collapsed model, it's a mode collapsed world where every AI conversation converges to the same answers: "AidanBench is a benchmark end of 2024 that measures model creativity. What we'd do is ask the models a question. Like, how do I solve traffic in Los Angeles? One of the answers would be like, put more lanes in your freeway." "Then we would ask for another answer, and another, and another. We would do this hundreds or thousands of times, so the models would have to come up with a thousand unique solutions to traffic in Los Angeles." "One of two things would happen. The first is they would say something purely nonsensical, like a great way to solve traffic in Los Angeles is have everybody drive a flying car." "The more common thing is they would just repeat themselves. Their 500th answer would just be to add more freeways again." "The models that could spit out the most unique answers without mode collapse really aligned with people's intuitions for frontier model capabilities. The models that scored best did feel like the most creative." "It's important to make sure the model you talk to isn't just doing the exact same thing as the model another user is talking to. This could lead to a mode collapsed world, not just a mode collapsed model." @OpenAI