A/B tests take weeks to run. What if real-data-driven AI personas could predict the outcome in advance?
Title: Data-Driven Persona-Conditioned Agents for A/B Test Simulation
URL:
โ What question format works best for AI personas?
๐ก Pairwise evaluation, where the agent directly compares variants, wins clearly: 0.75 accuracy on CTR and 0.80 on subscription tests. Independent scoring drops to around 0.40.
โ Do you need proprietary data to build good personas?
๐ก Open e-commerce data matched proprietary data almost as well. But out-of-domain data (movie reviews) hurt accuracy, showing domain match really matters.
โ Is it better to go deep on behavior data or wide on demographic diversity?
๐ก A "deep pool" of highly engaged users won for CTR prediction, but for subscription tests, a demographically diverse pool performed just as well.
โ Can you cut costs by using fewer personas?
๐ก Yes โ subsampling from 935 down to 500 personas kept accuracy nearly unchanged, roughly halving simulation cost.
#
ABTesting# #
LLMAgents#