A/B testing takes away the guesswork from the process of improving games.
In games, at any one time, countless variables have an impact on each player’s experience. That variability has a knock-on effect and makes optimization a much more complex undertaking than it might seem.
If you want to improve any aspect of your game, you need technology that isolates the aspect in question and tests your “improvements” effectively. If rigorous A/B testing bears out your hypothesis in specific conditions, on a subset of your total audience, you can roll it out on a larger scale.
What is an A/B Test?
An A/B test essentially compares different versions of the same variable to determine the impact of changes to that variable. The idea is to run concurrent tests in which conditions are as close to identical as possible, save for the variable being tested.
The control group continues to experience the “default” behavior and the treatment group receives the new behavior, so that you can compare the results and determine the impact of changing your chosen variable.
Why A/B test?
Testing changes in this way is important for several reasons:
- By testing changes on a small portion of your population, you minimize any negative impact of rolling out a change.
- You can statistically prove the effects of your proposed change, instead of relying on gut feeling.
- By deciding on what constitutes “significant” change before running the test, you can be objective in your assessment.
How to perform a successful A/B test
Testing is an iterative process. The diagram below breaks down A/B testing step-by-step.
A few key factors are crucial to running a successful A/B test. Stick to the following rules, and you can’t go wrong:
Patience and discipline are key to running a successful A/B test. Ending a test early, because you perceive a significant outcome appearing, will not give the best results. Wait for the test to run its full course.
You can only draw reliable conclusions from tests that action one change at a time. As soon as multiple variables are at work, you can’t accurately evaluate the precise impact of any of them. It’s one at a time, or not at all.
Always use the same audience
Your control and treatment groups must be made up of identical audiences, or the whole exercise is pointless. Geolocation, demographic, and play style are useful markers for segmentation because such things differentiate player profiles. If you want to know the true impact of a change, it must be tested on identical subjects.
A/B tests with Unity
We already have A/B testing functionality within the deltaDNA platform, and it is an integral part of optimizing everything – from game difficulty to in-app purchase (IAP) offer value. With cross-platform and rich data capability, this end-to-end solution enables publishers and developers to better understand different player behaviors and create personalized experiences, targeting individual players in real-time. Get a full-feature 30-day free trial of deltaDNA to discover what automated A/B testing can do for your game!