Data is science, as against intuition, which is a feeling and is subjective. Business decisions today are made on objective information: the data. But knowing which data to focus on is a decision. A/B testing is one such exercise that helps in making that choice before we commit all our efforts and resources to one option.

Understanding A/B Testing:

A/B testing is a user experience research methodology. A/B tests consist of a randomized experiment with two (sometimes more) variants, A and B. Through A/B testing, we use customers’ data to set two different versions of digital property like website, landing page, ads against each other to see which attracts more positive actions. It can be time-consuming but the advantages you get will outweigh the time that you have invested. By default, the A/B test divides the traffic into 50-50 between the original and the variant.

Benefits of A/B testing

How does the A/B test work?

One of the most important things in the A/B test is to decide what you want to test. Let’s say you want to test your ‘call-to-action’ button.

You want to show half of your visitors a Red login button and half of the visitors a Blue login button. The changes can be anything i.e CTA change, text change, page redesign, image change, banner change, etc.

The next step is to select the audience targeting (% of the audience that can see each set of variations) By default, the A/B test divides the traffic to 50-50 between original and variant.

A/B testing will show the two versions of the same website to different visitors randomly. One version is original and another is the variation where the changes are made.  After the test runs for a certain period of time,  you will get to know which variant resulted in more conversion.

The color of the subscribe button is a  basic example for understanding.  In reality, you might not be testing just the color but also the size, the text, the typeface, and the font size. 

Many managers run sequential tests for say size first, then color, typeface, and such. As managers want to combine the best of each test as the final result. But not all agree on this strategy.  Because it does not measure what happens when all selected factors interact in one single window.

Anyway, let’s continue understanding How-to implement your first A/B test.

9 Steps to implement Multivariate Test (MVT)

Step 1:

Go to your optimize account and create your container to run the experiment.

Step 2:

Click on the create experience CTA and enter the experiment name and editor page url.

Step 3:

Select Multivariate test in the type of experience

Step 4:

Now start creating the variants in the draft mode from here.

Step 5:

To create the variant, click on add variant under each section and enter the name of the variant 

Step 6:

Click on the edit option to open the editor window and make the changes as per your requirement and then click on save and then done.

Step 7:

Select the combination tab to see all the combinations that will be tested and Click the preview to see the specific combination

Step 8:

Once you are done with the configuration part,  you can configure the objectives, page targeting, and audience targeting.

Step 9:

Next. choose the activation event for your experiment and you are ready to go!

Variation A1B1

Variation A0B1

Best elements you can A/B test

Best practices for the A/B test:

Examine the behavior of your visitors to prepare optimization hypotheses. Make sure you target the correct audience. For better reliable results, don’t make or add any changes in the running test as your one change will be responsible for the change in the conversion.

A word caution: A/B tests are the most basic form of randomized studies and experiments. Simple randomized controlled experiments consist of two treatments, one of which acts as the control for the other. As with all randomized controlled experiments, you must estimate the threshold size necessary to achieve statistical significance.

A/B testing mistakes to avoid:

• Invalid hypothesis for A/B test
• To change the experiment settings or to add the new changes in the running tests.
• If you have plenty of pages then it might be possible that you test the wrong page.
• To run too many tests. Multiple results will add to the confusion.

Stay tuned! In our next release, we will share our thoughts about Firebase A/B testing in Mobile apps! Yes, because of Firebase it is easy to change app behavior and appearance of apps without requiring users to download an app update.