Firebase: Learn A/B Testing In Applications – How to configure

With Firebase, you can perform A/B testing in applications with minimal effort and better results. It’s a mighty easy platform.  

This is the second part of the series to guide you through set and Testing! Part 1- Ways to implement A/B tests in Firebase.

How to configure the A/B test?

We are starting with Firebase A/B testing using the Remote Config feature. Set up the A/B test in the Firebase console. 

1.  Click the A/B testing item from the left side menu and click the Create Experiment button.

Firebase A/B test console

2.   Popup box will ask to choose from- Remote Config, Notifications, or In-App Messaging to configure an A/B test. Since we will be configuring the test via config parameters select Remote Config option.

Firebase A/B test experiment types

3. Create an experiment via Remote config:  

In the A/B test configuration screen, allow users to define and set up the A/B test with a couple of conditions and parameters to be satisfied. 

Here we’ll use the same parameters that were created in the “Set up Remote Config in Firebase console” section.

Below are the steps listed to show you how to configure the A/B test in the Firebase console using Remote Config.

Step 1: Provide the Experiment name and details.

Create firebase remote config experiment

Step 2: Condition to target users

In this section, we need to select the app in which we need to activate this experiment.

Target Audience Apply for firebase A/B test

Here, for targeting, take All users to be eligible for an experiment based on conditions we use.

Activation Event(Optional): 

An event that activates the experiment for a user. All targeted experiment users will receive an experiment variant treatment from the start of the experiment. Only those users who trigger the activation event will be included in the experiment measurement.

Firebase A/B test Activation event

Here, select Continue_Shopping(custom event) as an activation event to track how many users are clicking the shopping button, then click Next.

Step 3: Define Goals

Firebase A/B Test Primary Metric

By utilizing goal metrics (such as Crash-free users, User retention) plus maximum occurrences of events (such as daily user engagement, app_remove, first_open, custom events). 

Goal will be deemed a metric that can be chosen for the variant that best achieves the experiment leader as the primary goal. Click Next.

Firebase A/B test Additional Metrics

• Types of metrics to track 

Same as goals, there is a list of metrics available in the above screencap, at max 5 metrics can be used per A/B test. Distinct metrics as additional metrics (secondary goals) can be configured as compared to goals as secondary goals.

For e.g. for the test experiment, we have set Purchase revenue as the primary metric and in additional metrics, we’ve selected other metrics

Firebase A/B Test Goal setup

Step 4: Select variants

By using the parameters which are defined in Set up Remote Config in Firebase console section, multiple parameters are applied for variant setup with values for Control Group and Variants. Also, the A/B test has been configured as an alternative text and color shade for the shopping button.

• Types of variants

Baseline indicates App’s original view, whereas variants define the different variations to accomplish the goal. At max 7 variants are allowed for configuration.
Multivariate A/B tests can be achievable by adding parameters/clicking Choose or create new.

Firebase A/B Test variants

• Adjust variant weights

Variant weight refers to the distribution of variants to users. It will be in the form of a ratio among all variants.

Firebase A/B Test variants traffic allocation

Step 5: Click the Review button.

4.  Save the draft and review it and test in the testing environment

Test A/B test in Testing Environment

1. It’s a cherry on a cake.

Firebase leverages users to verify A/B tests in the device before making it live. Click AB test tab -> Select your AB test from the drafted section -> Open settings using 3 dots  -> Click

Manage test devices.

Run Firebase A/B testing on test device

2. Provide device instance ID and select the variant which you want to test. To get Instance IDs for Android and iOS:

     a. Android snippet to fetch Instance ID as below:

    b. iOS snippet to fetch Instance ID as below:

3.  Add ID in the token box, select the variant which you want to verify. Click on Save.

Run Firebase variant A/B test on test device

4.  Below is our example screens with changes visualized:

Firebase A/B Test Variant vs Baseline

Deploy Experiment in Live Environment

Yeah! It’s time to make experiments live across all users to provide easy access and engagement. It’ll help us in deciding whether to roll out the variation in the live environment or not.

1. In the Firebase console, on the left side menu clicking the A/B Testing option will show a list of running experiments, drafted versions that are in review/testing environment, and completed A/B test in case any.
2. Our experiment will be visible in the Draft section as it is not yet published. Click the experiment title and open it. Experiments can start in two ways directly: one is from the Drafted section. Then, click the start menu.
3. 

   Firebase A/b Test starting to experiment

4. With this, we come to an end of our PART-2 of Firebase: A/B testing series. In the next part we will be covering Notification set-up, In-App messaging and exploring Multivariate A/B testing. Meanwhile, you can start implementing and reach out to us in case of additional queries at hello@tatvic.com. And we would be, as always, happy to help.