A/B Test your App using Firebase Remote Config

Last week I was in Nairobi and Cape Town giving a talk on Remote Config and Test Lab in Firebase. I had such a great time and thought I should share some of the content I presented online.

What is A/B Testing?

A/B testing is the process of experimentally testing your UI on different audiences in order to determine the best (or most profitable) user experience.  Firebase Remote Config enables us to do A/B testing by allowing us to randomly segment our app audiences easily.

Example A/B Test

In this example, we are going to test out a different version of a checkout button that we have in our app.

A/B test we will run using Firebase Remote config.

For variant A of the experiment, we want to show a pink “Checkout” button. For variant B we want to show a blue “Cart” button. For the rest of our users, we will show the original “Finished” button.

Now for the fun part, setting up the experiment![adwords_square]

Set up Firebase Remote Config for A/B Testing

  1. Create a Firebase Project.
  2. Head to Remote Config section.
  3. Create a parameter – let’s call this parameter experiment_variant. Make the default value return no_experiment.Setting up parameter in Firebase Remote Config
  4. Click “Add value for Condition”. Then click “Define a new Condition”. Give it the name “Variant A”. We will select “User in Random Percentile” as the conditional. We will then choose the users from 0% – 10% that will receive variant A as their experiment. Click “Create Condition“.Adding a A/B Testing conditional to use with Firebase Remote Config
  5.  Now you will see that conditional appear for the experiment_variant parameter. Assign  variant_a to the value that should be returned for that variant.Conditional in Firebase Remote Config
  6. We will repeat the same for variant B, by creating a conditional for the users from 10% – 20%. This means that only 20% of the entire user base will be running the experiment and 80% will see no change in the standard UI. The percentage of the experiment is totally up to you. Variant B for Firebase Remote ConfigThen return variant_b for the parameter value:All variants in Firebase Remote Config
  7. Make sure to update and publish the changes. We have completed the server side setup of our experiment.

Setting up the A/B Test on Android Devices

Once you have configured the server, the hard part is done! The next section will cover how to get setup for Android but the code is really similar for iOS too – so don’t be deterred!

  1. In your android app, make sure you have the Firebase dependencies set up. Place the following in your top level build.gradle:
  2. Add the following in your app level build.gradle:
  3. Download google-services.json file from Firebase console and add it to your app/ folder.
  4. Create a remote config defaults file in the res/xml folder like the one below. This file will be used if the user has no internet connection when loading up the app for the first time.
  5. Initialise Firebase remote config. This is also where we set the defaults to the file we created in the previous step.
  6. Then fetch the remote config values. Firebase will automatically cache the values for the specified time that you provide. By default, it caches for 12 hours.
  7. In order to run our experiment, we should fetch the experiment_variant parameter from our remote config and use it in our app.

    In this example, we fetch the parameter and set a custom analytics user property with the variant that has been assigned to the current device. This will allow us to properly track how our A/B test is performing.  In the conditional, we are free to make the changes we want to the app. We can set the text to different things and change the colours depending on the variant.
  8. Now that we have the experiment running, we need to track when a user clicks on the “Checkout” button in order to determine which button variant a user is more likely to tap on. We then log the event to Firebase analytics which will include the user property that we set previously.

    It is worth noting that there are a bunch of default events and parameters that you can make use of with Firebase Analytics. [adwords_square]
  9. In order to determine the success of your A/B test, you need to analyse your data to decide which UI is the most successful. After logging into the console, we can see there were 24 counts of the “CheckoutClicked” event.firebase_analytics_ab_testingAfter applying the filter for the user property “Experiment = Variant A”, we can see that 19 of the 24 clicks were from variant A of the experiment (granted this was just me clicking around in my sample app – but you get the idea).

Variant A filters - Remote config

For larger data sets where the result of the experiment may not be obvious at first, you can export your events to BigQuery. This will allow you to make more complex deductions from your experiments.

For now at least, we know the clear winner (in this very large test 😬) is Variant A – the pink checkout button wins!

AB Testing variant A winner

There you have it, an easy way to experiment with your UI using Firebase Remote Config.

Sample code can be found here (It is not very complex as all the hard work is done by the Firebase APIs.) 😊

Happy AB Testing!

Subscribe to Blog via Email

Enter your email address to subscribe to this blog and receive notifications of new posts by email.

Introduction to Automated Android Testing – Part 6

In the previous 5 blog posts, we covered different aspects of building an Android app from scratch. We focused on including tests in the process. Here are the links to the previous posts:

  • Post #1 – Why should we write tests?
  • Post #2 – Set up your app for testing
  • Post #3 – Creating API calls
  • Post #4 – Creating repositories
  • Post #5 – Following the MVP pattern

In this last post of the series, we will cover creating Espresso tests for the View we created in part 5. The Github repo for this post can be found here.

Testing that a view contains the exact information expected can be tricky if the data is dynamic. This data can change at any time and our tests should not fail because of it. In order for the tests to be reliable and repeatable, we should not call any production APIs.

Mocking out the responses of the API calls will enable us to write tests that depend on the mocked data. There are a couple of ways in which we can mock out our API calls:

  • Option 1 – Use WireMock and run a standalone server which serves up the same static JSON for specific network calls.
  • Option 2 – Use OkHttp’s MockWebServer which runs a webserver on your device and serves up any response you request.
  • Option 3 – Create a custom implementation of the Retrofit REST interface that returns dummy objects.

Obviously, the choice is entirely up to you as to how you would want to go about writing UI tests. In my case, WireMock is extra effort as I would need to ensure I have a standalone server running with a static IP address.

MockWebServer is a lot easier to use than WireMock as you don’t have to set up a standalone web server (The server runs on the device). MockWebServer is also flexible because you can give it different scenarios. Useful features like specifying the failure rate of a certain call or simulating a slow network are possible using MockWebServer. (Read more here).

I am going to use option 3 for the purpose of testing that the UI matches the mock response data. If I wanted to add tests for slow network conditions (or some kind of non-functional test), I would choose option 2. If you are unable to use OkHttp,  I would choose option 1 as Wiremock works with any HTTP client.

Mocking out data using Gradle flavors

By making use of Gradle flavors, we can easily mock out of API responses.  If you read post #2 on Gradle flavors, you should already have a “mock” and a “production” flavor set up.

  1. Make sure you are switched to the mockDebug flavor. Select mockDebug variant
  2. Create a mock folder in the src directory. Then create a package within the mock folder, that mimics the main package name. Make a class called MockGithubUserRestServiceImpl. Your resulting file structure should look like this: Folder structure for mock testing
  3. Create a prod directory. Move the Injection class defined previously into this folder. We will be creating another Injection class in the mock folder. This class will inject the mocked out Github service instead of the production API. Move Injection to prod and create one in mock folderIn the Injection class that is located in the mock folder, we simply return the MockGithubUserServiceImpl that we created. In the prod folder, we return the actual Retrofit Github service. [adwords_square]
    Mock Injection class:

    Prod Injection class:
  4. The data returned from the mock service is dependant on your specific requirements. Below is my implementation of the MockGithubUserRestServiceImpl:

    In this case, I am just returning some dummy data. Let’s run the mock version of the app and we should get the same results no matter what you search.

    Android Test app with mock data

    Cool. Now we have a working dummy app! We can now write Espresso UI tests. [adwords_square]

Basics of Writing an Espresso Test

When writing an Espresso test, the following formula is used for performing functions within your UI:

  • ViewMatchers – Used to find a view in an activity. There are a bunch of different kinds of matchers. For example: withId(R.id.menu_search), withText("Search"), withTag("custom_tag") .
  • ViewActions – Used to interact with a view. For example: click(), doubleClick(), swipeUp(), typeText().
  • ViewAssertations – Used to make assertions that certain views possess specific properties. For example:   doesNotExist(), isAbove(), isBelow().

There is a great cheat sheet for the different Espresso methods that can be found in pdf form here: android-espresso-testing.pdf. It is worth mentioning that ordinary hamcrest matchers can be used when writing Espresso tests. Methods such as not(), allOf()  and anyOf() are valid.

Writing Espresso UI Tests

If you can recall, in post #2 we covered what dependencies will need to be added in order to write espresso tests. Now we will cover writing an Espresso test.

  1. Create a folder androidTestMock. The tests in this folder will only run on the mock variant and not on the production variant. Then make a directory that matches the main package name. In that directory, add a new class called  UserSearchActivityTest. Your project should then look like this:androidTestMock folder
  2. We will start by writing a basic test that ensures that when the activity is started, the text “Start typing to search” is displayed:

    The @Rule  ActivityTestRule specifies which activity this test will run with. In this case this test will run with UserSearchActivity. This will automatically start up the UserSearchActivity. Passing extra parameters will indicate if you want the activity to auto start or not.

    The test searchActivity_onLaunch_HintTextDisplayed() is quite simple. It searches in the view for the text and asserts that the text is visible on the UI.

  3. The next test is slightly more complicated:

    After typing into the SearchView and pressing enter, we assert that the dummy results are displayed on the UI.

  4. We have now written tests for the positive scenarios, we should add a test for the negative case too. We will need to adjust the MockGithubUserRestServiceImpl in order to allow it to return custom error observables if required.

    In the code above, a method was created in order to set a dummy observable for the search results. That observable will be returned if it is not null when searchGithubUsers() is called.

  5. Now we can create a test that checks if the error is displayed on the UI.

    In this test, we first ensure that the service will return an exception. Then we assert that the error message is displayed on the UI.

  6. Let’s run the tests:


They all pass!

Code Coverage in Android

In order to know how effective your tests are, it is great to get code coverage metrics.

  1. To enable code coverage on your UI tests, add testCoverageEnabled = true  to your build.gradle:
  2. Run the task createMockDebugCoverageReport. You will then find the HTML report located here: app/build/reports/coverage/mock/debug/index.html.

Code Coverage Report in Android

Yay – we have 82% coverage just with the mock UI test. Taking into consideration the coverage reports we saw in post #4 and this post, it gives us a good indication of the test coverage of our entire app. Now we can iteratively go back and try cover more areas of our code.

PS – Code coverage currently doesn’t work with the Jack compiler. I switched to use Retrolambda in order to get the code coverage report to work. If you are interested in learning more, check out this branch.


We have finished writing our feature. Whew! 6 blog posts later. There is obviously a lot more testing that can be completed in this app. Non-functional tests such as testing how your app behaves on devices with low memory or with poor network connectivity can also be added.
That concludes the series on an “Introduction to Automated Android Testing”. I hope you enjoyed reading this series. If you enjoyed it, be sure to subscribe to the blog to receive future updates and share the post with your friends.

Further Reading

Subscribe to Blog via Email

Enter your email address to subscribe to this blog and receive notifications of new posts by email.

If you would like to see more posts from me, please consider buying me a cupcake to keep them coming.


Android Data Binding – Part 2

In Part 1 I gave an introduction into the Android Data Binding Library. I suggest reading that post first before reading further.

In this post I am going to discuss the following :

  • Binding Adapters
  • Binding Events
  • Two Way Binding

Binding Adapters

Creating an adapter and using Android Data Binding for each item is quite simple. Below is an example of how your RecyclerView adapter will look after using Data Binding:

  1. The first difference in this method is that our ViewHolder  object now only contains a ViewDataBinding  object (instead of ImageViews and TextViews)  and a method to get the binding object.
  2. The method onBindViewHolder binds the view to the item variables. This is shown by the lines holder.getBinding().setVariable(BR.book, bookDetail) . This sets the views book variable defined in the layout XML.
  3. The method holder.getBinding().executePendingBindings()  triggers the View to be updated with the new values provided. This method has to be run on the UI thread.
  4. Within our list_item_book.xml  layout file we define and access variables in the same way as in Part 1. With the <data> section defining all the variables to use in the xml, including the imports. The Views then contain reference to those objects. For example  android:text="@{book.bookTitle}"  accesses the books title and assigns the value to the text of the TextView.


In the same way that we bind variables in XML, we can set variables for click listeners in XML.

  1.  Create the click listener in code:

2. In list_item_book.xml, define the View.OnClickListener  and set the View that the click should fire on by using the variable defined android:onClick="@{click_listener}" . This will then trigger the click event.

3. In the adapter, set the click listener:

I ran into an issue whilst trying to set the Views Tag in XML due to a bug in the Android Data Binding Library (they did warn us it’s in beta 🙂 )

The following XML causes an issue with the Data Binding Library that will cause your app to not compile:

An explanation from StackOverflow as to why this wont work:

” When targeting devices pre-ICS, Android data binding takes over the tag of the outermost element of the layout. This tag is used for mostly for binding lifecycle and is used by DataBindingUtil.findBinding()  and DataBindingUtil.getBinding() . “

Following the advice from the post and because I am targeting ICS and above,  I used the mechanism specified in the answer:

Create a BindingAdapter to set the tag after the binding has occurred:

Then in the XML, I can easily attach any object onto the View:

When the item is clicked we can get more information about the item by using the  view.getTag()  mechanism.

Two Way Binding

This is where things get tricky as the library does not work as expected. Unfortunately, binding an EditText to a String field in an object doesn’t yield the results we would hope for.

The following is a way to get two way binding working:

  1. In our BookDetail object we create an ObservableField<String>  that will contain the books title. This will then inform anything that is watching of variable changes.
  2. In our XML, we will bind to that field on both the EditText and the TextView.
  3. We need to create a TextWatcher which will update the String field when the user types. It will notify the TextView of the changes.

As you can see this is not an ideal solution. Firstly a lot of boiler plate code is required to get it to work and it’s not very elegant.

Better Solution

Fabio Collini has a great detailed post with his solution to this problem – by creating a custom binding for the property. In a few simple steps:

  1. Create a custom BindableString  object
  2. To get the text out of the BindableString  object, we need to have a binding conversion. This instructs views on how to convert a BindableString into a String object:
  3. Create a custom BindingAdapter to handle the text watching and setting:
  4. Set the bindable string in XML.
  5. Change the BookDetail object to use a BindableString instead of the ObservableField<String>

6. Two way binding should now work within your app.

Final Verdict

The Data Binding Library is a powerful tool. I suspect it will become more useful and more widely adopted. For now though, there are a few bugs and some kinks that need to be ironed out.  After experimenting with it for a while, I am excited for what the library could potentially do in the future.

What are your thoughts on this library? Have you used it yet? Leave a comment below!