A/B Test your App using Firebase Remote Config

Last week I was in Nairobi and Cape Town giving a talk on Remote Config and Test Lab in Firebase. I had such a great time and thought I should share some of the content I presented online.

What is A/B Testing?

A/B testing is the process of experimentally testing your UI on different audiences in order to determine the best (or most profitable) user experience.  Firebase Remote Config enables us to do A/B testing by allowing us to randomly segment our app audiences easily.

Example A/B Test

In this example, we are going to test out a different version of a checkout button that we have in our app.

A/B test we will run using Firebase Remote config.

For variant A of the experiment, we want to show a pink “Checkout” button. For variant B we want to show a blue “Cart” button. For the rest of our users, we will show the original “Finished” button.

Now for the fun part, setting up the experiment!

Set up Firebase Remote Config for A/B Testing

  1. Create a Firebase Project.
  2. Head to Remote Config section.
  3. Create a parameter – let’s call this parameter experiment_variant. Make the default value return no_experiment.Setting up parameter in Firebase Remote Config
  4. Click “Add value for Condition”. Then click “Define a new Condition”. Give it the name “Variant A”. We will select “User in Random Percentile” as the conditional. We will then choose the users from 0% – 10% that will receive variant A as their experiment. Click “Create Condition“.Adding a A/B Testing conditional to use with Firebase Remote Config
  5.  Now you will see that conditional appear for the experiment_variant parameter. Assign  variant_a to the value that should be returned for that variant.Conditional in Firebase Remote Config
  6. We will repeat the same for variant B, by creating a conditional for the users from 10% – 20%. This means that only 20% of the entire user base will be running the experiment and 80% will see no change in the standard UI. The percentage of the experiment is totally up to you. Variant B for Firebase Remote ConfigThen return variant_b for the parameter value:All variants in Firebase Remote Config
  7. Make sure to update and publish the changes. We have completed the server side setup of our experiment.

Setting up the A/B Test on Android Devices

Once you have configured the server, the hard part is done! The next section will cover how to get setup for Android but the code is really similar for iOS too – so don’t be deterred!

  1. In your android app, make sure you have the Firebase dependencies set up. Place the following in your top level build.gradle:
  2. Add the following in your app level build.gradle:
  3. Download google-services.json file from Firebase console and add it to your app/ folder.
  4. Create a remote config defaults file in the res/xml folder like the one below. This file will be used if the user has no internet connection when loading up the app for the first time.
  5. Initialise Firebase remote config. This is also where we set the defaults to the file we created in the previous step.
  6. Then fetch the remote config values. Firebase will automatically cache the values for the specified time that you provide. By default, it caches for 12 hours.
  7. In order to run our experiment, we should fetch the experiment_variant parameter from our remote config and use it in our app.

    In this example, we fetch the parameter and set a custom analytics user property with the variant that has been assigned to the current device. This will allow us to properly track how our A/B test is performing.  In the conditional, we are free to make the changes we want to the app. We can set the text to different things and change the colours depending on the variant.
  8. Now that we have the experiment running, we need to track when a user clicks on the “Checkout” button in order to determine which button variant a user is more likely to tap on. We then log the event to Firebase analytics which will include the user property that we set previously.

    It is worth noting that there are a bunch of default events and parameters that you can make use of with Firebase Analytics. 
  9. In order to determine the success of your A/B test, you need to analyse your data to decide which UI is the most successful. After logging into the console, we can see there were 24 counts of the “CheckoutClicked” event.firebase_analytics_ab_testingAfter applying the filter for the user property “Experiment = Variant A”, we can see that 19 of the 24 clicks were from variant A of the experiment (granted this was just me clicking around in my sample app – but you get the idea).

Variant A filters - Remote config

For larger data sets where the result of the experiment may not be obvious at first, you can export your events to BigQuery. This will allow you to make more complex deductions from your experiments.

For now at least, we know the clear winner (in this very large test 😬) is Variant A – the pink checkout button wins!

AB Testing variant A winner

There you have it, an easy way to experiment with your UI using Firebase Remote Config.

Sample code can be found here (It is not very complex as all the hard work is done by the Firebase APIs.) 😊

Happy AB Testing!

Subscribe to Blog via Email

Enter your email address to subscribe to this blog and receive notifications of new posts by email.

Google Developer Launchpad Build SSA – Nairobi and Cape Town Events

I was lucky enough to be invited to speak in Nairobi and Cape Town this past week for the Google Developer Launchpad Build Series events.

The theme this year was Firebase. The event was a huge success and I had the best time! I gave a talk about Firebase Remote Config and Test Lab. Here are the slides from my talk:

The events were filled with great speakers and topics:

Slides from all the talks can be found here: https://bit.ly/lpb-slides 

Instead of posting photos and stuff from the event, I made a travel vlog. Check out my vlog below of all the fun we had!

See you soon Cape Town and Nairobi!

Introduction to Automated Android Testing – Part 6

In the previous 5 blog posts, we covered different aspects of building an Android app from scratch. We focused on including tests in the process. Here are the links to the previous posts:

  • Post #1 – Why should we write tests?
  • Post #2 – Set up your app for testing
  • Post #3 – Creating API calls
  • Post #4 – Creating repositories
  • Post #5 – Following the MVP pattern

In this last post of the series, we will cover creating Espresso tests for the View we created in part 5. The Github repo for this post can be found here.

Testing that a view contains the exact information expected can be tricky if the data is dynamic. This data can change at any time and our tests should not fail because of it. In order for the tests to be reliable and repeatable, we should not call any production APIs.

Mocking out the responses of the API calls will enable us to write tests that depend on the mocked data. There are a couple of ways in which we can mock out our API calls:

  • Option 1 – Use WireMock and run a standalone server which serves up the same static JSON for specific network calls.
  • Option 2 – Use OkHttp’s MockWebServer which runs a webserver on your device and serves up any response you request.
  • Option 3 – Create a custom implementation of the Retrofit REST interface that returns dummy objects.

Obviously, the choice is entirely up to you as to how you would want to go about writing UI tests. In my case, WireMock is extra effort as I would need to ensure I have a standalone server running with a static IP address.

MockWebServer is a lot easier to use than WireMock as you don’t have to set up a standalone web server (The server runs on the device). MockWebServer is also flexible because you can give it different scenarios. Useful features like specifying the failure rate of a certain call or simulating a slow network are possible using MockWebServer. (Read more here).

I am going to use option 3 for the purpose of testing that the UI matches the mock response data. If I wanted to add tests for slow network conditions (or some kind of non-functional test), I would choose option 2. If you are unable to use OkHttp,  I would choose option 1 as Wiremock works with any HTTP client.

Mocking out data using Gradle flavors

By making use of Gradle flavors, we can easily mock out of API responses.  If you read post #2 on Gradle flavors, you should already have a “mock” and a “production” flavor set up.

  1. Make sure you are switched to the mockDebug flavor. Select mockDebug variant
  2. Create a mock folder in the src directory. Then create a package within the mock folder, that mimics the main package name. Make a class called MockGithubUserRestServiceImpl. Your resulting file structure should look like this: Folder structure for mock testing
  3. Create a prod directory. Move the Injection class defined previously into this folder. We will be creating another Injection class in the mock folder. This class will inject the mocked out Github service instead of the production API. Move Injection to prod and create one in mock folderIn the Injection class that is located in the mock folder, we simply return the MockGithubUserServiceImpl that we created. In the prod folder, we return the actual Retrofit Github service. 
    Mock Injection class:

    Prod Injection class:
  4. The data returned from the mock service is dependant on your specific requirements. Below is my implementation of the MockGithubUserRestServiceImpl:

    In this case, I am just returning some dummy data. Let’s run the mock version of the app and we should get the same results no matter what you search.

    Android Test app with mock data

    Cool. Now we have a working dummy app! We can now write Espresso UI tests. 

Basics of Writing an Espresso Test

When writing an Espresso test, the following formula is used for performing functions within your UI:

  • ViewMatchers – Used to find a view in an activity. There are a bunch of different kinds of matchers. For example: withId(R.id.menu_search), withText("Search"), withTag("custom_tag") .
  • ViewActions – Used to interact with a view. For example: click(), doubleClick(), swipeUp(), typeText().
  • ViewAssertations – Used to make assertions that certain views possess specific properties. For example:   doesNotExist(), isAbove(), isBelow().

There is a great cheat sheet for the different Espresso methods that can be found in pdf form here: android-espresso-testing.pdf. It is worth mentioning that ordinary hamcrest matchers can be used when writing Espresso tests. Methods such as not(), allOf()  and anyOf() are valid.

Writing Espresso UI Tests

If you can recall, in post #2 we covered what dependencies will need to be added in order to write espresso tests. Now we will cover writing an Espresso test.

  1. Create a folder androidTestMock. The tests in this folder will only run on the mock variant and not on the production variant. Then make a directory that matches the main package name. In that directory, add a new class called  UserSearchActivityTest. Your project should then look like this:androidTestMock folder
  2. We will start by writing a basic test that ensures that when the activity is started, the text “Start typing to search” is displayed:

    The @Rule  ActivityTestRule specifies which activity this test will run with. In this case this test will run with UserSearchActivity. This will automatically start up the UserSearchActivity. Passing extra parameters will indicate if you want the activity to auto start or not.

    The test searchActivity_onLaunch_HintTextDisplayed() is quite simple. It searches in the view for the text and asserts that the text is visible on the UI.

  3. The next test is slightly more complicated:

    After typing into the SearchView and pressing enter, we assert that the dummy results are displayed on the UI.

  4. We have now written tests for the positive scenarios, we should add a test for the negative case too. We will need to adjust the MockGithubUserRestServiceImpl in order to allow it to return custom error observables if required.

    In the code above, a method was created in order to set a dummy observable for the search results. That observable will be returned if it is not null when searchGithubUsers() is called.

  5. Now we can create a test that checks if the error is displayed on the UI.

    In this test, we first ensure that the service will return an exception. Then we assert that the error message is displayed on the UI.

  6. Let’s run the tests:


They all pass!

Code Coverage in Android

In order to know how effective your tests are, it is great to get code coverage metrics.

  1. To enable code coverage on your UI tests, add testCoverageEnabled = true  to your build.gradle:
  2. Run the task createMockDebugCoverageReport. You will then find the HTML report located here: app/build/reports/coverage/mock/debug/index.html.

Code Coverage Report in Android

Yay – we have 82% coverage just with the mock UI test. Taking into consideration the coverage reports we saw in post #4 and this post, it gives us a good indication of the test coverage of our entire app. Now we can iteratively go back and try cover more areas of our code.

PS – Code coverage currently doesn’t work with the Jack compiler. I switched to use Retrolambda in order to get the code coverage report to work. If you are interested in learning more, check out this branch.


We have finished writing our feature. Whew! 6 blog posts later. There is obviously a lot more testing that can be completed in this app. Non-functional tests such as testing how your app behaves on devices with low memory or with poor network connectivity can also be added.
That concludes the series on an “Introduction to Automated Android Testing”. I hope you enjoyed reading this series. If you enjoyed it, be sure to subscribe to the blog to receive future updates and share the post with your friends.

Further Reading

Subscribe to Blog via Email

Enter your email address to subscribe to this blog and receive notifications of new posts by email.

If you would like to see more posts from me, please consider buying me a cupcake to keep them coming.

No. of Cupcakes

Introduction to Automated Android Testing – Part 5

In this series of blog posts, we are working through a sample app called Github User Search. Parts 1 – 4 covered why we should test, getting set up with testing, creating API calls and creating a presenter. Take a look at the previous posts as part 5 is a continuation of the series.

In part 5, we will take a look at interacting with the Presenter  created in part 4 and we will create the UI to display the list of search results.

Creating the UI

For the User Interface we want a simple list that displays the avatar, name and other user information in the list.

In part 4, we defined a View contract which the Activity should implement. This is where the Android specific code will be located (things such as visibility changes or any UI changes will be located here). To refresh your memory, this is the View contract we created in this last post:

Let’s implement the View!

  1. Create a class called  UserSearchActivity. This class will implement the UserSearchContract.View contract and extend AppCompatActivity. Define a variable called userSearchPresenter of type UserSearchContract.Presenter. This is the object that we will interact with in order to perform our network calls.

    In onCreate(), create the presenter object. Provide it with the User repo defined in the Injection class. Pass the io() scheduler and the AndroidSchedulers.mainThread() scheduler so that the RxJava subscriptions know which threads they should perform their work on.

    On the next line, you can see I call userSearchPresenter.attachView(this) . This attaches the view to the presenter, so that the presenter can notify the view of any changes. Because the presenter isn’t aware of the activity’s lifecycle, in onDestroy() we need to inform the presenter that the view is no longer in existence, so we should then call userSearchPresenter.detachView(). This will unregister any RxJava subscriptions and prevent memory leaks from occurring.

  2. Create activity_user_search.xml in the layout folder. This will contain a RecyclerView, a ProgressBar, an error TextView and a Toolbar. I am using ConstraintLayout to design my screen, so I won’t go into too much detail as it is mostly drag and drop. (If you want to read more about ConstraintLayout check out my blog post about it here)UserSearchActivity

  3. We also need to add a SearchView to the toolbar so we have somewhere to type. Add a menu_user_search.xml file to the menu resource folder. Inside it, you will need to add a SearchView:

  4. We need to create a layout that will be used for each item in the RecyclerView. Create a file named list_item_user.xml in the layout folder. I used ConstraintLayout with one ImageView for the avatar and two TextViews.List_item_user_designmode

  5. Now that we have all the layouts we need, let’s tie the XML to the Activity. First, in onCreate() we will get references to the views we need.

  6. We need to hook the SearchView up into our activity to make it trigger the presenters search() method. In the onCreateOptionsMenu(), add the following code:

    This will inflate the correct menu, find the search view and set a query text listener. In this case, only when someone presses submit on the keyboard, we will respond by calling the search presenter with the query. We could do it in onQueryTextChange too but due to rate limiting on the Github API I will stick to onQueryTextSubmit. By default, the item will be expanded.

  7. Next, we will implement the callbacks that the presenter will call when the items are finished loading.

    We are basically just toggling visibility of views here and setting the usersAdapter  to the new items that the service returned.

  8. For completeness, here is the UserSearchAdapter class which is used for the RecyclerView on the activity:


    Injection class

  9. Now if you run the app, you should be able to search for a username on Github and see results.github_user_search

Yay! We have a working app. Code for this post can be found here. In the next part, we will look at writing UI tests for the app. Make sure you subscribe so you don’t miss a post!

Subscribe to Blog via Email

Enter your email address to subscribe to this blog and receive notifications of new posts by email.

If you enjoyed this blog post, please consider buying me a cupcake to support future blog posts.

No. of Cupcakes

Introduction to Automated Android Testing – Part 4

In this blog post series, we are working through an example Android app called Github User Search. In the previous blog posts, we took a look at setting up your app for testing, creating API calls and writing the first basic test for the API transformations. Read part 1, part 2 and part 3 first.

This post will look at creating a presenter which communicates with the repository and conveys information to the view. This will also include writing unit tests for the presenter. The sample github repo this blog post will be working from can be found here.

Creating the Presenter

  1. In order to get started, create base interfaces called MvpView and MvpPresenter. All MVP functionality will extend these two interfaces.

  2. Create a BasePresenter. This will provide functionality to check whether a view is attached to the presenter and a way to manage RxJava subscriptions.

    As you can see above, there is a CompositeSubscription defined in the presenter. This object will hold a group of RxJava subscriptions. The detachView() method calls  compositeSubscription.clear() which will unsubscribe from all subscriptions, prevent memory leaks and view crashes (code will not run when the view is destroyed as it is unsubscribed). When a subscription is created in a presenter that subclasses this object, we will call addSubscription()
  3. Create the contracts between the view and the presenter in a class called UserSearchContract. Within this class, create two interfaces one for the view and one for the presenter.

    In the view, there are 4 methods, showSearchResults()showLoading(), hideLoading(), showError(). In the presenter, there is a method called search().

    A presenter does not care about how a view shows the results, nor how it shows an error. Similarly, a view doesn’t care how a presenter searches as long as it uses those callbacks to notify, the implementation doesn’t matter.

    Separating the logic between the view and presenter is simple. Think about reusing the presenter for another type of UI and that will make you realise where the code should live. For instance, if you had to use Java Swing, your presenter can remain the same in that case, only your view implementation would differ. This helps you to place logic by simply asking yourself the question: Would the logic in the presenter make sense if I had a different type of UI?

  4. Now that the contracts between the view and the presenter are defined. Create/navigate to UserSearchPresenter. This is where a subscription to the UserRepository will be created, which will call the Github API.

    Here the presenter extends BasePresenter  and implements the UserSearchContract.Presenter contract defined in step 3. This class will implement the search() method.

    Using constructor injection allows easy mocking of the UserRepository when trying to do unit testing. The schedulers are also injected into the constructor, as the unit tests will always use Schedulers.immediate() but in the view we will use different threads.

  5.  Now for the implementation of search() :

    First off, run checkViewAttached() this will throw an exception if the view is not attached when the method starts running. Then tell the view that it should start loading by calling  showLoading(). Create a subscription to  userRepository.searchUsers(). Set subscribeOn() to the ioScheduler variable as we want these network calls to happen on the IO Thread. Set observeOn()  the mainScheduler  as we want the result of this subscription to be observed on the main thread. Then add the subscription to our composite subscription by calling addSubscription.

In the onNext() method, handle the result by calling hideLoading() and showSearchResults() with the list of users returned by the API. In onError(), stop the loading and call showError() with the exception’s message.

Here is the full code for UserSearchPresenter :

Writing Unit Tests for the UserSearchPresenter

Now that the presenter is defined, let’s create some unit tests for it.

  1. Select the UserSearchPresenter class name. Press “ALT + Enter” and select “Create Test”. Select the “app/src/test/java” folder as this is a unit test that requires no Android dependencies.  The resulting location of the test is the following: app/src/test/java/za/co/riggaroo/gus/presentation
  2. In the UserSearchPresenterTest, create the setup method and define the variables needed for testing.

    By creating a mock instance of  UserRepository and the UserSearchContract.View we will ensure that we are only testing the UserSearchPresenter. In the setUp() method, we call MockitoAnnotations.initMocks() . Then create the search presenter with the mocked objects and immediate schedulers. Call attachView() with the mock view object as the presenter works only once a view is attached.
  3. The first test will test that a valid search term has the correct callbacks:

    This test asserts that: Given the user repository returns a set of users, when calling  search() on the presenter, then the view methods showLoading() and showSearchResults() are called. This test also asserts that the showError() method is never called.
  4. The next test is one that tests the negative scenario if the UserRepository throws an error:

    This test is testing the following: Given the userRepository  returns an exception, when calling search()  then showError() should be called.
  5. The last test we will add will assert that if the view is not attached, an exception will be thrown.
  6. Let’s run the tests and see how much test coverage we have. Right click on the test name and click “Run tests with coverage”Unit Test Coverage from the Presenter Test

We have 100% Coverage of the UserSearchPresenter! Yay!

The next blog post will cover creating the view and writing tests for the view. Make sure you subscribe so you don’t miss the next post in this series!

Subscribe to Blog via Email

Enter your email address to subscribe to this blog and receive notifications of new posts by email.

If you like my work, please consider buying me a virtual cupcake to keep the blog posts coming!

No. of Cupcakes


This post series was based on a couple of implementations of MVP that can be found around the web. For reference, these are some of the articles used to get to this solution: