Five steps to build a mobile experiment

Optimizely makes it easy to A/B test your mobile app. Once you've installed the Optimizely SDK, you can follow the five steps below to create an experiment. You'll create variations, define where the experiment runs and who sees it, set goals, and QA.

Once you're ready, you'll go ahead and publish your experiment live to the world!

Want to learn more about the process of building experiments and the strategy behind testing mobile apps? Check out this 20 minute Optimizely Workshop that walks you through the world of mobile app testing.


1. Create Variations

Once you've created a project and installed the SDK, you're ready to start creating a new experiment.
  1. In the Optimizely Application, select the project that you use for your Android or iOS experiment. Name your experiment and click Create Experiment.

  2. Once you've created an experiment, the Editor begins waiting for a device to enter the Edit mode.

    You can access the Edit mode through the mobile Simulator, which can be launched via Xcode or Android Studio. Or, you can use your physical device with a version of the Optimizely mobile SDK installed.

    To enter the Edit mode in your app, you can use one of the methods below.

    The URL Link:

    The 'O' gesture:

    If you'd like to connect to Optimizely with your device, please make sure that your developer implemented the steps listed in the iOS or Android guide.

    The enableEditor API:

  3. Next, you'll create a variation in the Editor.

Here's how to make changes in your variation:

  1. In your app, navigate to the screen where your experiment will run.
  2. Use Optimizely's Visual Editor to make changes by clicking on elements from inside the browser.
  3. In the right drawer, you will see different visual changes you can make.
  4. The change will be immediately reflected in your app! Click Save to commit your change(s) to the variation. 
Did you follow the steps above?  Congratulations, you've created an experiment using Optimizely's Visual Changes mode, represented by the arrow icon!
Continue exploring by selecting Live Variables or Code Blocks listed below Visual Changes. Don't see your Live Variables or Code Blocks in the Editor?  Make sure you've navigated through your app where those variables are being used.

2. Audiences

Audiences help you define who will your experiment or new experience. You'll be able to define Audience conditions based on device model, custom tags, system version, and more. For example, you might show an experiment to iPhone users who have set their language preferences to English or French.

Click the Audiences icon.

The following conditions are available for audiences:

  • App version: The version of your app, as defined by you.
  • Custom tag: Functions identically to custom tags in Web experiments. For more information on how to add custom tags to your code, see our developer resources (Android, iOS).
  • Device model (iOS): The hardware on which the app is currently running (iPhone, iPod, iPad, iPhone/iPad Simulator).
  • Is an iPad / iPhone / retina device (iOS): Boolean value for if the current device is / is not an iPad, iPhone, Retina.
  • Is a tablet/ phone (Android): Boolean value for if the current device is / is not a tablet, phone. 
  • Language: The language that the device is using within its System Preferences; this is also a subset of NSLocaleIdentifier.
  • SDK version: The version of Optimizely's SDK.
  • System version: The iOS or Android version on the device currently running the app.
  • Attributes: The value a user has for attributes you have set up in your project.

Drag and drop conditions to create an audience.



    Click Done to return to the Editor. Only users who meet this condition will be included in the experiment or new experience.

    3. Traffic Allocation

    By default, Optimizely divides users evenly into different variations. If you have two variations, 50% are bucketed into the original and 50% into the variation. 

    To adjust the traffic allocation, go to Options and select Traffic allocation


    Traffic allocation applies only to new visitors to the experiment. If someone has already seen the experiment, they will continue to see the same variation on repeat visits.

    It also applies only to users who meet the audience conditions. Users who don't meet your audience conditions will see the original experience and will not be included in the experiment.

    4. Goals

    Goals help you measure success in your experiment. Set conversion goals that track the most meaningful conversion events for your business.

    You can select three types of goals for a mobile experiment:

    • Taps: Tracks whether a certain view is tapped
    • Views: Tracks whether a certain view appears
    • Custom events: Tracks visitor events or actions distinct from taps or views, including revenue. For more information on setting up revenue goals, check out our revenue goal guide (Android, iOS). To learn how to set up other custom goals, check out our custom goals guide (Android, iOS).
    • Total Revenue: A default goal available to all experiments, this goal will track revenue per visitor on your variations. Learn how to set up revenue goals for Android and iOS.
    • Session Goals: Session goals are added to each experiment by default. Learn more about session goals here.

    To add a particular transition or screen tap as a goal, for example, open the Goals dialog.

    Click Create a New Goal.

    Use the View Selector feature to navigate through your app. As you do so, your view transitions and taps will automatically appear in the Create New Goal dialog.

    Select your goal, name it, and click Create

    Optimizely will now track the percentage of your users in each variation that complete that action. The results will appear on the Results page.


      Don't forget to check that all custom goals are predefined in your app, prior to releasing your app to the app store.

      5. Preview

      Always preview your experiments to make sure that variations look the way you'd like, before you publish them live to the world. Preview Mode simulates a particular variation just as the user will experience it in production.

      To use Preview:

      1. Click the Preview dropdown and select Launch Preview.

      2. This will cause the app to shut down and restart the app in Preview mode.
      3. When you reopen the app, it will have all the changes you specified in your variation.
      4. To exit Preview, tap the Optimizely icon from your device or simulator, and you should see an option to exit Preview mode.

      Then, when everything looks the way you like, hit Start Experiment to set it live to the world!


      Elements not showing up as expected in the Visual Editor? Try navigating to the view controller you are trying to modify. 

      Best practices

      Here are a few final best practices to consider when setting up mobile experiments.

      • Plan ahead and loop in your developers. Unlike the web, mobile optimization has a longer lead time, so work with your developers in advance to identify what you want to test. That way, you can tag views for the Visual Editor or identify Live Variables/Code Blocks that you want to change.
      • Look at the developer build and approve changes mutually. When you preview and QA experiments, go beyond Optimizely's Preview mode and make sure to actually review your developer build, including test cases, to ensure that the experiment is working as planned.
      • Wash, rinse, repeat. Once you're ready to launch your experiment, you can release to the app store and click Start Experiment in Optimizely. After that, start planning your future experiments in your development sprints!

      Post a Comment