5 Mobile A/B Testing Platforms Reviewed!

Posted on Jan 21 2014 - 10:28am by Editorial Staff

Mobile Apps

Swrve

Swrve positions itself as a complete in-app marketing solution rather than just an A/B testing platform. It was designed with game developers in mind and a clear focus towards mobile game apps. From in-game revenue maximisation to gamer behavioural testing – Swrve does it all.

Swrve’s dashboard is feature-rich and does an awesome job at keeping track of metrics that are relevant to your business. Its reporting dashboard is broken into two major areas:

  • Activity (shows interactive charts of KPIs like new users, DAU, Re-engagement, retention, number of sessions, average session length)
  • Monetization (Conversion, revenue, in-app purchases, ARPU, Daily Paying Users)

Detailed graphs come up on clicking on any chart. You can also create custom charts by combining one or more variables.

Segmentation is a powerful feature and Swrve gives you the power to create segments on multiple aspects of user behaviour. Complex queries like deploying an A/B test to a group of frequent users reaching a certain level who have not still bought any in-app item is possible. This also gives you the ability to optimize user experience and enable in-app messaging for specific segments.

Its in-app messaging feature lets you to send personalized offers and an ability to prompt messages at specific levels of a game. You can also A/B test with this feature.

From the technical point of view, Swrve implementation is entirely on the server side. It is able to handle user segmentation quite smoothly.  It works quite well with code level challenges.

On the flipside, there are a lot of integration issues. It takes around a month to integrate Swrve’s platform with your application. Swrve participation is necessary at all stages during the integration process. That is a major reason why they have not rolled out a self-serve model of their business. Moreover, Swrve’s feature rich platform proves to be a hindrance if you have a mobile app which does only one task. It also not designed to be suitable at handling discrete navigation pages, like shopping apps. It takes a fair bit of effort and has to be applied intelligently.

Use it: If you have a mobile game

Don’t use it: If you have an e-commerce app

Pathmapp

Pathmapp offers a basic in-code A/B testing feature that can operate only on the iOS platform. Its limitations include creating only one test per application at a time. You cannot create a multivariate test as well. On the bright side, it offers a 15-day free trial where you can test its features and decide whether you want to pay for it or not.

There is also no provision of changing variables on the dashboard itself. You have to create your entire test screens in Xcode before they are uploaded to the dashboard. This means that not only would you write separate codes for every single test, you would also need to re-write your code whenever you would want to change the value of your test variable (say, you want to change the colour of your ‘share’ button).

Technically speaking, they use similar open-source architecture as clutch.io.

Use it: If you need a cheap alternative

Don’t use it: If you need basic options like multivariate testing

Appiterate

Appiterate claims to be the world’s first A/B testing platform with a WYSIWYG interface. While that seems to be true, they sure have done a good job of creating a complete drag n drop interface for A/B testing on both iOS and Android.

Appiterate is miles ahead of its competitors because it has solved the basic problem of A/B testing for native mobile apps – hard-coded tests. With in-code tests, you needed to re-write your code every time you change the value of a test variable or whenever you create a new test.

Appiterate started with creating data-driven tests, where you got the ability to change our test variable on the fly, with any need to code. But, they got one more step ahead and created a visual dashboard, like the guys at Visual Website Optimizer have for A/B testing for the web. This eliminates any coding whatsoever after the initial SDK integration. SDK integration is a breeze and does not take more than 10 minutes.

It is quite feature rich like Swrve and includes multivariate testing, real-time deployment and data collection. But, the comparison with Swrve stops there. Appiterate’s reporting and analytics composes only of a few charts, which show conversions and engagement, both cumulative and over a time series.

Appiterate is working towards a complete app management suite to let developers bypass app stores when introducing new features.

Appiterate is live and open for all, but has a 1 month free trial to people who sign up. Plans start at a relatively inexpensive $49.

Use it: If you need an inexpensive live app content management solution

Don’t use it: If you need rich insights and charts

Useartisan

Useartisan is the most expensive of the platforms listed here, with plans starting from $1,500 per month and going up to $15,000 per month, with a 30-day free trial. It is still invitation only and has structured its offering among its three products – analyze, optimize and personalize.

Its reporting includes charts showing device and OS breakdowns, daily active users and sessions,

It has an impressive set of features too. You have the ability to A/B test in real time and have instant publishing as well. Segments like demographics, device type, user behaviours and specific attributes are enabled and its reporting also has confidence scoring. Furthermore, they have recently introduced a WYSIWYG (visual) editor which allows you to make changes to your application through intuitive visual interface.

Their integration process is fast and easy and you will not need more than 15 minutes of your time.

 Use it: If you want analytics, A/B testing and personalization tools in one platform

Don’t use it: If you have a limited budget

Apptimize

Apptimize is a Y-Combinator funded A/B testing platform for iOS and Android. Their dashboard is quite minimal and exhaustive charting options are also not present. Their pricing ($50 and $300 for two adjacent tiers) is also focused at serving the independent developer and a semi-enterprise client, with no option for the person in between.

Only one chart is shown and is limited to conversions for each goal. There is no option to test by segments. Moreover, you need to re-write code in your app after changing variations or when you are creating a new test. But, their dashboard has a provision to auto-generate the code after experiment creation. So, it can be said that their solution lies somewhere in between in-code and data-driven tests.

Use it: If you do not mind spending more than $300 a month even for basic features

Don’t use it: If you need rich insights and personalized experiences

Comparison

Swrve Pathmapp Appiterate Useartisan Apptimize
Pricing Not revealed $19-$399 $49-$1999 $1,500-$15,000 $50-$800
A/B or MVT Not revealed A/B Both Both Both
Platform Android, iOS iOS only Android, iOS Android, iOS Android, iOS
Segmentation Yes No Yes Yes No
Support Not revealed Phone Support 24/7 Support Phone Support 24/7 email and phone support
Self/Full Service Full service only Both Both Both Both
WYSIWYG editor No No Yes Yes No
Ability to edit variations during test Not revealed No Yes No No
Max number of active tests Not revealed 1 Unlimited Unlimited Unlimited
Change test distribution Not revealed No Yes Yes Yes
One-time coding Yes No Yes Yes No

Glossary

A/B or MVT: Whether the company offers only A/B testing or Multivariate Testing or both. A/B testing concerns with testing only 2 variations of a single component, while in multivariate testing you can have more one component being A/B tested simultaneously.

Platform: Which phone OS can the tool work on

Segmentation: Ability to run your tests on a particular demographic or behavioural group.

Support: Client support features

Self/Full Service: Self service means the client registers on the website uses the tool without any proactive involvement of the tool provider. Full service includes activities like on-boarding, consultation, price negotiations etc, like enterprise business.

WYSIWYG editor: Using a WYSIWYG (What You See Is What You Get) editor, you can change visual elements directly on the screen using an intuitive code-free editor and instantly see the outcome on a mobile device.

Ability to edit variations during test/Change test distribution: You might want to include new variations or change the distribution percentage of the tests.

One-time coding: Minimal coding is required only during SDK integration and none thereafter for creating A/B tests.

About the Author
Editorial Staff

Editorial Staff at I2Mag is a team of subject experts led by Karan Chopra.