The purpose of A/B [/C] testing is to evaluate the effectiveness of similar services to identify the best one.

Goal

Before performing an A/B test, one needs to determine the evaluation criterion:

• the amount of checkouts;

• the amount of paid orders
• the final profit for the store (paid orders and sales margin).

Audience

The audience should be divided evenly between all visitors during the test. Ideally, if the audience is divided evenly not just randomly, but:

• equally between traffic sources;
• uniformly by the time of day;
• evenly by the type of access device (smartphones, tablets, high-grade computers and laptops)

The audience accounted should not include:

• Retailer's employees;
• Robots and search engines.

The user's group should not depend on any mathematical operation (e.g. the remainder of the division of the hexadecimal representation of his/her e-mail), which does not contain any element of randomness.

The user's group cannot be changed over time: if the user has been assigned group A and two weeks later he returned to the site, he should still be in group A. 

Clearing of the results 

From the final results it is necessary to remove:

• orders are done by retailer's managers, test orders, issued by company employees;

• orders which cost is too low (e.g., orders with a price of $0.10 being AOV equal to $100, since these are most likely erroneous orders or tests);
• orders which cost too high (e.g., orders for the amount of $2000 being AOV equal $100);
• orders from regular customers, making more than two orders per month regularly because these orders are not due to any personalization system.