Email Send: Split (A/B) Testing

What is Split Testing?

With Split Testing you can test two versions of your email with varying subject lines, content (i.e. template) and/or ‘from’ address.  The two versions are referred to as the Version A and the Version B.  You will determine what percentage of your recipients should receive the A and B versions until a winning version is chosen automatically by our system or manually by you.  The winning email version, A or B, is then sent to the remainder of the recipients.  For example, let’s say you have a marketing list of 45,000 recipients and you want to test how an email performs with two versions of the subject line.  You will create an A and B version with the two subject lines and then you might decide to send these versions to 1,000 of the 45,000 recipients and let the system determine the winner after a period of 5 hours.  The number of recipients in version A and B will be the same. In this example, each version will be sent to 500 recipients. At the end of the 5 hours the system will choose a winner based on either the most unique opens or unique clicks (your choice).  The winning version, A or B, will then be sent to the remaining 44,000 recipients.  The benefit is that the vast majority of your recipients will receive the better performing email version and you will learn what caused the increased engagement.

Previous Steps: Create an Email Send and Adding Recipients.


To set up a Split Test, you need to have already created your Email Send and added at least 3 recipients.

On the Email Send record, choose Split Test.

The Split Testing Wizard will open. On the left are the details for the Email Send you’ve created, which is automatically assigned as Strategy A. The information in Strategy A cannot be edited. On the right side is the information for Strategy B. At least one of the fields should be edited (Subject, From Name, or From Address).

You can also choose a different template as well by using the lookup field to select an existing template. If you would like to edit the email content for Strategy B, click on the Edit button. This will open the editor an allow you to make changes just for version B. Just be sure to save!

Press Next when you’re ready to move to the next page of the Split Test Wizard.

This page allows you to set the size of the test groups and winner group. The test and winning groups are Split 50/50 by default: 25% to each test group and the other 50% to the winning group. To edit the size of the test groups and winning group, simply slide the slider to the appropriate number. The slider is the black bar in the middle of the distribution graph. You can choose the winning strategy based upon the most Unique Clicks or Unique Opens.

Next is the option to manually or automatically send the winning version. The benefit of automatically sending the winning version is that the Split Test can be set to a certain amount of time (Hours, Days, Weeks) and after that time the system will automatically evaluate and send out the winning version. For greater control, you can choose to manually send out the winning version based on your evaluation of the Split Test.

Choose Save and Send to begin the Split Test or chose Save and go back to the Email Send record.

The information entered in the Split Test Wizard can also be found under Split Test Preferences in the Email Send record.

The Split Test Preferences on the Email Send record will show the details of how the Split Test was set up in the Wizard. The fields are not to be manually filled as it will not trigger a Split Test.

To check on the progress of the Split Test, go to Marketing > Email Sends , change the view to Split Test, open the Split Test record and scroll down to Email Statistics. The Split Test section of the Email Statistics shows Unique Clicks and Unique Opens for both Strategy A and Strategy B. Read more about Split Test Statistics.

To manually select and send the winning version, go to Marketing > Email Sends , change the view to Split Test, open the Split Test record and click Send. A window will pop up that provides the winning strategy based on the set criteria and you’ll be able to choose which strategy you want to send regardless of which one is winning. Click Approve. The winning group will now be sent whichever strategy was chosen as the winner.


To set up a Split Test, you need to have already created your Email Send and added at least 3 recipients.

On the Email Send record, choose Split Test.

SplitTest00

The Split Testing Wizard will open. On the left are the details for the Email Send you’ve created, which is automatically assigned as Strategy A. The information in Strategy A cannot be edited. On the right side is the information for Strategy B. At least one of the fields should be edited (Subject, From Name, or From Address).

SplitTest01

You can also choose a different template as well by using the lookup field to select an existing template. Click Next, which will either open the new template or a copy of the Strategy A template.  You can make any changes which will be reflected on Strategy B, then click Next. 

SplitTest02

This page allows you to set the size of the test groups and winner group. The test and winning groups are Split 50/50 by default: 25% to each test group and the other 50% to the winning group. To edit the size of the test groups and winning group, simply slide the slider to the appropriate number. The slider is the black bar in the middle of the distribution graph. You can choose the winning strategy based upon the most Unique Clicks or Unique Opens.

SplitTest03

Next is the option to manually or automatically send the winning version. The benefit of automatically sending the winning version is that the Split Test can be set to a certain amount of time (Hours, Days, Weeks) and after that time the system will automatically evaluate and send out the winning version. For greater control, you can choose to manually send out the winning version based on your evaluation of the Split Test.

Choose Save and Send to begin the Split Test or chose Save and go back to the Email Send record.

SplitTest04

The information entered in the Split Test Wizard can also be found under Split Test Preferences in the Email Send record.

SplitTest06

The Split Test Preferences on the Email Send record will show the details of how the Split Test was set up in the Wizard. The fields are not to be manually filled as it will not trigger a Split Test.

To check on the progress of the Split Test, go to Marketing > Email Marketing > Email Sends, change the view to Split Test, open the Split Test record and scroll down to Email Statistics. The Split Test section of the Email Statistics shows Unique Clicks and Unique Opens for both Strategy A and Strategy B. Read more about Split Test Statistics.

To manually select and send the winning version, go to Marketing > Email Sends , change the view to Split Test, open the Split Test record and click Send. A window will pop up that provides the winning strategy based on the set criteria and you’ll be able to choose which strategy you want to send regardless of which one is winning. Click Approve. The winning group will now be sent whichever strategy was chosen as the winner.

SplitTest08

Next Step: Test your Email.


Feature Added: Original
Feature Updated: Original
ClickDimensions Version Need: Any