UT Foundation Policies
10.1.8 A/B Testing Campaigns
EFFECTIVE: 8/11/2020 · REVISED: 5/18/2022
Audience: Email Author, Communications Admin
General Purpose
Instead of guessing what resonates with your constituents, A/B testing helps you make decisions based on data. It will tell you if your email marketing campaign is working or more importantly if it is not. Variables include subject line, pre-header or from name and sending address. If you have questions, contact [email protected].
A/B Testing Campaign Enhancements (1/20/21)
- Copy an A/B Testing Campaign from the More Options Menu and the Gear Icon
- View Template from the Content Page in the workflow
- Navigation Buttons added to Email Editor Page
- Convert HTML general campaign to an A/B testing Campaign
Quick Takeaways: How A/B Testing Differs from Other Campaigns
- A/B Campaigns have a breadcrumb navigation where you can click ahead to steps in process.
- Previews: Recipients receive a preview of each version; the same comment appears at the top of both.
- A/B Split: Email admins can set split criteria without selecting an audience. In other words, email admins who have the OE Team upload their lists can set these preferences in advance.
- Once deployed, the A/B Campaign will remain in the queued area until the winner is released.
- The notification email address(es) receive a copy of the A, B, and winning version.
- Reporting: Overall performance section will show rows for A/B/Winner
Jump ahead to any section—
| NavigationDetails & Testing VariablesBuild from Template | AudienceA/B SplitSummary Page | Email ReportingParametersBest Practices |
Accessing
Log into Encompass at your site below. If you’re unsure of the email address associated with your login, email [email protected].
| alumni.tennessee.edu/loginalumni.utc.edu/loginalumni.utk.edu/loginalumni.utsouthern.edu/login | advanceutia.tennessee.edu/logintalumni.uthsc.edu/loginalumni.utm.edu/login |
Hover over Email on the top menu and select Email Home.
Navigation
With A/B Testing Campaigns, a new breadcrumb navigation has been introduced. If admins click a radio button further along in the build process and the work on the current step hasn’t saved, a warning message will appear. Admins can also use the NEXT button to move to the next step in the process.
Note: This navigation won’t show in the content editing step of the build process.
Related video: Navigation
Email Details & Testing Variables
The Campaign Details for A/B Testing Campaigns is very similar to General Campaigns. You will need to provide the required information and click NEXT: Continue in order to move past the Details.
You will need to select one of the following testing variables:
Subject Line Test
You will need to set two subjects limited to 70 characters each. The subject line cannot contain the webview or unsubscribe token.
You can change the subjects until the message is within 5 minutes of the scheduled date/time of send.
Related video: Subject Line Test
Pre-header Test
You will need to set two pre-headers. They cannot contain the webview or unsubscribe token.
You can change the pre-headers until the message is within 5 minutes of the scheduled date/time of send.
Related video: Pre-header Test
Sender/From Test
You will need to set values for both Sender A and Sender B. Tokens cannot be used to overwrite From Name box.
You can change the Sender/From until the message is within 5 minutes of the scheduled date/time of send.
Related video: Sender/From Test
Build From Template
- From Email Home, choose Template.
- If a template hasn’t been chosen yet, click Content in the navigation or Next: Content.
- Choose the template from the dropdown list and use the navigation or button to move on and work with the message body.
Note: Once a template is chosen and you proceed to the Content step, you cannot choose a different template.
Related video: Build from Template
Audience
Selecting the Audience for A/B Testing Campaigns is the same as General Campaigns and Segmented Campaigns.
There are a few different ways to get to the Audience area.
- Click Audience in the navigation from any step of the A/B Testing Campaign process.
- Use the Next: Audience button on the Preview step.
- From the Summary step, use the Edit button.
Note: If the Audience has been selected, information will be provided in the Inclusive Audience and Exclusive Audience areas.
Related video: Audience
A/B Split
When an A/B Testing Campaign is created, the admin will need to set configuration details.
Accessing
There are a few different ways to get to the A/B Split area.
- Click A/B Split in the navigation from any step of the A/B Testing Campaign process.
- From the Audience selection screen since working with the A/B Split is the next step in the process. Audience selection is the same for A/B Testing Campaigns as it is for General Campaigns and Segmented Campaigns. You can either set the Audience and then click NEXT: A/B Split or move to the A/B Split and set the Audience later.
- Select Schedule from Email Home. You will be taken directly to the A/B Split screen.
Determine test group size
Use the drag and drop slider to set the test group. The two testing groups must be the same percentage. If the Audience and Subscription Category have been set, the overall estimated Audience
number will be shown. The tests and winner will also indicate the number of people assigned to each.
Related video: Set Test Group Size
Identity how the winner will be decided
At this time, the Open Rate will be the deciding factor. The version with the highest open rate at the end of the testing period will be sent to the remaining recipients.
Decide when test will start
You will need to decide when the test will start. You can have the test start immediately or send at a later date/time. Time Zone will default to the time zone set for your computer.
Related video: Schedule
Choose how long the test will run
Set how many hours the test will run for. The default value is 4. You may find the Testing Parameters article helpful in determining how long you would like your test to run for.
Related video: Set Test Duration
Click NEXT: Summary or Summary in the navigation to see a recap of the A/B Testing Campaign. You can release the Campaign from the Summary.
Summary Page
The Summary page provides a summary of the A/B Testing Campaign settings.
Edit buttons are available for each area so admins can easily navigate to an area they need to work with.
If the Audience and Subscription Category have been set, admins will also be able to export the estimated recipients.
Admins can also release the A/B Testing Campaign from this screen. The campaign can also be released from the gear icon on Email Home.
Related video: Summary
Email Reporting
Email Reporting for A/B Testing Campaigns has a different look when you’re drilled into a specific campaign; however, all of the same great information you’re used to is still present!
Accessing
- From Email Reporting home, click on the name of a test message or the winner.
- From the History area of Email Home, click on the name of a test message or the winner.
- From the Released for Queued screen, click on the View Live Results button (only applicable for Send Immediately).
Test Complete
If the test is complete and a winner has been sent, the screen will indicate which test was the winner (A or B).
Test In Progress
If the test is still taking place, the page will indicate the test is in progress. There is an option to Cancel Automated Send and Manually Select Winner.
Clicking the Cancel Automated Send button will cause a warning box to display so you can confirm that you want to cancel the automated send. If you confirm, then no winner will be sent.
Clicking the Manually Select Winner button will cause a box to display so you can pick Test A or Test B as the winner and immediately send that one to the remaining audience.
Layout
When you are looking at a specific A/B Testing Campaign, the landing page will start with Campaign Activity on the left and Campaign Details on the right (as shown above).
The lower portion of the screen will display Overall Performance. If the test is still taking place, only the tests will display (as shown below). If the winner has been sent, there will be a row for the winner.
The Export Results can be used to export the information.
Links at the top allow you to move to specific areas to get information.
Drilldown Reports
When looking at the drilldown reports, the top area will show metric information. The View Details link can be used to expand the area to see how each test did as well as the winner (if sent).
A search box is available as well as filtering options.
Based upon the filters being used, the results will be shown. In the image below, the Audience area is being looked at.
The Export Results button can be used to export the data.
Related video: Email Reporting
Parameters
Cochran’s formula was used to calculate the recommended test size group.
| Audience Size300 or less400500600700800-9001,0001,100 – 1,2001,300 – 1,4001,500 – 1,8001,900 – 2,3002,400 – 3,2003,300 -4,9005,000 or more | Test Group100%80%70%65%55%50%45%40%35%30%25%20%15%10% | A50%40%35%32.5%27.5%25%22.5%20%17.5%15%12.5%10%7.5%5% | B50%40%35%32.5%27.5%25%22.5%20%17.5%15%12.5%10%7.5%5% | Winner0%20%35%35%45%50%55%60%65%70%75%80%85%90% |
Based on research, the winner will be correctly chosen 80% of the time if you run your test 2-5 hours. Your own institutional results may vary.
Best Practices
Why you should A/B test your marketing campaigns
Instead of guessing what resonates with your constituents, A/B testing helps you make decisions based on data. It will tell you if your marketing campaign is working or more importantly if it is not.
Guidelines
Set Goals – Know what and why you want to test
Do you know your baseline? Know where you stand now to show positive/negative trends. Look at how you can improve your open rates. This will help you determine your hypothesis. A hypothesis is an idea about what you need to test, why it needs to be tested, and what changes you’ll see after you make any changes.
Focus on frequently sent emails
To start, only focus on testing emails you send frequently. Then start branching out to other email campaigns.
Audience
Ensure your audience is large enough so when the system randomly selects the test group your results will be statistically significant.
Test one element at a time
To understand what is really impacting your results, it’s best to select one variable at a time.
Wait the optimum amount of time
Be patient! it is important to wait long enough to gather enough data to ensure the right email is being selected as the winner. See the recommended parameters.
Check results
You want to make sure the results mean something before you move forward and implement them into your marketing strategy. Confirm the results of tests and build on your leanings.
Test and test again
What resonates with your audience today may change as time goes on. There fore, it is important to continue to test and identify how your constituents respond.
Avoid these mistakes
- Not having a hypothesis
- Comparing results from outdated campaigns
- Changing metrics mid-way
- Not segmenting your audience before testing
Variables – ideas and tips to test
Subject Line
- Personalization – use tokens to personalize one of the subject lines
- Question vs. Statement
- Short subject line vs. a longer subject line – Ideally subject lines should be 40 characters or less
- Emoji vs no emoji
- Add humor vs. being straight forward
- Change around the words slightly between subject lines
- Sense of urgency vs. no urgency
Pre-Header
- Personalization – use tokens to personalize one of the subject lines
- Question vs. Statement
- Emoji vs no emoji
- Short Pre-Header vs. a longer Pre-Header
- Generally, keep Pre-Header text between 40-130 characters
- Most mobile Pre-Header text is between 30-55 characters long
- Add call to action vs. no call to action
- Do not repeat the subject line
- Make your subject line and Pre-Header text work together
- Build curiosity
From Name and Sender Address
Ask yourself – Will the targeted audience more likely recognize and respond to an individual’s name, institution or group?
- Department/Group Name vs. Individual’s Name (maybe a high profile figure name)
- Full Name vs. First Name only
- Different email addresses