Hello! In this article we're going to go over our newest feature A/B Test goals! A/B Test goals must be enabled by an administrator at the account level in order for it to be used in an organization's groups.
If you do not see the option to enable A/B Test goals, please contact your CSM so they can make this feature available to you 🤗
Skip to:
Setting Up the Goal
A/B Testing allows admins and managers to launch a goal with variant initial texts to see which scripts perform the best, and engage the most amount of people! Each variant script will be sent to at least 10 leads targeted in the goal (you can have up to five variants!) and depending on the response and engagement rate of each one, you will be prompted to select a winner!
Please Note: An A/B test goal cannot be launched without a minimum of 405 leads per variant.
Once a winner is selected, your agents will receive the remaining batch of initial texts to send out in their workflows!
Let's go over how to set up your A/B Test goal!
From your goals page in your group, instead of clicking the usual ‘Add Goal’ button on the top right hand corner of the screen, you’ll click the drop-down carrot next to it and select ‘Add A/B Test Goal’:
After that, it’s much like launching a typical P2P goal!
You’ll select the Goal Type, Name the goal, choose an Event Start Time, and set the targeting for the goal.
Then you’ll fill out the Start Date & Time and the End Date & Time:
After that is where we get to the fun A/B testing part of the goal!
Variants are sets of test scripts that are sent out in batches of ~10 messages per variant when the goal is first launched to determine which script your audience engages with the most! You can use these different scripts to test out what kinds of different media perform best! Try emojis in one, videos in another and gifs in a third! Variant workflows will be available to agents based on the Agent Workflow Start Date & Time.
You can create up to five by clicking “Add Variant” at the bottom of this section!
Variants will be named alphabetically by default, but variant names can be edited by admins and managers from the Edit modal.
Don't for get to click Save after making edits!
Once you get to the 5th variant the ‘Add Variant’ button will become grayed out and no other scripts will be able to be created! To test more variants, you can launch a new A/B Testing goal in the group.
Once you’re happy with your variant scripts you can save as a draft or click launch!
Sending Test Workflows
Once launched, your A/B test goal will be live! When you click into the goal, the stats will look a bit different from a typical goal. You’ll see one set of stats per Variant!
You can see how many variant texts have been given to agents in workflows to determine which test script performs the best!
There will also be a yellow banner stating that the A/B Testing is in progress. (see above screenshot). This banner will turn green five minutes after enough responses have come in to prompt an admin or managerto select a winner (fact check the timing on this)
Once the goal is live, agents can send the first set of messages from the mobile app or web.hustle.com just like a normal P2P goal.
A workflow for an A/B Test goal looks like this:
As mentioned earlier, each workflow will have a set of at least10 messages to see which script performs the best with your audience! Agents will see a "Start" button for each variant's workflow.
By clicking start, they'll begin the workflow and this process will be just like any other P2P goal! Reply workflows will show up the same way as any other reply workflows.
Understanding the Results
After your agents have sent each variant's messages for this first set of texts, an admin or manager can select a winner based on which variant got the best results from the goal stats page in the admin panel!
Click confirm!
Confirming a Winning Variant
And voilà, when confirmed as the Test Winner, Variant C will be the script sent out for the remainder of leads targeted in this goal Your agents will receive workflows for the remaining initial messages using Variant C.
Please Note: At this time, A/B goals are not yet editable from the goal page.
For more guidance or to answer any questions, don't hesitate to email support@hustle.com with the link to your A/B Test Goal draft or goal in the admin panel for help!