A/B test a campaign
Send two to four messages to a test group, pick the winner by delivery, read or reply rate, and send it to the reserve audience.
An A/B test sends two to four versions of a campaign message to a test group. When the test window ends, Polymorfa compares the versions on the outcome you choose and sends the winning message to the reserve: everyone not in the test group or the optional holdout. If no version is clearly ahead, the campaign pauses and the reserve gets nothing.
A/B campaigns follow every other campaign rule in Send a campaign, including tiers, sending Numbers, opt-outs and send windows.
How the audience is split#
At launch, each recipient is assigned once to one of three groups. The assignment is saved and does not change when a send is retried or the campaign is paused and resumed.
| Group | Size | What it receives |
|---|---|---|
| Holdout | holdoutPercent of the audience, 0 to 30 | Nothing. Holdout recipients are marked skipped with lastError experiment_holdout at launch and are not retried |
| Test group | testSlicePercent of the audience, 5 to 50, default 20 | One variant each, split by the variants' weight |
| Reserve | Everyone else | The winning variant, after the test is decided |
The split is by chance, so group sizes are close to the percentages rather than exact. A recipient skipped for another reason, such as an opt-out or a missing variable, keeps its group but is not sent.
Create an A/B campaign#
Send variants and variantStrategy when you create the campaign. Each variant is a text message; {{variable}} placeholders are filled per recipient as in composed messages, and every variable needs a value or fallback, as described in Send a campaign. An A/B campaign sends its variants, so it cannot also set templateId.
Console
- Open Campaigns, select New campaign and choose A / B test.
- On Variants, write each variant's message and set its weight. Select Add another variant for up to four.
- Set the Winner criterion (Delivery rate, Read rate or Reply rate), the test window, the test group percent and the holdout percent.
- Continue through the builder and launch as in Send a campaign.
API
curl -X POST "https://api.polymorfa.com/messaging/projects/$PROJECT_SLUG/campaigns" \
-H "Authorization: Bearer $POLYMORFA_KEY" \
-H "Content-Type: application/json" \
-d '{
"name": "Spring offer test",
"recipientListId": "'"$AUDIENCE_ID"'",
"variants": [
{ "key": "short", "label": "Short", "weight": 50,
"blueprint": { "version": 2, "source": "Hi {{first_name | there}}, 20% off this week." } },
{ "key": "story", "label": "Story", "weight": 50,
"blueprint": { "version": 2, "source": "Hi {{first_name | there}}, spring is here and so is 20% off everything until Sunday." } }
],
"variantStrategy": {
"winnerCriterion": "reply",
"holdoutPercent": 10,
"testSlicePercent": 20,
"autoPromote": true,
"testWindowMinutes": 240
}
}'| Field | Rules |
|---|---|
variants | 2 to 4 entries |
variants[].key | Unique in the campaign; a lowercase letter, then up to 15 lowercase letters, digits, _ or - |
variants[].label | 1 to 100 characters |
variants[].weight | Integer from 1 to 99; the weights add up to 100 |
variants[].blueprint | version 2 and source, 1 to 4,096 characters |
variantStrategy.winnerCriterion | delivery, read or reply |
variantStrategy.holdoutPercent | Integer from 0 to 30 |
variantStrategy.testSlicePercent | Integer from 5 to 50; defaults to 20 |
variantStrategy.autoPromote | Must be true. The winner is always sent automatically |
variantStrategy.testWindowMinutes | Integer from 15 to 1,440 |
A request that breaks a rule returns 400 invalid_parameter with error.param naming the field. While the campaign is an unlaunched draft, send new variants or variantStrategy with an update to change them, or null for both to make it a broadcast. After launch an update returns 409 state_conflict.
On the Platform API, send the same fields to API Create campaign and API Update campaign.
How the winner is chosen#
The test window starts when the campaign starts sending: at launch, or at scheduledAt for a scheduled campaign. Test-group recipients are sent during the window. Reserve recipients stay queued until a winner is chosen.
When the window ends, Polymorfa decides once:
- Every test send must be settled. A test recipient that is still queued or sending, whose send outcome is unknown, or whose result arrived after the deadline makes the test inconclusive.
- For each variant, the rate is the number of test recipients delivered, read or replied to by the deadline, divided by the test recipients sent by the deadline. Results that arrive later do not change the decision.
- Every variant needs at least 20 sent test messages.
- The winner's rate must be clearly higher than every other variant's: the lower bound of its 95% confidence interval (Wilson score) must be above the upper bound of each other variant's interval. A higher rate alone, a tie, or overlapping intervals is inconclusive.
When there is a winner, the reserve is sent the winning message and the campaign continues as usual. experimentOutcome on the campaign becomes { "state": "promoted", "winnerKey": "story" }.
When the result is inconclusive, the campaign pauses and the reserve stays unsent. experimentOutcome becomes { "state": "inconclusive", "reason": "insufficient_evidence" } and a campaign.paused event is recorded. The decision is final: later results do not release the reserve, and resume returns 409 state_conflict. Stop the campaign to cancel the reserve, or duplicate it to run a new test.
Plan the window so the test group can be sent and answer in time. A send window, a pause, or throttling that keeps test recipients queued past the deadline ends the test inconclusive.
Follow the results#
API Get project campaign analytics returns an experiment object for an A/B campaign, and null for a broadcast:
{
"success": true,
"data": {
"campaignId": "01924f1e-6a2b-7c3d-8e4f-5a6b7c8d9e0f",
"recipientCount": 1000,
"sentCount": 178,
"experiment": {
"criterion": "reply",
"outcome": null,
"holdoutCount": 102,
"reserveCount": 718,
"variants": [
{ "key": "short", "label": "Short", "weight": 50, "assigned": 91,
"sent": 90, "delivered": 88, "read": 61, "replied": 9, "outcomeRate": 0.1 },
{ "key": "story", "label": "Story", "weight": 50, "assigned": 89,
"sent": 88, "delivered": 86, "read": 70, "replied": 23, "outcomeRate": 0.2614 }
]
}
}
}The response also carries the campaign's other analytics fields. outcomeRate counts every result received so far, so after the deadline it can differ from the rate the decision used. The Platform API returns the same object from API Get campaign analytics. In the Console, the campaign's page shows the results.
Leaving the beta#
When a team withdraws from the beta, or the beta is no longer offered to it, new A/B campaigns are refused. A running A/B campaign pauses before its next send with a campaign.paused event, and its reserve stays unsent. Messages already sent, saved assignments and results remain readable, and a paused campaign can still be stopped. Resuming it requires the beta again.
Limits#
- Test sends of an A/B draft are not supported; they return
409 state_conflict. - A/B tests send from linked WhatsApp numbers and test numbers. A campaign that would send from a Cloud API Number pauses before its first send.
- An A/B campaign cannot use a template.