A/B test a campaign

An A/B test sends two to four versions of a campaign message to a test group. When the test window ends, Polymorfa compares the versions on the outcome you choose and sends the winning message to the reserve: everyone not in the test group or the optional holdout. If no version is clearly ahead, the campaign pauses and the reserve gets nothing. A/B tests are a beta. A team owner or admin enrolls the team on the Campaigns page of the console. Until the team is enrolled and the beta is offered to it, creating or duplicating an A/B campaign, changing its variants, launching it and resuming it return 403 feature_unavailable. A/B campaigns follow every other campaign rule in Send a campaign, including plans, sending numbers, opt-outs and send windows.

How the audience is split

At launch, each recipient is assigned once to one of three groups. The assignment is saved and does not change when a send is retried or the campaign is paused and resumed.

Group	Size	What it receives
Holdout	`holdoutPercent` of the audience, 0 to 30	Nothing. Holdout recipients are marked `skipped` with `lastError` `experiment_holdout` at launch and are not retried
Test group	`testSlicePercent` of the audience, 5 to 50, default 20	One variant each, split by the variants’ `weight`
Reserve	Everyone else	The winning variant, after the test is decided

The split is by chance, so group sizes are close to the percentages rather than exact. A recipient skipped for another reason, such as an opt-out or a missing variable, keeps its group but is not sent.

Create an A/B campaign

Send variants and variantStrategy when you create the campaign. Each variant is a text message; {{variable}} placeholders are filled per recipient as in composed messages, and every variable needs a value or fallback, as described in Send a campaign. An A/B campaign sends its variants, so it cannot also set templateId.

curl -X POST "https://api.polymorfa.com/messaging/projects/$PROJECT_SLUG/campaigns" \
  -H "Authorization: Bearer $POLYMORFA_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Spring offer test",
    "recipientListId": "'"$AUDIENCE_ID"'",
    "variants": [
      { "key": "short", "label": "Short", "weight": 50,
        "blueprint": { "version": 2, "source": "Hi {{first_name | there}}, 20% off this week." } },
      { "key": "story", "label": "Story", "weight": 50,
        "blueprint": { "version": 2, "source": "Hi {{first_name | there}}, spring is here and so is 20% off everything until Sunday." } }
    ],
    "variantStrategy": {
      "winnerCriterion": "reply",
      "holdoutPercent": 10,
      "testSlicePercent": 20,
      "autoPromote": true,
      "testWindowMinutes": 240
    }
  }'

Field	Rules
`variants`	2 to 4 entries
`variants[].key`	Unique in the campaign; a lowercase letter, then up to 15 lowercase letters, digits, `_` or `-`
`variants[].label`	1 to 100 characters
`variants[].weight`	Integer from 1 to 99; the weights add up to 100
`variants[].blueprint`	`version` `2` and `source`, 1 to 4,096 characters
`variantStrategy.winnerCriterion`	`delivery`, `read` or `reply`
`variantStrategy.holdoutPercent`	Integer from 0 to 30
`variantStrategy.testSlicePercent`	Integer from 5 to 50; defaults to 20
`variantStrategy.autoPromote`	Must be `true`. The winner is always sent automatically
`variantStrategy.testWindowMinutes`	Integer from 15 to 1,440

A request that breaks a rule returns 400 invalid_parameter with error.param naming the field. While the campaign is an unlaunched draft, send new variants or variantStrategy with an update to change them, or null for both to make it a broadcast. After launch an update returns 409 state_conflict. On the Platform API, send the same fields to POST /platform/campaigns and PATCH /platform/campaigns/{campaignId}. In the console, choose A / B test when you create a campaign.

How the winner is chosen

The test window starts when the campaign starts sending: at launch, or at scheduledAt for a scheduled campaign. Test-group recipients are sent during the window. Reserve recipients stay queued until a winner is chosen. When the window ends, Polymorfa decides once:

Every test send must be settled. A test recipient that is still queued or sending, whose send outcome is unknown, or whose result arrived after the deadline makes the test inconclusive.
For each variant, the rate is the number of test recipients delivered, read or replied to by the deadline, divided by the test recipients sent by the deadline. Results that arrive later do not change the decision.
Every variant needs at least 20 sent test messages.
The winner’s rate must be clearly higher than every other variant’s: the lower bound of its 95% confidence interval (Wilson score) must be above the upper bound of each other variant’s interval. A higher rate alone, a tie, or overlapping intervals is inconclusive.

When there is a winner, the reserve is sent the winning message and the campaign continues as usual. experimentOutcome on the campaign becomes { "state": "promoted", "winnerKey": "story" }. When the result is inconclusive, the campaign pauses and the reserve stays unsent. experimentOutcome becomes { "state": "inconclusive", "reason": "insufficient_evidence" } and a campaign.paused event is recorded. The decision is final: later results do not release the reserve, and resume returns 409 state_conflict. Stop the campaign to cancel the reserve, or duplicate it to run a new test. Plan the window so the test group can be sent and answer in time. A send window, a pause, or throttling that keeps test recipients queued past the deadline ends the test inconclusive.

Follow the results

GET /messaging/projects/{projectSlug}/campaigns/{id}/analytics returns an experiment object for an A/B campaign, and null for a broadcast:

{
  "success": true,
  "data": {
    "campaignId": "01924f1e-6a2b-7c3d-8e4f-5a6b7c8d9e0f",
    "recipientCount": 1000,
    "sentCount": 178,
    "experiment": {
      "criterion": "reply",
      "outcome": null,
      "holdoutCount": 102,
      "reserveCount": 718,
      "variants": [
        { "key": "short", "label": "Short", "weight": 50, "assigned": 91,
          "sent": 90, "delivered": 88, "read": 61, "replied": 9, "outcomeRate": 0.1 },
        { "key": "story", "label": "Story", "weight": 50, "assigned": 89,
          "sent": 88, "delivered": 86, "read": 70, "replied": 23, "outcomeRate": 0.2614 }
      ]
    }
  }
}

The response also carries the campaign’s other analytics fields. outcomeRate counts every result received so far, so after the deadline it can differ from the rate the decision used. The Platform API returns the same object from GET /platform/campaigns/{campaignId}/analytics. The console shows the results on the campaign’s page.

Leaving the beta

When a team withdraws from the beta, or the beta is no longer offered to it, new A/B campaigns are refused. A running A/B campaign pauses before its next send with a campaign.paused event, and its reserve stays unsent. Messages already sent, saved assignments and results remain readable, and a paused campaign can still be stopped. Resuming it requires the beta again.

Limits

Test sends of an A/B draft are not supported; they return 409 state_conflict.
A/B tests send from linked WhatsApp numbers and test numbers. A campaign that would send from an Official (Cloud API) number pauses before its first send.
An A/B campaign cannot use a template.

Get started

Tools and clients

TypeScript SDK

Connect numbers

WhatsApp

Campaigns

Your team

Testing

Help and updates

A/B test a campaign

How the audience is split

Create an A/B campaign

How the winner is chosen

Follow the results

Leaving the beta

Limits

​How the audience is split

​Create an A/B campaign

​How the winner is chosen

​Follow the results

​Leaving the beta

​Limits

How the audience is split

Create an A/B campaign

How the winner is chosen

Follow the results

Leaving the beta

Limits