> ## Documentation Index > Fetch the complete documentation index at: https://docs.polymorfa.com/llms.txt > Use this file to discover all available pages before exploring further. # A/B test a campaign > Send two to four messages to a test group, pick the winner by delivery, read or reply rate, and send it to the reserve audience. An A/B test sends two to four versions of a campaign message to a test group. When the test window ends, Polymorfa compares the versions on the outcome you choose and sends the winning message to the reserve: everyone not in the test group or the optional holdout. If no version is clearly ahead, the campaign pauses and the reserve gets nothing. A/B tests are a beta. A team owner or admin enrolls the team on the **Campaigns** page of the console. Until the team is enrolled and the beta is offered to it, creating or duplicating an A/B campaign, changing its variants, launching it and resuming it return `403 feature_unavailable`. A/B campaigns follow every other campaign rule in [Send a campaign](/guides/campaigns/send-a-campaign), including plans, sending numbers, opt-outs and send windows. ## How the audience is split At launch, each recipient is assigned once to one of three groups. The assignment is saved and does not change when a send is retried or the campaign is paused and resumed. | Group | Size | What it receives | | - | - | - | | Holdout | `holdoutPercent` of the audience, 0 to 30 | Nothing. Holdout recipients are marked `skipped` with `lastError` `experiment_holdout` at launch and are not retried | | Test group | `testSlicePercent` of the audience, 5 to 50, default 20 | One variant each, split by the variants' `weight` | | Reserve | Everyone else | The winning variant, after the test is decided | The split is by chance, so group sizes are close to the percentages rather than exact. A recipient skipped for another reason, such as an opt-out or a missing variable, keeps its group but is not sent. ## Create an A/B campaign Send `variants` and `variantStrategy` when you create the campaign. Each variant is a text message; `{{variable}}` placeholders are filled per recipient as in composed messages, and every variable needs a value or fallback, as described in [Send a campaign](/guides/campaigns/send-a-campaign). An A/B campaign sends its variants, so it cannot also set `templateId`. ```bash theme={null} curl -X POST "https://api.polymorfa.com/messaging/projects/$PROJECT_SLUG/campaigns" \ -H "Authorization: Bearer $POLYMORFA_KEY" \ -H "Content-Type: application/json" \ -d '{ "name": "Spring offer test", "recipientListId": "'"$AUDIENCE_ID"'", "variants": [ { "key": "short", "label": "Short", "weight": 50, "blueprint": { "version": 2, "source": "Hi {{first_name | there}}, 20% off this week." } }, { "key": "story", "label": "Story", "weight": 50, "blueprint": { "version": 2, "source": "Hi {{first_name | there}}, spring is here and so is 20% off everything until Sunday." } } ], "variantStrategy": { "winnerCriterion": "reply", "holdoutPercent": 10, "testSlicePercent": 20, "autoPromote": true, "testWindowMinutes": 240 } }' ``` | Field | Rules | | - | - | | `variants` | 2 to 4 entries | | `variants[].key` | Unique in the campaign; a lowercase letter, then up to 15 lowercase letters, digits, `_` or `-` | | `variants[].label` | 1 to 100 characters | | `variants[].weight` | Integer from 1 to 99; the weights add up to 100 | | `variants[].blueprint` | `version` `2` and `source`, 1 to 4,096 characters | | `variantStrategy.winnerCriterion` | `delivery`, `read` or `reply` | | `variantStrategy.holdoutPercent` | Integer from 0 to 30 | | `variantStrategy.testSlicePercent` | Integer from 5 to 50; defaults to 20 | | `variantStrategy.autoPromote` | Must be `true`. The winner is always sent automatically | | `variantStrategy.testWindowMinutes` | Integer from 15 to 1,440 | A request that breaks a rule returns `400 invalid_parameter` with `error.param` naming the field. While the campaign is an unlaunched draft, send new `variants` or `variantStrategy` with an update to change them, or `null` for both to make it a broadcast. After launch an update returns `409 state_conflict`. On the Platform API, send the same fields to `POST /platform/campaigns` and `PATCH /platform/campaigns/{campaignId}`. In the console, choose **A / B test** when you create a campaign. ## How the winner is chosen The test window starts when the campaign starts sending: at launch, or at `scheduledAt` for a scheduled campaign. Test-group recipients are sent during the window. Reserve recipients stay `queued` until a winner is chosen. When the window ends, Polymorfa decides once: 1. Every test send must be settled. A test recipient that is still queued or sending, whose send outcome is unknown, or whose result arrived after the deadline makes the test inconclusive. 2. For each variant, the rate is the number of test recipients delivered, read or replied to by the deadline, divided by the test recipients sent by the deadline. Results that arrive later do not change the decision. 3. Every variant needs at least 20 sent test messages. 4. The winner's rate must be clearly higher than every other variant's: the lower bound of its 95% confidence interval (Wilson score) must be above the upper bound of each other variant's interval. A higher rate alone, a tie, or overlapping intervals is inconclusive. When there is a winner, the reserve is sent the winning message and the campaign continues as usual. `experimentOutcome` on the campaign becomes `{ "state": "promoted", "winnerKey": "story" }`. When the result is inconclusive, the campaign pauses and the reserve stays unsent. `experimentOutcome` becomes `{ "state": "inconclusive", "reason": "insufficient_evidence" }` and a `campaign.paused` event is recorded. The decision is final: later results do not release the reserve, and resume returns `409 state_conflict`. Stop the campaign to cancel the reserve, or duplicate it to run a new test. Plan the window so the test group can be sent and answer in time. A send window, a pause, or throttling that keeps test recipients queued past the deadline ends the test inconclusive. ## Follow the results `GET /messaging/projects/{projectSlug}/campaigns/{id}/analytics` returns an `experiment` object for an A/B campaign, and `null` for a broadcast: ```json theme={null} { "success": true, "data": { "campaignId": "01924f1e-6a2b-7c3d-8e4f-5a6b7c8d9e0f", "recipientCount": 1000, "sentCount": 178, "experiment": { "criterion": "reply", "outcome": null, "holdoutCount": 102, "reserveCount": 718, "variants": [ { "key": "short", "label": "Short", "weight": 50, "assigned": 91, "sent": 90, "delivered": 88, "read": 61, "replied": 9, "outcomeRate": 0.1 }, { "key": "story", "label": "Story", "weight": 50, "assigned": 89, "sent": 88, "delivered": 86, "read": 70, "replied": 23, "outcomeRate": 0.2614 } ] } } } ``` The response also carries the campaign's other analytics fields. `outcomeRate` counts every result received so far, so after the deadline it can differ from the rate the decision used. The Platform API returns the same object from `GET /platform/campaigns/{campaignId}/analytics`. The console shows the results on the campaign's page. ## Leaving the beta When a team withdraws from the beta, or the beta is no longer offered to it, new A/B campaigns are refused. A running A/B campaign pauses before its next send with a `campaign.paused` event, and its reserve stays unsent. Messages already sent, saved assignments and results remain readable, and a paused campaign can still be stopped. Resuming it requires the beta again. ## Limits * Test sends of an A/B draft are not supported; they return `409 state_conflict`. * A/B tests send from linked WhatsApp numbers and test numbers. A campaign that would send from an Official (Cloud API) number pauses before its first send. * An A/B campaign cannot use a template. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.