Skip to main content

A/B Testing

6 qualifying programs compared.

Compare inside a category, not across the study. Two programs here face the same buyers and the same norms, so a difference between them means something. A difference between categories mostly measures the categories.

Axis scores for each program in A/B Testing
AxisAdobe TargetOptimizelyKameleoonAB TastyVWOLaunchDarkly
Overall60.95 axes47.55 axes30.35 axes30.25 axes30.25 axes29.85 axes
Transparency3.82.51.30.60.61.9
Demand2.02.51.50.50.51.0
Entry cost3.03.02.52.52.52.5
Distribution3.01.51.02.02.00.5
Economicsnot applicable — nothing can be sold through this listingnot applicable — nothing can be sold through this listingnot applicable — nothing can be sold through this listingnot applicable — nothing can be sold through this listingnot applicable — nothing can be sold through this listingnot applicable — nothing can be sold through this listing
Terms2.53.52.52.52.52.5

n/athe axis does not apply to that program — read it as “question not asked”, never as 0.

Where they actually differ

17 of 30 checks diverge; 13 are identical across all 6
The checks on which these programs differ, with each program’s value
CheckAdobe TargetOptimizelyKameleoonAB TastyVWOLaunchDarkly
A1 Guide publicpublished, and we read itpublished, and we read itpublished, and we read itpartly publishedpartly publishedpublished, and we read it
A2 Requirements numericpublished, and we read itpartly publishedwe looked for it and it is not publishedwe looked for it and it is not publishedwe looked for it and it is not publishedwe looked for it and it is not published
A5 Changes datedpublished, and we read itpartly publishedwe looked for it and it is not publishedwe looked for it and it is not publishedwe looked for it and it is not publishedpartly published
B1 Lead sharewe looked for it and it is not publishedpartly publishedwe looked for it and it is not publishedpartly publishedwe looked for it and it is not publishedwe looked for it and it is not published
B2 Shared pipelinepartly publishedpartly publishedwe looked for it and it is not publishedwe looked for it and it is not publishedwe looked for it and it is not publishedwe looked for it and it is not published
B3 Co-marketing specifiedpartly publishedpartly publishedpublished, and we read itwe looked for it and it is not publishedpartly publishedpartly published
B5 Merchandisingpublished, and we read itpublished, and we read itpartly publishedwe looked for it and it is not publishedwe looked for it and it is not publishedpartly published
C1 No entry feepublished, and we read itpartly publishedpartly publishedpartly publishedpartly publishedpartly published
C2 No submission feepartly publishedpublished, and we read itpartly publishedpartly publishedpartly publishedpartly published
D1 Browsable directorypublished, and we read itpublished, and we read itpublished, and we read itpublished, and we read itpublished, and we read itpartly published
D2 Earnable placementpublished, and we read itwe looked for it and it is not publishedwe looked for it and it is not publishedwe looked for it and it is not publishedwe looked for it and it is not publishedwe looked for it and it is not published
D4 Listing controlpublished, and we read itpartly publishedwe looked for it and it is not publishedwe looked for it and it is not publishedwe looked for it and it is not publishedwe looked for it and it is not published
D5 Audience disclosedwe looked for it and it is not publishedwe looked for it and it is not publishedwe looked for it and it is not publishedpublished, and we read itpublished, and we read itwe looked for it and it is not published
F1 No exclusivitypartly publishedpublished, and we read itpartly publishedpartly publishedpartly publishedpartly published
F2 No broad IP grantpartly publishedpublished, and we read itpartly publishedpartly publishedpartly publishedpartly published
F3 No clone reservationpartly publishedwe looked for it and it is not publishedpartly publishedpartly publishedpartly publishedpartly published
F4 Fair terminationpartly publishedpublished, and we read itpartly publishedpartly publishedpartly publishedpartly published
Show the 13 checks they all score the same

Agreement is a finding too: where every program in a category fails the same check, that is the category’s norm rather than one vendor’s oversight.

The checks on which every program in A/B Testing scores the same, with each program’s value
CheckAdobe TargetOptimizelyKameleoonAB TastyVWOLaunchDarkly
A3 Reward numericnot applicable — nothing can be sold through this listing, so there are no economics to publishnot applicable — nothing can be sold through this listing, so there are no economics to publishnot applicable — nothing can be sold through this listing, so there are no economics to publishnot applicable — nothing can be sold through this listing, so there are no economics to publishnot applicable — nothing can be sold through this listing, so there are no economics to publishnot applicable — nothing can be sold through this listing, so there are no economics to publish
A4 Rules of engagementwe looked for it and it is not publishedwe looked for it and it is not publishedwe looked for it and it is not publishedwe looked for it and it is not publishedwe looked for it and it is not publishedwe looked for it and it is not published
B4 Fundingwe looked for it and it is not publishedwe looked for it and it is not publishedwe looked for it and it is not publishedwe looked for it and it is not publishedwe looked for it and it is not publishedwe looked for it and it is not published
C3 Fast reviewpartly publishedpartly publishedpartly publishedpartly publishedpartly publishedpartly published
C4 No annual feepartly publishedpartly publishedpartly publishedpartly publishedpartly publishedpartly published
C5 No paid tier gatepartly publishedpartly publishedpartly publishedpartly publishedpartly publishedpartly published
D3 Analytics returnedwe looked for it and it is not publishedwe looked for it and it is not publishedwe looked for it and it is not publishedwe looked for it and it is not publishedwe looked for it and it is not publishedwe looked for it and it is not published
E1 Price visible on the listingnot applicable — nothing can be sold through this listing, so there are no economics to publishnot applicable — nothing can be sold through this listing, so there are no economics to publishnot applicable — nothing can be sold through this listing, so there are no economics to publishnot applicable — nothing can be sold through this listing, so there are no economics to publishnot applicable — nothing can be sold through this listing, so there are no economics to publishnot applicable — nothing can be sold through this listing, so there are no economics to publish
E2 They collect and remitnot applicable — nothing can be sold through this listing, so there are no economics to publishnot applicable — nothing can be sold through this listing, so there are no economics to publishnot applicable — nothing can be sold through this listing, so there are no economics to publishnot applicable — nothing can be sold through this listing, so there are no economics to publishnot applicable — nothing can be sold through this listing, so there are no economics to publishnot applicable — nothing can be sold through this listing, so there are no economics to publish
E3 You set the pricenot applicable — nothing can be sold through this listing, so there are no economics to publishnot applicable — nothing can be sold through this listing, so there are no economics to publishnot applicable — nothing can be sold through this listing, so there are no economics to publishnot applicable — nothing can be sold through this listing, so there are no economics to publishnot applicable — nothing can be sold through this listing, so there are no economics to publishnot applicable — nothing can be sold through this listing, so there are no economics to publish
E4 Provisioning documentednot applicable — nothing can be sold through this listing, so there are no economics to publishnot applicable — nothing can be sold through this listing, so there are no economics to publishnot applicable — nothing can be sold through this listing, so there are no economics to publishnot applicable — nothing can be sold through this listing, so there are no economics to publishnot applicable — nothing can be sold through this listing, so there are no economics to publishnot applicable — nothing can be sold through this listing, so there are no economics to publish
E5 Payout terms publishednot applicable — nothing can be sold through this listing, so there are no economics to publishnot applicable — nothing can be sold through this listing, so there are no economics to publishnot applicable — nothing can be sold through this listing, so there are no economics to publishnot applicable — nothing can be sold through this listing, so there are no economics to publishnot applicable — nothing can be sold through this listing, so there are no economics to publishnot applicable — nothing can be sold through this listing, so there are no economics to publish
F5 You keep the customerpartly publishedpartly publishedpartly publishedpartly publishedpartly publishedpartly published