A/B Testing
6 qualifying programs compared.
Compare inside a category, not across the study. Two programs here face the same buyers and the same norms, so a difference between them means something. A difference between categories mostly measures the categories.
| Axis | Adobe Target | Optimizely | Kameleoon | AB Tasty | VWO | LaunchDarkly |
|---|---|---|---|---|---|---|
| Overall | 60.95 axes | 47.55 axes | 30.35 axes | 30.25 axes | 30.25 axes | 29.85 axes |
| Transparency | 3.8 | 2.5 | 1.3 | 0.6 | 0.6 | 1.9 |
| Demand | 2.0 | 2.5 | 1.5 | 0.5 | 0.5 | 1.0 |
| Entry cost | 3.0 | 3.0 | 2.5 | 2.5 | 2.5 | 2.5 |
| Distribution | 3.0 | 1.5 | 1.0 | 2.0 | 2.0 | 0.5 |
| Economics | not applicable — nothing can be sold through this listing | not applicable — nothing can be sold through this listing | not applicable — nothing can be sold through this listing | not applicable — nothing can be sold through this listing | not applicable — nothing can be sold through this listing | not applicable — nothing can be sold through this listing |
| Terms | 2.5 | 3.5 | 2.5 | 2.5 | 2.5 | 2.5 |
n/athe axis does not apply to that program — read it as “question not asked”, never as 0.
Where they actually differ
17 of 30 checks diverge; 13 are identical across all 6| Check | Adobe Target | Optimizely | Kameleoon | AB Tasty | VWO | LaunchDarkly |
|---|---|---|---|---|---|---|
| A1 Guide public | published, and we read it | published, and we read it | published, and we read it | partly published | partly published | published, and we read it |
| A2 Requirements numeric | published, and we read it | partly published | we looked for it and it is not published | we looked for it and it is not published | we looked for it and it is not published | we looked for it and it is not published |
| A5 Changes dated | published, and we read it | partly published | we looked for it and it is not published | we looked for it and it is not published | we looked for it and it is not published | partly published |
| B1 Lead share | we looked for it and it is not published | partly published | we looked for it and it is not published | partly published | we looked for it and it is not published | we looked for it and it is not published |
| B2 Shared pipeline | partly published | partly published | we looked for it and it is not published | we looked for it and it is not published | we looked for it and it is not published | we looked for it and it is not published |
| B3 Co-marketing specified | partly published | partly published | published, and we read it | we looked for it and it is not published | partly published | partly published |
| B5 Merchandising | published, and we read it | published, and we read it | partly published | we looked for it and it is not published | we looked for it and it is not published | partly published |
| C1 No entry fee | published, and we read it | partly published | partly published | partly published | partly published | partly published |
| C2 No submission fee | partly published | published, and we read it | partly published | partly published | partly published | partly published |
| D1 Browsable directory | published, and we read it | published, and we read it | published, and we read it | published, and we read it | published, and we read it | partly published |
| D2 Earnable placement | published, and we read it | we looked for it and it is not published | we looked for it and it is not published | we looked for it and it is not published | we looked for it and it is not published | we looked for it and it is not published |
| D4 Listing control | published, and we read it | partly published | we looked for it and it is not published | we looked for it and it is not published | we looked for it and it is not published | we looked for it and it is not published |
| D5 Audience disclosed | we looked for it and it is not published | we looked for it and it is not published | we looked for it and it is not published | published, and we read it | published, and we read it | we looked for it and it is not published |
| F1 No exclusivity | partly published | published, and we read it | partly published | partly published | partly published | partly published |
| F2 No broad IP grant | partly published | published, and we read it | partly published | partly published | partly published | partly published |
| F3 No clone reservation | partly published | we looked for it and it is not published | partly published | partly published | partly published | partly published |
| F4 Fair termination | partly published | published, and we read it | partly published | partly published | partly published | partly published |
Show the 13 checks they all score the same
Agreement is a finding too: where every program in a category fails the same check, that is the category’s norm rather than one vendor’s oversight.
| Check | Adobe Target | Optimizely | Kameleoon | AB Tasty | VWO | LaunchDarkly |
|---|---|---|---|---|---|---|
| A3 Reward numeric | not applicable — nothing can be sold through this listing, so there are no economics to publish | not applicable — nothing can be sold through this listing, so there are no economics to publish | not applicable — nothing can be sold through this listing, so there are no economics to publish | not applicable — nothing can be sold through this listing, so there are no economics to publish | not applicable — nothing can be sold through this listing, so there are no economics to publish | not applicable — nothing can be sold through this listing, so there are no economics to publish |
| A4 Rules of engagement | we looked for it and it is not published | we looked for it and it is not published | we looked for it and it is not published | we looked for it and it is not published | we looked for it and it is not published | we looked for it and it is not published |
| B4 Funding | we looked for it and it is not published | we looked for it and it is not published | we looked for it and it is not published | we looked for it and it is not published | we looked for it and it is not published | we looked for it and it is not published |
| C3 Fast review | partly published | partly published | partly published | partly published | partly published | partly published |
| C4 No annual fee | partly published | partly published | partly published | partly published | partly published | partly published |
| C5 No paid tier gate | partly published | partly published | partly published | partly published | partly published | partly published |
| D3 Analytics returned | we looked for it and it is not published | we looked for it and it is not published | we looked for it and it is not published | we looked for it and it is not published | we looked for it and it is not published | we looked for it and it is not published |
| E1 Price visible on the listing | not applicable — nothing can be sold through this listing, so there are no economics to publish | not applicable — nothing can be sold through this listing, so there are no economics to publish | not applicable — nothing can be sold through this listing, so there are no economics to publish | not applicable — nothing can be sold through this listing, so there are no economics to publish | not applicable — nothing can be sold through this listing, so there are no economics to publish | not applicable — nothing can be sold through this listing, so there are no economics to publish |
| E2 They collect and remit | not applicable — nothing can be sold through this listing, so there are no economics to publish | not applicable — nothing can be sold through this listing, so there are no economics to publish | not applicable — nothing can be sold through this listing, so there are no economics to publish | not applicable — nothing can be sold through this listing, so there are no economics to publish | not applicable — nothing can be sold through this listing, so there are no economics to publish | not applicable — nothing can be sold through this listing, so there are no economics to publish |
| E3 You set the price | not applicable — nothing can be sold through this listing, so there are no economics to publish | not applicable — nothing can be sold through this listing, so there are no economics to publish | not applicable — nothing can be sold through this listing, so there are no economics to publish | not applicable — nothing can be sold through this listing, so there are no economics to publish | not applicable — nothing can be sold through this listing, so there are no economics to publish | not applicable — nothing can be sold through this listing, so there are no economics to publish |
| E4 Provisioning documented | not applicable — nothing can be sold through this listing, so there are no economics to publish | not applicable — nothing can be sold through this listing, so there are no economics to publish | not applicable — nothing can be sold through this listing, so there are no economics to publish | not applicable — nothing can be sold through this listing, so there are no economics to publish | not applicable — nothing can be sold through this listing, so there are no economics to publish | not applicable — nothing can be sold through this listing, so there are no economics to publish |
| E5 Payout terms published | not applicable — nothing can be sold through this listing, so there are no economics to publish | not applicable — nothing can be sold through this listing, so there are no economics to publish | not applicable — nothing can be sold through this listing, so there are no economics to publish | not applicable — nothing can be sold through this listing, so there are no economics to publish | not applicable — nothing can be sold through this listing, so there are no economics to publish | not applicable — nothing can be sold through this listing, so there are no economics to publish |
| F5 You keep the customer | partly published | partly published | partly published | partly published | partly published | partly published |