Lenss to design experiments

Know whether your A/B test can reach significance before you build it.

Experiment briefs and test plans, checked against the traffic and baseline conversion they will actually run on.

Works on: Experiment briefs · A/B and multivariate test plans · Growth roadmaps with test timelines

How the audit runs

  1. 01

    Pin the decision the test informs

    What ships or doesn't depending on the result, and the lift that would justify it.

  2. 02

    Quote the timeline claim

    The sentence that says how fast the test will be conclusive.

  3. 03

    Work the sample size

    Baseline rate, expected lift and daily traffic from the brief, at standard confidence and power.

  4. 04

    Compute the real duration

    How long the test must run, and the smallest lift it could detect in the time planned.

Typical findings

Underpowered tests

Timelines that can only detect lifts far larger than the one expected.

Peeking

Plans to stop as soon as the result looks significant.

Wrong denominator

Traffic counted at the top of the funnel, conversion measured at the bottom.

Worked example

The same draft, read by a standard AI assistant and by Lenss. Every figure is computed from the draft.

Source draft

Experiment brief: one-page checkout

Experiment: replace the three-step checkout with a single page. Baseline. 3.0% of checkout visitors complete a purchase. Hypothesis. The single page lifts conversion by 10% relative, to 3.3%. Traffic. About 1,000 checkout visitors a day, split 50/50 between control and variant. Timeline. We will reach statistical significance within two weeks and ship the winner in sprint 14. Cost. Two sprints of engineering to build the test harness and the variant.

Pinned objective

Decide whether to ship the single-page checkout on evidence of a 10% relative lift.

Hidden assumption

That two weeks of traffic can detect a 0.3-point lift. At 1,000 visitors a day it can only detect lifts of about 27% or more.

“We will reach statistical significance within two weeks and ship the winner in sprint 14.”

If it’s wrong: The test ends inconclusive, or someone stops it early on noise and ships a false positive.

Flip threshold

Duration needed to detect a 10% lift (95% confidence, 80% power)

≈ 15.2 weeks brief plans 2 weeks

Baseline → target conversion
3.0% → 3.3%
Visitors needed per arm
53,208
Visitors needed in total
106,416
Daily checkout traffic
1,000
Smallest lift detectable in 2 weeks
≈ 27% relative

As planned, the two-sprint build produces a result that can't answer the question. Test a bigger change, or plan for 15 weeks.

Cheapest test: Run the sample-size check before sprint planning; if 15 weeks is too long, test on the full funnel or a bolder variant.

Used most by Product managers · Founders & growth

Audit your own draft now.

Paste up to 24,000 characters. You get the objective, every quoted assumption, the tensions, the flip threshold and a hardened rewrite.

Audit your draft