Lenss to design experiments
Know whether your A/B test can reach significance before you build it.
Experiment briefs and test plans, checked against the traffic and baseline conversion they will actually run on.
Works on: Experiment briefs · A/B and multivariate test plans · Growth roadmaps with test timelines
How the audit runs
01
Pin the decision the test informs
What ships or doesn't depending on the result, and the lift that would justify it.
02
Quote the timeline claim
The sentence that says how fast the test will be conclusive.
03
Work the sample size
Baseline rate, expected lift and daily traffic from the brief, at standard confidence and power.
04
Compute the real duration
How long the test must run, and the smallest lift it could detect in the time planned.
Typical findings
Underpowered tests
Timelines that can only detect lifts far larger than the one expected.
Peeking
Plans to stop as soon as the result looks significant.
Wrong denominator
Traffic counted at the top of the funnel, conversion measured at the bottom.
Worked example
The same draft, read by a standard AI assistant and by Lenss. Every figure is computed from the draft.
Source draft
Experiment brief: one-page checkout
Pinned objective
Decide whether to ship the single-page checkout on evidence of a 10% relative lift.
Hidden assumption
That two weeks of traffic can detect a 0.3-point lift. At 1,000 visitors a day it can only detect lifts of about 27% or more.
“We will reach statistical significance within two weeks and ship the winner in sprint 14.”
If it’s wrong: The test ends inconclusive, or someone stops it early on noise and ships a false positive.
Flip threshold
Duration needed to detect a 10% lift (95% confidence, 80% power)
≈ 15.2 weeks brief plans 2 weeks
- Baseline → target conversion
- 3.0% → 3.3%
- Visitors needed per arm
- 53,208
- Visitors needed in total
- 106,416
- Daily checkout traffic
- 1,000
- Smallest lift detectable in 2 weeks
- ≈ 27% relative
As planned, the two-sprint build produces a result that can't answer the question. Test a bigger change, or plan for 15 weeks.
Cheapest test: Run the sample-size check before sprint planning; if 15 weeks is too long, test on the full funnel or a bolder variant.
Used most by Product managers · Founders & growth
Audit your own draft now.
Paste up to 24,000 characters. You get the objective, every quoted assumption, the tensions, the flip threshold and a hardened rewrite.