ab-testing
Use when designing, implementing, monitoring, or analyzing A/B tests. Also use when calculating sample sizes, setting up PostHog experiments, checking statistical significance, or documenting experiment results.
| Model | Source |
|---|---|
| sonnet | pack: cro |
Full Reference
A/B Testing
Section titled “A/B Testing”Full experiment lifecycle for service businesses: hypothesis → sample size → implementation → monitoring → analysis → documentation. Built around PostHog experiments with awareness that lower traffic means larger MDE requirements.
Quick Reference
Section titled “Quick Reference”| Item | Value |
|---|---|
| Significance threshold | 95% (p < 0.05) — early stop only at 99.5% |
| Statistical power | 80% (Zβ = 0.84) |
| Max variants | 2 (control + test) |
| SRM tolerance | ±5% of 50/50 split — stop if exceeded |
| PostHog flag type | Boolean, 50/50 multivariate |
| PostHog SDK (client) | posthog-js/react — usePostHog() hook |
| PostHog SDK (server) | posthog-node — PostHog class |
| Goal event tracking | posthog.capture() with $feature_flag + $feature_flag_response |
| Experiment archive location | .claude/progress/experiments/ |
Reference Index
Section titled “Reference Index”| I want to… | File |
|---|---|
| Calculate sample size, understand MDE lookup table, write a hypothesis | reference/hypothesis-and-sample-size.md |
Usage: Read the reference file matching your current task from the index above. Each file is self-contained with code examples and inline gotchas.
Announcement
Section titled “Announcement”┏━ 🧪 ab-testing ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓ ┃ Full A/B test lifecycle — hypothesis, sample size, PostHog, and analysis ┃ ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛