Experiment Designer | Chalie Skills

What it does

Planning product experiments, writing testable hypotheses, estimating sample sizes, prioritizing tests, and interpreting A/B outcomes.

When this skill is activated, Chalie follows these steps:

Use memory to recall any prior experiments or hypotheses on this topic, then write the hypothesis in If/Then/Because format: if we change [intervention], then [metric] will change by [expected direction/magnitude], because [behavioral mechanism].
Use document to define and record metrics before running the test: one primary decision metric, guardrail metrics for quality protection, and secondary diagnostic metrics.
Use code_eval to estimate the required sample size given baseline conversion rate, minimum detectable effect, significance level (alpha), and statistical power.
Use code_eval to score the experiment using ICE: Impact (potential upside), Confidence (evidence quality), Ease (cost and speed) — ICE Score = (Impact × Confidence × Ease) / 10.
Use document to define and record stopping rules before launch: fixed sample size or fixed duration, with explicit guidance against early peeking.
After results arrive, interpret using code_eval — compare the point estimate and confidence interval to the decision threshold, check for novelty effects and segment heterogeneity.
Use document to produce a decision recommendation: ship, iterate, or abandon — with the effect size and confidence interval clearly stated alongside the p-value.
Use document to save the experiment design, results, and decision rationale for future reference.

v1 (curated)