Literature-grounded experimental design

Using Praxis for experimental design

This tool evaluates statistical designs that use a Null-Hypothesis Statistical Test to make inferences. It can be used both as a sample size calculator and as a statistical power calculator. Usually one would determine the sample size required given a particular power requirement. In cases where there is a predetermined sample size one can instead calculate power for a given effect size of interest (a.k.a. perform a power analysis). Praxis is used across life sciences, wet-lab bench science, dry-lab bioinformatics, clinical research, pharmaceutical R&D, and public health surveillance, with reporting-standard alignment in the certificate that adapts to the research context.

Parameters for sample size and power calculation

1. Number of test groups. Praxis supports experiments gathering data on a single sample to compare against a general population or known reference value (one-sample), a control group compared to a treatment group (two-sample), or three or more groups compared simultaneously (k-sample). The specific test, t-test, ANOVA, z-test, chi-square, or log-rank, depends on this group-count dimension combined with the outcome type and the within-subject correlation structure (see Type of outcome and Pairing). For three or more groups Praxis sizes a standard one-way ANOVA (an omnibus F-test) assuming approximately equal effect sizes and equal group sizes; it does not yet apply a per-contrast Dunnett’s correction for pairwise treatment-versus-control comparisons. Pairing within a two-sample design (the paired t-test) takes advantage of within-subject correlation and typically requires fewer subjects than the equivalent independent design.

2. Type of outcome. The outcome can be the absolute difference of two means (continuous data, e.g. expression level, OD600, CFU count, plate-reader signal), the absolute difference of two proportions (binomial data, e.g. response rate, resistance rate, success rate), or the relative difference between two means or two proportions (percent difference, percent change). For two-mean comparisons with potentially unequal variances, Welch’s correction is available. For time-to-event outcomes where the variable of interest is the time until an event occurs (e.g. relapse, death, culture conversion), the log-rank test is used with the hazard ratio as the effect-size metric, under the assumption of proportional hazards and exponential survival.

3. Baseline. The mean under H₀ is the parameter value one would expect if all participants were assigned to the control group, the mean expected if the treatment has no effect whatsoever. If entering means data, specify the mean under the null hypothesis and the standard deviation of the data, from a known population or estimated from a sample. When a Lumen session is loaded, Praxis pools a standardised effect (Cohen’s d) from comparable studies rather than σ directly, full-text Methods sections recover substantially more standard-deviation and effect-size reporting than abstracts alone^[1].

4. Minimum detectable effect. The minimum effect of interest, often called the minimum detectable effect (MDE, more accurately MRDE, minimum reliably detectable effect), should be a difference one would not like to miss if it existed^[2]. Enter it on the same scale as the outcome, Cohen’s d for two-mean comparisons, Cohen’s f for ANOVA, the proportion difference for binomial data, or the hazard ratio for time-to-event. The MDE is always relative to the mean or proportion under H₀ ± the superiority, non-inferiority, or equivalence margin. When a Lumen session is loaded, this field pre-fills with the median effect pooled from comparable published studies, accept the pooled value, override it (a divergence of more than 25% triggers a warning), or skip Lumen entirely and enter a custom value.

5. Type of alternative hypothesis. Praxis supports superiority, non-inferiority, and equivalence alternative hypotheses. The equivalence margin cannot be zero. Non-inferiority and equivalence designs are continuous-only in the current build; binomial non-inferiority is on the roadmap. Non-inferiority and equivalence trial reporting follows the CONSORT 2010 extension^[3].

6. Maximum error rates. The Type I error rate, α, should always be provided. It is the cut-off point for the p-value calculation and equals 1 − the confidence level of the corresponding confidence interval. Power, calculated as 1 − β where β is the Type II error rate, is only required when determining a sample size; in a power-analysis run the power is the output rather than an input.

Praxis output

In Sample-size-calculation mode, Praxis returns the required sample size per group and the total across all groups. In Power-analysis mode, the output is the achieved power as a proportion (0–1) and as a percentage. Both modes also return the effective effect size used (literature-pooled, hybrid, or custom), the achieved power at the rounded-up sample size, and the G*Power reference-match status. Beyond the headline numbers, Praxis returns contextual outputs that distinguish it from a standalone calculator:

Why sample size determination matters

Estimating the required sample size before running a study that will be judged by a statistical test allows one to:

• determine the sample size needed to detect an effect of a given magnitude with a given probability;
• estimate the magnitude of the effect that can be reliably detected with a chosen sample size;
• calculate statistical power for a given sample size and effect size of interest.

This is critical for making a study cost-efficient and ethical. A study with statistical power below 80% (β > 0.20) for a meaningful effect has a low probability of rejecting the null even when the effect is real, making the study unlikely to produce an informative answer either way. In medical or biological research this concern is amplified: an underpowered study consumes participants, time, and funding while contributing little to the record, and may be ethically problematic when the resources could have supported a sufficiently powered design.

The reproducibility literature has documented this at scale. A 2016 survey of 1,576 researchers reported that more than 70% had failed to reproduce another scientist’s experiments, and more than half had failed to reproduce their own^[4]. An audit of 53 landmark preclinical cancer studies found only 6 (11%) could be independently confirmed^[5]. Underpowered designs are one contributor; effect-size inflation in the published literature compounds the problem because subsequent researchers anchor their power calculations on upward-biased estimates. Sample-size justification should therefore be argued explicitly, from a literature-pooled effect, pilot data, or a pre-specified smallest effect of interest, rather than pulled from convention^[2]. Praxis surfaces this justification at the point of design and records it in the reproducibility certificate.

Statistical power explained

Statistical power is the probability of rejecting a false null hypothesis at a given significance level, against a particular alternative. Equivalently, it is the probability of detecting a true effect of a certain magnitude at the chosen α, the correct interpretation of the Praxis output in Power-analysis mode. Power is inversely related to the Type II error rate β, since it equals (1 − β). The Type II error at a point alternative μ₁ is^[6]:

β(T_α; μ₁) = P(d(X) ≤ c_α; μ = μ₁)

And statistical power, since POW = 1 − β^[6]:

POW(T_α; μ₁) = P(d(X) > c_α; μ = μ₁)

Here c_α is the critical value for rejecting the null (the significance threshold), d(X) is a statistic of the parameter of interest (usually a transformation to a standardised score), and μ₁ is a specific value from the alternative-hypothesis space. The whole power function is traced by computing power at many points under the alternative. Due to its S-shape, power rises rapidly toward 100% for effects larger than the MDE and decreases more gradually toward zero for smaller effects. Praxis returns this power function with every calculation through the sensitivity analysis. At the zero-effect point of a simple superiority alternative, power equals exactly (1 − α), as can be shown by reducing the effect size to zero. Power is also positively related to the number of observations.

Post-hoc power (observed power)

A power calculation can be useful even after a test, since failing to reject the null can be an argument for the null against particular alternatives, to the extent the test had power to reject them, defined more explicitly in Mayo’s severe-testing framework^[6]. Computing observed power is only useful when the test failed to reject the null. It is uninformative to compute post-hoc power for a significant result^[7]: if the effect is significant, the test by definition had enough power to detect it, and there is a near one-to-one inverse relationship between observed power and the p-value. A test planned at α = 0.05 that passed with p = 0.0499 will have exactly 50% observed power. When the user runs a power analysis on a setup that looks like a post-hoc calculation on a significant result, Praxis flags this in the result panel and records the flag in the certificate.

∑Sample size formula

The sample size per group in a one-sided test of absolute difference is:

where Z_1−α is the Z-score for the significance threshold α, Z_1−β is the Z-score for the power (1 − β), σ is the known or estimated standard deviation, and δ is the minimum effect of interest. The formula uses the Z (normal) distribution. The standard deviation is estimated analytically for proportions, and supplied from the Lumen-pooled corpus, pilot data, or prior knowledge for continuous outcomes. It applies to one- and two-sample absolute-difference tests; the relative-difference case uses an adapted form that accounts for the extra variance from the baseline estimate. Praxis’s calculations are built on statsmodels and parity-checked against G*Power within 1% across 40 reference cases.

Types of null and alternative hypotheses in significance tests

Every Praxis calculation rests on an explicit choice of null and alternative hypothesis. The null (H₀) is the position the test argues against; the alternative (H₁) is the position it argues for. Praxis exposes this as the Alternative-hypothesis radio in the form. Each configuration places the null and alternative in a different region of the parameter space, and the choice directly determines the sample size required, the margin to be specified, and the reporting standard the certificate is aligned against.

Superiority is the most common choice in life-sciences research. H₀ is “no difference”; H₁ is “the conditions differ.” Praxis supports both two-sided (no directional commitment) and one-sided (a pre-specified direction). One-sided superiority cuts the required sample size by roughly a third for the same power, but the directional commitment must be defensible from prior data, Praxis records the one-sided flag in the certificate.

Non-inferiority is the workhorse of biosimilar trials, antibiotic comparisons, and any setting aiming to show the new condition is not meaningfully worse than the reference. H₀ is “the new condition is worse by more than the margin Δ”; H₁ is “at most Δ worse.” The margin Δ is a substantive scientific decision and Praxis requires it explicitly. Reporting aligns to the CONSORT 2010 non-inferiority extension^[3].

Strong superiority (superiority with a margin) shifts the null further into the rejection region. H₀ is “any difference smaller than the margin is not enough”; H₁ requires the effect to exceed it, the right configuration when a statistically detectable but clinically trivial improvement is not sufficient. The margin is recorded in the certificate.

Equivalence is evaluated through two one-sided tests (the TOST procedure). H₀ is “the conditions differ by more than the equivalence margin in either direction”; H₁ is “within ±δ of each other.” This is the right configuration for claims that two protocols, pipelines, or formulations are interchangeable. The equivalence margin cannot be zero.

Absolute versus relative difference and why it matters

Praxis can evaluate either the absolute difference between conditions (a raw gap, μ₁ − μ) or the relative difference (a proportional change, (μ₁ − μ) / μ, often a percentage). The two framings answer different questions: an absolute difference of 0.5 ng/mL is unambiguous, but a 50% relative reduction is only meaningful if the baseline μ is known and stable.

The framing affects the required sample size. Relative-difference tests divide by μ, itself a variance-carrying estimate; the division compounds the noise and increases the sample size needed for a given power. The smaller and noisier the baseline, the larger the inflation. The framing must be committed to before the data are seen, choosing after looking at the data is a researcher-degrees-of-freedom problem that inflates the false-positive rate. Praxis exposes the choice as the Inference radio and captures the selection in the certificate at design time, so a reviewer can verify it was pre-specified.

How Praxis grounds effect sizes in the literature

Effect-size estimates are the single largest source of error in sample-size calculations. Researchers consistently overestimate them when guessing from memory, and underestimated effect sizes lead to underpowered studies that fail to replicate^[4]. Praxis pools a literature-grounded effect size from the Lumen corpus directly into the form, with a three-band confidence label (HIGH / MEDIUM / LOW) indicating how much the pool can be trusted. Alongside the pooled effect, Praxis returns a confounder list, variables comparable studies controlled for, ranked by paper count and shown with citation evidence, a starting point for the randomisation, blocking, and adjustment conversation, not a definitive list for any specific study. Three modes govern how the pool is used: Literature-grounded uses the pooled median directly; Custombypasses Lumen and accepts a user value; Hybrid uses the user’s value but keeps the pool visible and warns when the override diverges by more than 25%^[2].

The pitfall audit

Every Praxis calculation runs an audit against the user’s original spec, not the literature-substituted version, so it can fire when an optimistic assumption is itself the problem. Each rule returns HIGH, MEDIUM, or LOW severity with a concrete remediation, sorted top-to-bottom by severity. The audit covers three classes: effect-size sanity (flags an expected effect more than twice the literature median, or a one-sided test paired with an unusually large expected effect); sample-size feasibility (flags required n below five per group, or allocation more extreme than 2:1 without justification); and literature support (flags a pool drawing on fewer than three comparable papers, or no extractable effect-size evidence at all). Each finding includes the rule that fired, the threshold crossed, and a one-line remediation. The full audit is captured in the certificate so a reviewer can see which warnings were issued and which were accepted.

The reproducibility certificate

Every Praxis calculation produces a certificate, a structured, on-screen record of every decision that went into the design: the effect size and its source (Lumen pool or custom value), α, target power, design type and hypothesis, the test direction (one-sided flag), the absolute-versus-relative framing, the allocation ratio, the G*Power parity status, the pinned software versions (statsmodels, scipy), and the IntendedUse.RESEARCH_ONLY annotation, all captured at the moment of calculation. The computation is deterministic, so the same inputs always reproduce the same number (no random seed is involved).

The certificate also lists the reporting standards Praxis aligns to for the chosen research context: ARRIVE 2.0^[16] and MIQE for wet-lab in-vivo and qPCR work, MINSEQE and the FAIR principles for dry-lab analyses, CONSORT 2010^[3], STROBE, and SPIRIT for clinical research, ICH E9 / GLP / 21 CFR Part 11 for pharma R&D, and STROBE / PRISMA 2020^[12] / GATHER for public-health surveillance. It is the artifact a peer reviewer, regulator, or collaborator would inspect to answer the question every design review asks first, “where did this number come from?”, with provenance rather than memory.