Statistical Significance Calculator
Your variant B converted 15% better than variant A. Is that a real win or random noise? Statistical significance is the test that answers it, and it is two tests in one: compare two conversion rates with a two-proportion z-test, or turn any z-statistic into a p-value. Most online tools do exactly one of those and leave you to find the other. This calculator does both, then goes further than any ranking competitor with the confidence interval for the difference and a plain-English verdict, so the number comes with its interpretation attached.
Choose A/B test mode for experiment results (visitors and conversions for both variants) or p-value mode when you already have a test statistic. Pick your confidence level and tail, and the calculator returns the z-score, the p-value, the verdict, and the full working.
Quick Answer
Variant A: 5,000 visitors and 250 conversions (5.00%). Variant B: 5,200 visitors and 300 conversions (5.77%). The two-proportion z-test gives z = 1.719 and a two-tailed p-value of 0.0855, which is above the 0.05 threshold, so the result is not statistically significant at 95% confidence. The 15.4% relative uplift looks exciting, but the 95% confidence interval runs from -0.11 to +1.64 percentage points, which includes zero: this could easily be noise.
Key takeaways
- A p-value is the probability of seeing data this extreme if the null hypothesis were true. It is not the probability that your hypothesis is true.
- The standard bar is p < 0.05 (95% confidence), but that bar is a convention, not a law of nature: treat 0.049 and 0.051 as nearly identical evidence.
- Always check the confidence interval, not just the verdict. An interval that includes zero means the true effect could be nothing.
- Use two-tailed tests by default. One-tailed tests make significance easier to reach, which is exactly why they invite skepticism.
- Quick Answer
- What statistical significance actually means
- A/B test mode: comparing two conversion rates
- P-value mode: from a z-statistic to a decision
- One-tailed vs two-tailed: pick before you look
- The p-value misreadings that ruin conclusions
- What competing calculators miss
- Frequently asked questions
- Add This Calculator to Your Website
- References
What statistical significance actually means
Every significance test asks one narrow question: if nothing were really different, how surprising would this data be? The p-value quantifies that surprise. A p-value of 0.0855 means that even with no real difference between the variants, random sampling alone would produce a gap this big about 8.6% of the time. At the conventional 0.05 threshold, that is not surprising enough to declare a winner. Note what the test does not say: it does not say the variants are equal, it does not measure how big the effect is, and it certainly does not say there is a 91.45% chance B is better. Those are the three misreadings that ruin more conclusions than any arithmetic error.
Two-tailed p-value: p = 2 x (1 - Phi(|z|)), where Phi is the standard normal CDF
Decision: significant if p < alpha (alpha = 1 - confidence level)
A/B test mode: comparing two conversion rates
This is the workhorse mode and the one behind most "statistical significance calculator" searches. Enter visitors and conversions for both variants, measured over the same time window. The calculator computes each conversion rate, pools them under the null hypothesis (which assumes no difference), and derives the z-score from the pooled standard error. The worked example above walks through real numbers: a pooled rate of 0.0539, a standard error of 0.00447, and z = 1.719.
The confidence interval uses the unpooled standard error, which is the better choice for interval estimation even though the test itself pools. In the example, the 95% interval spans -0.11 to +1.64 percentage points. Because zero sits inside that range, the data remain compatible with no effect at all, which is the interval's way of saying what the p-value said: not significant.
P-value mode: from a z-statistic to a decision
Sometimes you already have the test statistic from your own analysis: a z of 2.13 from a regression output, a z of -1.2 from a survey comparison. Enter it and the calculator converts it to a p-value using the standard normal distribution, computed with a high-accuracy error-function approximation rather than a lookup table. A z of 1.96 two-tailed gives p = 0.0500, right on the conventional boundary, which is exactly why 1.96 is the number every statistics student memorizes. One-tailed mode credits only the predicted direction: a z of 2.13 one-tailed gives p = 0.0166, significant at 0.05, while the same statistic two-tailed would need p < 0.05 against double the tail area.
One-tailed vs two-tailed: pick before you look
The tail choice must be made before seeing the data, because it changes the bar. Two-tailed tests split alpha across both directions (a difference in either direction counts), which is the honest default when you would act on B losing too. One-tailed tests put all of alpha in one direction, making significance easier to reach, and reviewers rightly treat that as a red flag unless the direction was pre-registered and only one direction is actionable. The calculator defaults to two-tailed; switching to one-tailed after peeking at the results is p-hacking with extra steps.
The p-value misreadings that ruin conclusions
The American Statistical Association's 2016 statement on p-values exists because these errors are so persistent. Keep this table next to the calculator:
| What people say | What it actually means |
|---|---|
| "p = 0.085, so there is a 91.5% chance B is better" | Wrong. The p-value says nothing about the probability your hypothesis is true. It only describes the data under the null. |
| "Not significant, so the variants are equal" | Wrong. Absence of evidence is not evidence of absence. The test may simply lack power; check the confidence interval. |
| "p = 0.049 vs p = 0.051 are totally different" | Wrong. They are nearly identical evidence. The 0.05 line is a convention, and hundreds of scientists have argued it should not be a bright line at all. |
| "Significant, so the effect is large and important" | Wrong. With enough data, trivially small effects become significant. Practical importance comes from the effect size and the interval, not the p-value. |
| "I tested five variants and one is significant" | Suspicious. Multiple comparisons inflate false positives. What looks like a win may just be the luckiest of five coin flips. |
What competing calculators miss
The significance tools ranking today are each missing half the job. Pearson's p-value calculator is beautifully built, with Z, t, chi-square, and F distributions and step-by-step working, but it cannot run an A/B test: you must compute the z-score yourself first. MiniWebtool splits the job across two separate tools, a p-value calculator and an A/B test calculator, so users bounce between pages, and its A/B tool gates features behind a daily usage limit. Complete.tools and the StudyPug-style explainers cover the A/B test but never touch the general p-value case. CalculatorsHub's F-ratio page is a niche ANOVA tool dressed as a general answer. None of them pair the verdict with the confidence interval in one view, and most present the p-value without any warning about the misreadings above. This calculator's winning moves are the two modes in one tool, the pooled z-test with the unpooled confidence interval (the statistically correct pairing), the plain-English verdict badge tied to your chosen confidence level, and the full step-by-step working with the misreading guardrails right beside it.
Frequently asked questions
By convention, below 0.05 (95% confidence). But 0.05 is an arbitrary threshold: treat p-values as continuous evidence, always look at the confidence interval and effect size, and remember that many statisticians argue against bright-line cutoffs entirely.
It depends on your baseline conversion rate and the smallest lift you care about. Detecting a 10% relative lift on a 5% baseline at 95% confidence typically needs tens of thousands of visitors per variant. Small samples can only detect large effects, which is why most early tests come back "not significant."
Because relative uplift ignores sample size. A 15% lift on 250 conversions out of 5,000 visitors has a wide confidence interval that includes zero. The same 15% lift on 25,000 conversions would be overwhelmingly significant. Significance is about evidence strength, not effect size.
Two-tailed, unless you pre-registered a directional hypothesis and would genuinely ignore a loss. One-tailed tests make significance easier to reach, which is why switching tails after seeing the data is considered p-hacking.
Statistical significance says the effect is unlikely to be noise; practical significance says the effect is big enough to matter. With huge samples, a 0.01 percentage-point lift can be statistically significant and completely worthless. Judge the confidence interval, not just the p-value.
Not safely with standard methods. Repeatedly checking and stopping at the first green inflates your false-positive rate far above the nominal alpha. Fix your sample size or test duration in advance, or use sequential testing methods built for interim looks.
The range of effects compatible with your data. A 95% interval of -0.11 to +1.64 percentage points says the true lift could be slightly negative or as large as 1.64 points. That range is far more useful for decisions than a binary significant/not-significant label.
Add This Calculator to Your Website
Put the Statistical Significance Calculator on your own site with a single snippet. Pick a size to preview it, then copy the code. The preview only loads when you click a size.
Powered by Calculator Zilo
References
- National Institute of Standards and Technology. “NIST/SEMATECH e-Handbook of Statistical Methods: Inference for Two Proportions.” NIST e-Handbook: Two-Sample Proportion Test Accessed October 19, 2026.
- Wasserstein, R. L., and Lazar, N. A. “The ASA Statement on p-Values: Context, Process, and Purpose.” The American Statistician, 2016. ASA Statement on p-Values Accessed October 19, 2026.
- Amrhein, V., Greenland, S., and McShane, B. “Scientists Rise Up Against Statistical Significance.” Nature, 2019. Nature: Against Statistical Significance Accessed October 19, 2026.
- OpenStax. “Introductory Statistics: Two Population Means with Unknown Standard Deviations; Hypothesis Testing for Two Means and Two Proportions.” OpenStax Introductory Statistics Accessed October 19, 2026.
- National Institute of Standards and Technology. “NIST/SEMATECH e-Handbook of Statistical Methods: Critical Values of the Normal Distribution.” NIST e-Handbook: Normal Critical Values Accessed October 19, 2026.