# A/B Testing: p-value

2 min readNov 16

--

The p-value in A/B testing is a statistical measure that helps assess the evidence against a null hypothesis. A/B testing is a common experimental method used in fields such as marketing, product development, and healthcare to compare two versions (A and B) of a variable (e.g., a webpage, an advertisement, or a treatment) and determine which performs better. The p-value is a crucial component in interpreting the results of A/B tests.

# I. Key Points about P-Value in A/B Testing:

## 1. Definition:

- The p-value is the probability of observing the observed data or more extreme results when the null hypothesis is true.

## 2. Null Hypothesis (H0):

- In A/B testing, the null hypothesis typically states that there is no statistically significant difference between the two groups (A and B), implying that any observed difference is due to random chance.

## 3. Alternative Hypothesis (H1):

- The alternative hypothesis contradicts the null hypothesis and asserts that there is a significant difference between the groups.

## 4. Interpretation:

- A small p-value (typically less than the chosen significance level, often 0.05) suggests that the observed data is unlikely under the assumption of the null hypothesis.
- If the p-value is less than the significance level, the null hypothesis is rejected, indicating that the observed difference is statistically significant.

## 5. Decision Rule:

- Common decision rules include:
- If p-value < α (e.g., 0.05), reject the null hypothesis.
- If p-value ≥ α, fail to reject the null hypothesis.

## 6. Statistical Significance:

- A result is considered statistically significant if the p-value is below the significance level. This means that the observed difference is unlikely to be purely due to chance.

## 7. Not a Measure of Effect Size:

- The p-value does not quantify the size or practical importance of the observed effect. It only indicates whether the observed effect is statistically significant.