Two things changing together does not establish that one caused the other. First check the population and denominator, then compare plausible explanations and the timing of evidence. A count can rise while its corresponding rate falls.
An original software-support example
Imagine a team releases an update. In the earlier week, 1,000 active users opened 50 support tickets. In the later week, 2,000 active users opened 80 tickets. These are hypothetical figures, not Ithy customer data or a measured product experiment.
| Period | Active users | Tickets | Tickets per 100 active users |
|---|---|---|---|
| Before | 1,000 | 50 | 5 |
| After | 2,000 | 80 | 4 |
Ticket count change = (80 − 50) / 50 = +60%
Earlier rate = 50 / 1,000 = 5%
Later rate = 80 / 2,000 = 4%
Rate change = (4% − 5%) / 5% = −20%
Percentage-point change = 4% − 5% = −1 point
The count increased, while this rate decreased. Neither result alone proves the update improved or harmed the product. Also note that tickets per user is not necessarily the percentage of users with a problem: one person can open several tickets.
What else could explain the change?
The later population may include a different mix of new users, devices or tasks. Support instructions, reporting tools, staffing or the definition of an active user may have changed. A campaign might attract more demanding use cases. These are potential confounders and measurement changes to investigate, not facts about the hypothetical team.
NIST's scatter-plot guidance distinguishes detecting an association from establishing cause. A strong-looking chart can still mix different populations or omit a common driver.
A more informative investigation
- Define the outcome: unique affected users, tickets, severe failures or time lost?
- Use comparable periods and the exposure that can actually produce that outcome.
- Check device, task, geography and new/returning-user composition where relevant and permitted.
- Confirm the update preceded the suspected effect and identify a plausible mechanism.
- Where appropriate, use a well-designed randomized rollout or another justified causal design rather than a simple before/after label.
Segmentation should answer a substantive question, not search endlessly for a favorable subgroup. Record missing data and changes in definitions. Even an experiment needs attention to interference, implementation and the population to which its results apply.
Rates also have uncertainty
A point estimate is not the complete result. Sampling variation and systematic error can both matter, and the right uncertainty calculation depends on the data-generating process. For example, tickets clustered within users are different from independent yes/no outcomes. Do not automatically feed aggregated counts into a formula whose assumptions are not met.
Report the observation and the inference separately
A precise description is: “Tickets rose by 60%, while tickets per active user fell from 5 to 4 per 100; population and reporting changes remain possible explanations.” A causal statement such as “the update reduced problems by 20%” requires more evidence. That distinction makes a report useful even when the final causal answer remains uncertain.