The problem is that statistical significance can make a result sound more conclusive than it is. It tells the team the change is unlikely to be random, but not whether the product made a meaningful difference.

More specifically, that number doesn’t tell us whether a two-point change is large enough to matter clinically, whether it’s still true three months later, or whether the average is being skewed by the handful of users who logged in every day while everyone else’s scores barely moved.

When we review outcomes data with digital health companies, the topline result is rarely where the conversation ends. A score can move in the right direction and still leave important questions unanswered: Does the change matter? Does it hold? Compared to what? And for whom?

Those questions determine how much confidence anyone should place in the result. They mark the difference between evidence that holds up in an investor meeting and a claim that falls apart the moment someone asks a follow-up question.

A Lower Score Starts the Conversation. It Doesn’t Win It.

A statistically significant drop in a validated symptom score is a useful signal, but it cannot tell us whether the change was large enough to matter to users or whether the product caused it.  A result like this represents the floor of outcomes evidence. Here, the floor is the minimum needed to take the result seriously, not enough to conclude that the product works.

Moving beyond that floor means showing that the improvement was meaningful, lasted over time, can reasonably be attributed to the product, and reached the intended users. The initial result may be enough to open a conversation with an investor, health system, or payer, but it is rarely enough to answer all the questions that follow.

Statistical and clinical significance often get treated as synonyms, but they answer different questions. Statistical significance helps assess whether a result could reasonably be explained by chance. Clinical significance asks whether the change was large enough to matter in someone’s life.

With enough participants, even a very small difference can clear the bar for statistical significance. Researchers have found results that were statistically significant even when the actual effect was small enough to be potentially trivial. In our example, that could mean the average GAD-7 score fell two points across a large user base while many individual users stayed within the same symptom range and noticed little or no difference in their daily lives.

This is where the minimal clinically important difference, or MCID, comes in. It provides a benchmark for whether the size of a change is likely to matter to patients or be relevant to their care. A drop in a symptom score can be statistically real and still fall below that benchmark.

For the GAD-7 specifically, published estimates of meaningful change vary depending on the population, baseline severity, and method used. That variation is part of the point. There is no single number that can be applied blindly to every study. Meaningful change needs to be interpreted in the context of the specific measure, population, and claim being evaluated.

Using a validated measure means the underlying tool is credible. It does not mean every movement on that tool matters. Making that claim requires putting the size of the change in context.

What Else Gets Missed When the Score Is the Whole Story

Once a validated symptom score starts moving in the right direction, the topline number can quickly become the whole pitch. But stakeholders will want to know what sits behind that number.

Before using a digital health outcome as evidence, ask four questions: Does the change matter? Does it hold over time? Compared to what? And for whom?

• Does it matter? As the GAD-7 example shows, a two-point drop may be statistically significant without producing a noticeable difference in users’ daily lives. Effect size, which describes how large the change was, and meaningful-change thresholds help distinguish a small movement from an improvement that is likely to matter.
• Does it hold? An improvement at eight weeks may fade by six months. If measurement stops when the program ends, it is impossible to tell whether the product created lasting change or a temporary lift. Recent reviews of digital mental health research call for standardized, longer follow-up periods so companies can show whether gains hold over time.
• Compared to what? A before-and-after result shows that users changed, but not why. They may have enrolled when their symptoms were at their worst, started therapy or medication at the same time, or improved naturally over time. A control group, waitlist, or standard-of-care benchmark helps separate the product’s effect from changes that might have happened anyway.
• For whom? An average improvement may be driven by highly engaged users who completed every session, while people who used the product less saw little benefit or dropped out entirely. The same review also found limited representation from marginalized groups, which raises another question: Have these outcomes been demonstrated in the populations that health systems, payers, and employers actually need to serve?

Not every gap requires a larger study or a bigger budget. Some can be addressed by reporting the data you already have more clearly. Others reveal the next evidence investment your company needs to make. The goal is to understand how far your current evidence can carry the claim, and understand what evidence to build next. 

What This Looks Like in Practice

Start by being transparent and precise about what the result shows.

Instead of saying, “Our product works,” a company might say: “Average GAD-7 scores fell two points over eight weeks among the users we studied. The change was statistically significant, but we have not yet established whether it was large enough to matter clinically, whether it lasts, or how much of the improvement can be attributed to the product.”

Then ask what your existing data can tell you beyond the average. How many users improved enough to notice a difference? Did the results vary based on how severe their symptoms were at the start, how much they used the product, or across demographic groups? And if you have follow-up data, did the improvement last?

Founders are not expected to run these analyses themselves. If your team does not have this expertise in-house, this is where we often step in. At Fit Minded, we help teams choose the right benchmarks, determine what their existing data can credibly support, and translate the results into a claim that holds up under scrutiny.

These analyses will not resolve every limitation in the study design. But they may show that the evidence is stronger than the topline number suggests, or identify the specific question the next study needs to answer.

The Result Is the Floor, Not the Conclusion

The two-point GAD-7 improvement is still a real result. The question is how much that result allows the company to claim.

Treating the topline number as the floor means using it as a starting point for deeper analysis, not as proof that the product works. Before presenting any outcome, decide exactly what the evidence supports today, what remains unanswered, and what you need to measure or test next.

A result that cannot yet support the full claim has not failed. It has shown you where to build next.

If you are ready to move from a statistically significant result to an outcomes claim that can stand up to the next question, let’s talk.

Let’s Talk!

FAQ

Is a statistically significant symptom-score improvement enough to prove a digital health product works?

No. Statistical significance suggests that the observed change is unlikely to be explained by chance under the study’s assumptions. It does not establish that the product caused the change, that the improvement was large enough to matter to users, that it will last, or that similar results would be seen in a broader population.

What is the difference between statistical significance and clinical significance?

Statistical significance helps assess whether an observed difference could reasonably be explained by chance. Clinical significance asks whether the change was large enough to make a meaningful difference in someone’s symptoms, functioning, or daily life. With a large enough sample, a very small change can be statistically significant without being clinically meaningful.

What is the minimal clinically important difference, or MCID?

MCID is a benchmark used to help determine whether the size of a change is likely to be meaningful to patients or relevant to their care, rather than simply statistically detectable. The appropriate benchmark can vary by measure, population, baseline severity, and context, so it should not be treated as a universal cutoff.

How long should outcomes be tracked to call an effect durable?

There is no single follow-up period that makes an effect durable. The right timeframe depends on the outcome, intervention, and claim being made. The goal is to measure long enough to show whether the improvement persists beyond the immediate intervention period and across a timeframe that matters to the stakeholder evaluating the evidence.

Do outcomes need a comparison or control group to be credible?

Not always. Early-stage evidence can still be useful without a control group, particularly when evaluating feasibility, engagement, or early outcome signals. But if you want to claim that your product caused the improvement, a credible comparison becomes much more important. That comparison might be a control group, waitlist, standard-of-care group, or another appropriate benchmark.

What should a digital health company do after finding a statistically significant result?

Start by asking what the existing data can show beyond the average. Look at how large the change was, how many users improved enough to notice a difference, whether results varied across different users, and whether the improvement lasted. Then identify what remains unanswered and whether it can be addressed through additional analysis or if it requires a new study.