Is Your Marketing Data Statistically Significant? Here’s How You Can Tell

When you watch election results on television, at the bottom of the screen, you may notice a percentage called “margin of error.” It’s typically around 3% in the polls.

What does this mean exactly? This number indicates that the reported results could be over or under by about 3%. It’s also why someone winning by one, two or three percentage points may not really be “winning” at all.

Similarly, when data scientists measure response rates for direct mail, they know whether or not the margin of error will deem the results inconclusive. In some cases, it’s not necessarily that the result isn’t true—it’s just that it may be “too close to call.” 

What do you do with results that aren’t statistically significant? Simply use them as directional information. But note that if the test is conducted again, a different result may occur.

There are two primary causes of statistically insignificant results. They include:

• Not having enough quantity in the test

• The results of each test are really close together—meaning they are not different enough to measure

In this case study, a company wanted to test Hallmark cards against its other direct mail formats to see if it would make an impact on marketing results. The company was large enough to have enough quantity to get statistically significant results. And significant results they saw! For an even deeper look into this example, read our case study that explains how an online retailer improved its ROAS by 50% by using Hallmark cards. Get the case study here.