When in doubt, test. This is especially true with direct response products and services as your investment can be self-liquidating. Consumer research, whether primary, secondary or focus groups, is a sunk cost to be recovered by informing a smarter strategy and execution over time. Direct-to-consumer experimental design test costs are reduced by the in-market response. You not only get the benefit of learnings, you subsidize the research costs through collecting revenue from responders. Additionally, traditional research surfaces attitudes, beliefs and intent, while in-market testing captures actual behavior.
It’s important to first identify a set of hypotheses and then test to validate or refute the expectations. An example in the credit card industry may be testing a range of attributes’ lift (increase) on response and subsequent activation, usage, evolvement, credit quality and retention. In the direct marketing world, whether online or offline, an overlay of message, offer, creative, list (or target audience) and frequency testing can add exponential complexity and insights.
Any experimental test design can breakdown to single- or multi-factor testing. Single factor requires isolating each variable and testing against a control attribute. This can be illustrated by testing a new price point against an old price point. In addition to “test vs. control” testing, it’s also referred to as champion/challenger testing.
Multi-factor testing looks at how independent variables effect a dependent variable such as response. So, rather than isolate a single attribute, you strive to see the effects of several variables on response. These might include (in the example of credit cards again) duration of introductory interest rates, the go-to rate (the rate in effect following the intro teaser rate), product naming or other variables. Often, there may be enough quantity of response, to “back test,” which is to analyze responders on the backend and see how the previously collected data correlates with overall-response. The greater discrimination between responders and non-responders, the higher the predictability.
In any experimental design it’s important to ensure that the sample size is enough to generate a reasonable confidence interval that covers the population parameter with an acceptable standard deviation (the + or – range of error within the predictability or confidence interval). The larger the test population, the more predictable the outcome in a rollout of the test attributes. The approach would be to identify your test method, the test attributes, the reach (or list quantity for example, if direct mail) by test cell assuming a response rate of X%. For a credit card solicitation, it would require a certain size test population to read gross response and potentially a larger population to read net-response post a credit screen for example.
Keep in mind cause and effect. Each variable needs to be isolated to measure true impact on response or other metric that drives profitability. For creative alone, it may entail isolating messaging, specific copy points, color or nomenclature. It needs to be viewed as a continuous process of improvement and challenging your proposition in advance of the competition. Marketing is both art and science.
At Creative Solutions, we can help you push forward for breakthroughs, which are achievable and quantifiable, given establishing the right framework or, in this case, experimental design. Call us at 302.543.8533 or email us today.