Test and Analysis Design

Introduction to Test and Analysis Design

Test and analysis design is the way by which you are able to ensure that you will learn with each campaign and continuously improve.  Prospecting campaigns that don’t include build testing and analysis the start have no mechanism to continually improving results and are destined to be sub-optimal. We can help ensure that winning approaches that is often hidden in the response data will be systematically identified in order to maximize results and marketing ROI.

Gaining a deep understanding of your Prospect Target Audience is a necessary first step, whether you are planning to use a single media or Multi-channel Acquisition Campaigns.

Variables to Test

Each campaign, regardless of type, has many variables that need to be tested, either individually or in combination.  That’s where creating a Test and Analysis Plan that fits your budget comes into play.

Typically, the most important element to test are:

• Targeting (using Demographic and Behavior Data, Segmentation Models or Pre-built Predictive Models)
• Offer and/or creative
• Media combinations (plus ad sizes and formats)
• Subject lines (email only)

If time and budget permits, and to further improve results, also test:

• The impact of personalization (for direct mail and email)
• Lead capture mechanism (Landing Pages and Nurturing)

Test Cells vs. Individual Records

The two “levels” at which test can be conducted are the test cell, which is an aggregation of a significant number of targeted individuals (typically at least 10,000 or 20,000 but we can calculate how many are needed for statistical validity), and the individual level:

• Pre-set Test Cells – works for any media for both intermediate metrics (such as opens or clicks) and conversion metrics (such as form fills or purchases).
• Individual Post-Analysis – works for all media for conversion metrics and also for intermediate metrics for email (opens and clicks, which is one reason why prospecting email is a great test media).

Here are some simple test cell designs:

2 x 2 Test Cell Design with 100,000 total recipients

3 x 3 Test Cell Design with 180,000 total recipients

The wining test cell is the one with the highest open, click, form fill, purchase or other conversion event.  Teach which individual is associated with each test cell can be accomplished by either pulling actual names prior to deployment and assigning to each test cell, or by setting up response tracking mechanisms dedicated to each test cell.

Analyzing Results at the Individual Record Level

For direct mail and email it is possible to track which test cell a person is in because we know their personally identifiable information (i.e., name and physical or email address).  This is not possible with digital, mobile or social campaigns where the target audience is pulled “natively” (e.g., a target audience is created from within Facebook or a Demand Side Platform) since, for privacy reasons, we do not know the names of each individual prior to deployment.  However, it may be possible to first pull name and address/email information first then “onboard” this data (i.e., upload it into DSP) to generate the known target audience associated with each test cell.

Individual Post-Analysis allows deeper Campaign Predictive Analytics and can only be performed if the list of individuals is known ahead of time.  This involves performing logistic regression on the target audience and the responders, then scoring the remaining non-responders to determine which are most likely to respond to another campaign.    This is especially useful for relatively expensive media such as direct mail.

If dedicated lead capture mechanisms have not been set up for the campaign (overall or by test cell) then it will be necessary to perform a Match Back Analysis to determine which recent customers were a result of the campaign.   This can be performed by providing you with the list to do the matching but there are key advantages to providing us with the conversion data to perform the matching:

• Use of advanced Customer Identification technology such as AbiliTec Consumer or Household Links will increase the match rate significantly.
• Email campaign match backs automatically include both individual and household level matches.

Of course, programmatic buying is also able to optimize campaigns over time without needing to know the individuals ahead of time.  In this case you would be optimizing within the one media (e.g., digital ads)

Let us know what types of testing you would like to perform and we will help you create your Test and Analysis Plan.