From target group insights to testable messages

Your target group profile suggests that people want certainty. What should you put on your website? ‘A reliable partner’ is one possible message, but not yet a testable proposal. You need to define what certainty means in this situation, which information you will change and which outcome will indicate whether that change helps.

Here, a testable message is a specific communication variant for which you define in advance the audience, comparison variant and outcome on which you expect a difference. This is our working definition for translating audience insights into research.

The process is to describe the insight, formulate an explanation as a hypothesis, create variants, choose an appropriate outcome and conduct the comparison. At every step, distinguish what has already been investigated from what you still expect.

Matz and colleagues investigated messages matched to personality characteristics in digital campaigns. Their studies illustrate how alignment between message and audience can be examined empirically. The publication does not investigate a Neurofactor audience classification. Matz et al., 2017.

We use ‘testable message’ as a working term for a concrete research question. We do not attribute the origin of this term to that publication. The relevant point for your brief is that a plausible match between an insight and a text still needs testing in your application.

In this knowledge base, an associative target group is a group of people who decide on the basis of similar associations within a bounded choice context and can be delineated on that basis. This is a working definition. Establish how you identify the group before examining message outcomes. Calling people ‘certainty seekers’ after they click a certainty-focused message does not independently test that profile.

Start with one claim from the target group profile. Record:

• Original evidence, such as interviews, association measures or choice research.
• People, situation and period studied.
• Meaning of the concept in that context.
• Counterexamples, missing information and uncertainties.
• Decision for which you intend to use the insight.

‘This audience wants certainty’ might become: ‘In the interviews studied, participants asked who was responsible for implementation before requesting a quote.’ That is a bounded observation. ‘Showing the allocation of responsibilities increases enquiries’ is the next prediction, which still needs testing.

Here, a communication hypothesis is a prediction stated in advance about the difference between communication variants for a particular audience and outcome. Use this sentence template:

> For [defined audience] in [situation], we expect [variant B], compared with [variant A], to produce [outcome], because we hypothesise that [explanation].

Keep the predicted outcome separate from the explanation. An increase in enquiries might be consistent with reduced uncertainty, but does not establish that mechanism by itself. To investigate the explanation too, specify the additional evidence needed. A single follow-up question does not automatically establish a causal mechanism either.

Make clear what A and B do differently. You might change the emphasis on a benefit, explain a process or add evidence. Where possible, hold other components constant: offer, price, visual presentation, CTA and technical implementation.

If you change several components together, state that you are comparing two complete packages. Such a test can help choose which version to use. Without additional design features, it does not identify which individual element caused the difference.

If you want to distinguish the effects of two elements and their combination, discuss an appropriate factorial design and its sample requirements. ‘Always change only one word’ is too restrictive; the question is which contrast must support your conclusion.

Cronbach and Meehl addressed construct validity in a classic methodological publication: interpreting a measurement requires supporting evidence. This also matters when interpreting a communication metric as, for example, trust. Cronbach & Meehl, 1955.

The following is an editorial planning aid for your test brief.

Choose one primary outcome for the main question and record supplementary measures separately. Also define limits a variant must not exceed, such as more unsuitable enquiries or inaccurate expectations. Specify the numerator, denominator and measurement period for each metric.

Kohavi and colleagues discuss randomised online experiments as an approach to studying the effects of changes on user behaviour. Their methodological guide also addresses practical and statistical requirements for these comparisons. Kohavi et al., 2009.

For your test, specify assignment and the unit being compared: person, account or organisation, for example. Where required, keep a person assigned to the same variant. Document the handling of repeat visits, attrition and missing data.

Comparing one campaign in April with another in May requires additional care: audience, timing and circumstances may also differ. On advertising platforms, check whether the selected feature actually supports an appropriate experimental allocation. Running two separate advertisements does not guarantee randomisation.

Lakens describes different ways to justify sample size, including power analysis and desired precision. The choice depends on what the study needs to establish about relevant effects. Lakens, 2022.

Show which difference matters for your decision and what can be investigated with the available reach. No participant count is sufficient for every message test. Limited traffic may justify examining comprehension or interpretation first and conducting a quantitative effect test later.

Simmons and colleagues show that undisclosed flexibility in data collection, analysis and reporting can increase the probability of false-positive findings. Simmons et al., 2011.

Before analysing effects, record the main comparison, exclusion rules and stopping rule. Choose a fixed design or an appropriate sequential analysis method. Do not stop at your own discretion when an interim result looks favourable. Label unplanned analyses as exploratory and account for multiple variants or subgroups in the analysis plan.

A software provider suspects that unclear responsibilities make people hesitate to request a quote. The following example is fictional: the statements, proposition and test were invented and have not been tested.

The provider must be able to deliver the promise in variant B. This is a change in communication emphasis about the same service, not the addition of a service feature that does not exist.

The pretest can help identify unclear wording. Use the revised texts in the effect test and retain their versions. Someone who saw both messages in the pretest has had different exposure from a new visitor in a single-variant test.

Suppose B generates more enquiries. You still assess uncertainty, enquiry quality and test execution. The conclusion concerns the tested variant, audience and situation. ‘This works for all certainty seekers’ requires further research.

Use a decision process agreed in advance:

• Sufficient support for a relevant benefit: consider use within the studied context and monitor the agreed quality measures.
• Insufficient differentiation: report uncertainty. An unclear result does not establish equivalence.
• An adverse effect or inaccurate expectations: revise the message or retain the other variant.
• An unexpected difference between groups: treat it as follow-up research when that comparison was not planned.

A variant producing a ‘significant’ result in group X but not in group Y does not, by itself, establish a difference between those effects. Ask for a direct assessment of the difference in effects. Also distinguish the effect of the assigned message from the proposed psychological explanation.

Calling a click increase ‘more trust’; reporting only the most favourable metric; changing audience boundaries after seeing results; comparing two entire campaigns and attributing the result to one word; interpreting an unclear outcome as ‘no effect’; testing a promise the offer cannot fulfil.

Keep one test dossier containing the insight, hypothesis, versions, audience rule, measurement plan, analysis agreements and decision. Record what the test still leaves unknown.

A target group profile helps choose what to investigate. A testable message makes that choice concrete: for whom are you changing which information, compared with what, and with which expected result? An appropriate comparison and predefined decision process turn a text variant into an informative test.