The first challenge is to understand why some subject lines are good, and why some are bad.

Subject lines are important. This isn’t news. But how can you quantify the ‘goodness’ and ‘badness’ of a subject line?

Consider these two subject lines Econsultancy sent out earlier in 2015:

  1. Marketers, vent your frustrations here!
  2. Marketing Pain Points – share your stresses, get free report

These two subject lines invited people to fill out a survey about marketing pain points.

Before we look at results, here’s some similarities and differences between the two subject lines:

  1. Version A is shorter than Version B.
  2. Version A uses an exclamation point, while Version B doesn’t have any ending punctuation.
  3. Version A begins with a collective noun, while Version B begins with a proper noun.
  4. Version A uses the word ‘frustrations’, while Version B uses ‘stresses’.
  5. Version A has a comma to separate fragments, while Version B uses a hyphen.
  6. Version B offers a tangible incentive, while Version A doesn’t.
  7. Version B uses the word ‘free’, while Version A doesn’t.
  8. Version B uses jargon (‘pain points’), while Version A is more colloquial.
  9. Both use the imperative voice.
  10. Both use second person pronouns.
  11. ….. and so on (I could go on – there are dozens of differences and similarities between the subject lines!)

Which subject line won – Version A or B?

Version B won with a 7% open rate lift (at 99% confidence). Whoo hoo!

So now the important question: why did it win, and what can Econsultancy learn from this?

As mentioned above, there’s a litany of ways that Versions A and B differ. Dozens, or maybe even hundreds.

Here are a few plausible hypotheses that this split test could answer:

  1. Longer subject lines are better than shorter.
  2. Using the word ‘free’ is good.
  3. Exclamation points are bad.

And so on. The list could go on ad infinitum.

This is just two subject lines! What happens when you look at dozens, hundreds or thousands of them?

How can you tell which are the best… and understand why they’re the best so you can replicate the success?

What about these 10 subject lines?

These subject lines are the ones that experienced the biggest variation from the mean.

Subject Line StDevs from Mean
Upcoming training course I thought you’d be interested in 5.65
Where will marketers prioritise spending in 2015? -2.59
eCRM and analytics – join a three-day masterclass 1.79
Training: content and SEO working together -1.45
*drum roll* And the top 100 UK digital agencies are…. 1.34
Adopting a lean methodology: training in agile marketing 1.22
Understanding the Customer Journey – get your free report -1.20
What are marketers’ priorities in the year ahead? -1.12
Understanding the Customer Journey – new report 0.91
Intensive training in developing a positive customer experience -0.78

(Note: we normalised the results to determine the distance from the mean. This is to quantify variance that exists while controlling for other factors. A positive number means it was good, and a negative number means it was bad. The larger the number, the greater magnitude of its goodness or badness.)

Remember the differences we found between Versions A and B above? Let’s say we can find 10 differences between any two subject lines in the set of 10 above.

This means that A is different from B in 10 ways, from C in 10 ways, and D in 10 ways, plus B is different from C in 10 ways, D in 10 ways… and so on.

Suddenly you have a combinatorial explosion of millions of hypotheses, all of which could be correct. Discovering which is ‘the one’ is like finding the proverbial needle in the haystack.

So, what’s an email marketer to do?

Here’s how you can quantify why some subject lines work and others don’t

You read a subject line as a chunk of text and emotionally respond to it. That’s how language works. You ingest an entire sentence, have a subconscious response, and act upon your impulse. Email subject lines work when they pique particular emotions in recipients.

But how do you quantify the emotions that resonate with your audience?

There are a couple of challenges here. First of all, hand-quantifying emotion is wrought with cognitive biases. And second, it’s a time-consuming and cumbersome task.

We ran Econsultancy’s data through our email subject line sentiment analysis tool – Phrasee Pheelings. It’s the world’s first technology that measures the emotional power of any of your previous subject lines.

Here’s the bad news for Econsultancy

Econsultancy tends towards two distinct emotional clusters in their subject lines, loosely described as:

  1. Direct and devoid of personality (i.e. ‘The Consumer Conversation – new report’) – in blue below
  2. Mid-level familiar and curious (i.e. ‘How healthy is your content strategy?’) – in pink below

Phrasee Pheelings 2 cluster

The response variance of the two clusters was noticeable, though small. Why is this?

When marketers have to write loads of subject lines in their day jobs, they either

  1. follow the status quo; or
  2. follow gut instinct.

Both of these strategies run many risks, not least of which are:

  • If the status quo subject lines weren’t good in the first place, you’re just polishing a turd. You aren’t progressively optimising your language decisions.
  • If unquantifiable assumptions (like the Versions A and B comparison above) drive your gut instinct, you are making uninformed decisions.

People tend to do the same things over and over, because “if it ain’t broke, don’t fix it.” The results don’t get worse, but also they don’t sustainably improve.

This is a missed opportunity.

Here’s the good news for Econsultancy

Econsultancy’s two main emotional clusters don’t vary much, and the results don’t either. But this is an opportunity in itself!

There are limitless possible emotional combinations of language available. Knowing which work best for your audience is impossible without testing.

The key is to get split testing. But don’t test pointless things like changing a single word. That’s a tiny learning point that won’t tell you much about your audience. It will just make you buy a thesaurus.

Instead, test out big things. Like the overall emotional power of your subject lines.

And do it in a methodological, robust, quantifiable manner.

So, Econsultancy, here’s our advice to you:

When writing your subject lines, test the emotional impact of your subject lines.

You’ve tested out two emotional clusters ad infinitum, and the results aren’t bad. But what about the clusters you haven’t tried yet?

For example:

  • What about high urgency/’FOMO’ sentiments? There’s few like this in the sample set… but what if they’re key drivers for your audience? In orange below.
  • How about more familiar tones? Econsultancy has a friendly brand voice, yet that doesn’t translate in your subject lines. In purple below.
  • And, what about high levels of curiosity and ambiguity? In green below.
  • And so on, and so on… there’s limitless combinations to consider.

Phrasee Pheelings Untested clusters

To complicate matters – none of these emotions exist in isolation. You can have something that’s both urgent and familiar, for example. Or something that’s not that urgent, but has middling values of both familiarity and curiosity.

The challenges are:

  • to quantify what emotions to test;
  • to generate subject lines that adhere to these emotional clusters; and
  • to measure on an ongoing basis response, while controlling for external factors.

When testing out emotional clusters, some will work, and some won’t.

Your audience responds to the emotional power of your subject lines. Until you figure out what combination of emotions work the best, you’re using suboptimal subject lines.

The take-home point is this:

Small language sets are suboptimal if you don’t quantify their success or failure. You don’t know if what you’re doing is good… or if you’re missing an opportunity.

Big thanks go to…

What I like about the good people at Econsultancy is they’re always up for trying new things… and sharing what they learn with their audience. They love innovation and that’s why they’re such a well-respected group.

So thanks a lot to the Econsultancy team for sharing their data with us here at Phrasee, and for being good sports through the process.

And it’s not just us who think this is important…

When writing this blog post I received the following email from Econsultancy’s Festival of Marketing. Check out the subject line:

Emotional power is what makes marketing awesome.

We all know that emotion drives response. That’s not the opportunity.

The opportunity is to quantify emotion… then optimise based on the results… and then profit from better subject lines.