The announcement, which was written by Starbucks’ founder Howard Schultz and also came with statements about the American Dream “being called into question,” sparked calls for a Starbucks boycott, and #BoycottStarbucks became a trending topic on Twitter. At the same time, Starbucks also found itself receiving calls of support for standing up for its beliefs.

So what was the cost or benefit of its announcement? As it turns out, answering that question with any level of confidence is really, really difficult.

According to the YouGov BrandIndex, which “tracks public perception of thousands of brands across the world every day,” Starbucks’ reputation took a big hit following its announcement. 

Prior to the announcement, YouGov’s data indicated that 30% of consumers would consider making their next coffee purchase at Starbucks. After the announcement, that dropped to 24%.

Additionally, Starbucks’ YouGov BrandIndex Buzz score dropped from 12 to 4. The Buzz score measures how consumers respond to the question, “If you’ve heard anything about the brand in the last two weeks, through advertising, news or word of mouth, was it positive or negative?” and can range from 100 to -100. 

According to YouGov BrandIndex CEO Ted Marzilli, “Consumer perception dropped almost immediately. That would indicate the announcement has had a negative impact on Starbucks, and might indicate a negative impact on sales in the near term.”

Most interestingly, Marzilli noted that while Starbucks has seen its Buzz score drop before in connection with initiatives that have political overtones, this time Starbucks’ announcement negatively affected its purchase consideration numbers, signaling that perhaps the calls for a boycott were indeed resonating with some Starbucks customers.

Starbucks says “wait a minute”

Not surprisingly, Starbucks felt compelled to respond to headlines suggesting that its brand was hurt by its announcement. On March 10, the company issued a press release detailing how Kantar Millward Brown, a market research firm, found no evidence that the announcement had any negative impact whatsoever.

The press release quoted a letter from Brian James, Kantar Millward Brown’s Brand and Communications practice president, which stated: “Following the recent release of results from a YouGov Brand Index Survey, several news organizations have reported that Starbucks is suffering from consumer backlash related to its announcement to hire 10,000 refugees globally over the next five years.

“Such backlash or declines are not substantiated in our own measurement of Starbucks Brand Health and Consumer Sentiment.  Kantar Millward Brown has conducted on-going monthly measurement of Starbucks Brand Perceptions and Consumer Sentiment toward the Brand and saw no such impact in February 2017.  

“In fact, in February 2017 — after the announcement — we did not observe any substantive impact on Customer Consideration, Future Visitation Intent or Brand Perceptions or any other key performance metrics for the Starbucks brand.

So who should we believe?

It’s reasonable to be somewhat skeptical about Kantar Millward Brown’s findings. A cynic would suggest that because Kantar Millward Brown works for Starbucks, it is biased. But even if that’s not a totally legitimate or fair argument, it’s interesting to note that no actual figures for the firm’s “Starbucks Brand Health and Consumer Sentiment” were released.

While Starbucks would likely cite the confidential nature of such figures, if the company is going to issue a press release promoting the fact that a market research firm it hired “did not observe any substantive impact” following an announcement that was heard around the world, refuting an independent source that did provide actual numbers, it’s not unfair to suggest that Starbucks should have provided more data and details of its methodology.

At the same time, it’s worth considering the possibility that YouGov’s data might not paint the most accurate picture of sentiment either.

While the firm says that BrandIndex data is “nationally representative of adults in each country,” its data is gathered by “interview[ing] thousands of people from its panel of 2.5m people worldwide online.” For comparison, Starbucks serves more than 60m customers per week, so tracking sentiment accurately for the brand based on daily polls of just thousands of consumers seems fraught with challenges.

At the end of the day, Starbucks offers an interesting case study that demonstrates just how difficult it is to, with any confidence, take the temperature of consumers.

This doesn’t mean that sentiment analysis isn’t worthwhile, but as more and more businesses are implored to invest in it, particularly as part of their social media investments, companies should remember that ultimately, there’s no substitute for keeping a close eye on basic KPIs like revenue and store traffic.