The Fundamental Theorem of Favorability Analysis

By Chris Scully, VP of Research at CARMA International

In 1987, professional poker player David Sklansky published The Theory of Poker outlining his thoughts on the underlying theories and concepts for winning at all the variations of the card game. In this book, he unveiled the Fundamental Theorem of Poker.

Photo credit: Viri GSimply put, the theorem states that anytime you're playing poker and your opponents do something (such as bet, call, raise, or fold) that they wouldn't do if they knew all your cards, then you win money. Also, anytime you do something (again, such as bet, call, raise, or fold) that you wouldn't do if you knew all your opponents' cards, then you lose money.

Using this as an inspiration, I'd like to offer what I call the Fundamental Theorem of Favorability Analysis: Anytime a story says something about a company or organization that the entity would want the story to say, then that discussion is favorable. Anytime a story says something about a company or organization that the entity would not want the story to say, then that discussion is unfavorable.

For this theorem, I define "about the company or organization" broadly such that it includes discussion of that entity's products and services, organizational mission or goals, management, financial performance, standing as an employer or corporate citizen, etc.  I also use the word "story" broadly to incorporate new reports, opinion pieces, and all types of social media hits (blogs, tweets, Facebook status updates, etc.).  Lastly, I define "that discussion" as being the part of the story saying that certain something the entity would or would not like the story to say.  This discussion could be as brief as a word or two or as expansive as several paragraphs or more. 

Incorporating this theorem into a favorability assessment methodology is relatively easy to do when using human analysts. I think many people – even those outside the PR industry – already understand this concept intuitively, and it's easy to formalize it by establishing guidelines that enable coders to recognize instances when the theorem should be applied. 

In contrast, I think it's a pretty difficult task for automated offerings to incorporate the Fundamental Theorem of Favorability Analysis into their sentiment algorithms. Foremost, software doesn't have any intuition, which means that every specific sentiment rule that a software offering follows must be programmed. Also, the ways in which a story can convey information that invokes the Fundamental Theorem of Favorability Analysis is limitless, and thus, no programmers could ever devise software that accounts for all possible favorable and unfavorable discussions. 

I believe the practical impossibility of incorporating the Fundamental Theorem of Favorability Analysis into media analysis software is a main cause of The Neutral Problem that is so prevalent in automated offerings.  However, since many programmers of media analysis software don't come from a PR background, it's possible that some don't quite grasp how truly vital this theorem is to assessing media coverage accurately, and thus, they don't devote enough of their efforts to accounting for the theorem in their programming.

Regardless of the causes, I think it's clear that until automated offerings can incorporate the Fundamental Theorem of Favorability Analysis into their algorithms, human-based media analysis is always going to produce more accurate favorability assessments.

The Neutral Problem

By Chris Scully, VP of Research at CARMA International

One of the big flaws I see when examining sentiment results derived from automated media analysis services is the overabundance of stories classified as being neutral. In my 11 years at CARMA, I've looked at results from a number of automated media analysis offerings (even ones that use natural language processing), and I've seen them classify 60 percent, 70 percent, or 80 percent of stories as being neutral. 

Such a high frequency of neutral attention just doesn't comport with what I see every day when I examine my clients' traditional and social media coverage. In fact, when analysts use CARMA's favorability rating methodology, it is the rare entity that sees even 40 percent of its coverage being neutral. For instance, across all of CARMA's work in the United States during 2011, we determined that 26.1 percent of stories were neutral, compared to 56.3 percent being favorable and 17.6 percent being unfavorable.  We saw a similar breakdown in 2010 for the U.S. media coverage we analyzed, when 25.5 percent of stories were neutral, 51.8 percent were favorable, and 22.7 percent were unfavorable. 

So when automated offerings return results that say that more than half of all stories were neutral, count me as a skeptic. 

A major factor in this overabundance of neutral stories in automated media analysis offerings is how the news media discusses features and benefits of products and services. In news reports on products and services (as opposed to opinion pieces, such as product reviews, op-eds, and editorials), journalists typically use neutral language in describing the offering's features and benefits, and they frame this discussion as factual information about the product or service itself. Such discussion is not designed to be a commentary on the quality of the product or service, but rather simply to report its functionalities.

Please take a quick look at this recent Chicago Tribune news story that served as a primer on how to use Pinterest, the latest hot thing in social media, to demonstrate this concept. Don't worry, it's short (less than 400 words).  

The news article began by outlining the site's purpose in neutral language, saying, "Pinterest is a virtual pin board that allows you to organize and share the things that inspire you." It then stated without tonality that Pinterest could be viewed by non-members, but that people had to join the site if they want to create their own material. In even-handed language, it continued by giving directions on how to create new pin boards based on one's interests and hobbies, describing how to post new material, and explaining the site's culture and etiquette. It ended by suggesting that users should be on watch for classes on how to use the site, including ones geared to helping small businesses use the site to market to a wider audience. 

In essence, none of this discussion contained words or phrases that I believe to be overt praise of Pinterest, nor did it contain verbiage that I consider to be a consequential criticism of the site.  I view it as a straightforward reporting of the facts, with little to no sentiment expressed explicitly. 

I did, however, notice some words or phrases that an automated offering might classify as being positive or negative for Pinterest.  Here's a table outlining what I saw for each category and where in the story that language appeared:

Positive Words/Phrases (Location in Story)                  Negative Words/Phrases (Location in Story)

1.  Inspire (first paragraph)                                                1.  Sarcastic (fifth paragraph)

2.  Have Fun (fourth paragraph)                                         2.  Still Needs Work (sixth paragraph)

3.  Spread Virally (fifth paragraph)                                      3.  Be Patient (sixth paragraph)

4.  Sweet (fifth paragraph)                                                 4.  Addictive (eighth paragraph)

5.  Be Nice (seventh paragraph)                               

6.  Superstar (ninth paragraph)

While these words or phrases can be blatantly positive or negative, my assessment is that, in most cases, they were directed at users of Pinterest and not the site itself. I consider all of the positive words and phrases to be aimed at users, with three of the four negative words and phrases ("still needs work," "be patient," and "addictive") being directed at Pinterest itself. So, if an automated system analyzed a story according to these key words, only three of the ten words or phrases with overt sentiment would have been applied accurately to the assessment of how favorably Pinterest was presented in this article. 

Without any explicit praise of the site, and given the near equal amount of language that software might classify as being positive and as being negative, my experience suggests that an automated media analysis offering would categorize this article as being neutral or as being slightly positive. (I invite any purveyor of an automated offering to analyze this story with your software and report back to me your sentiment result and how it was derived.  I will post each response to the CARMA.com blog.) 

However, if we imagine ourselves as being on Pinterest's PR team, we all would agree that this Tribune piece is a home run. Despite having not a single word of direct, explicit praise of the site, this report is just fantastic for Pinterest, as it's a veritable checklist of why someone should use the site. I bet that Pinterest's PR folks did back flips in joy after reading this, and I'd also be willing to bet a lot of money that Pinterest's web traffic from the Chicago-area surged after this article appeared. 

In contrast to how automated offerings likely would analyze the article, CARMA's favorability score for this article aligns with the reaction of the Pinterest PR team. We classify this article as being highly favorable and give it a 75 rating in our favorability rating system. CARMA gave it such a strong score because we recognize that, despite the neutral language, the article repeatedly highlights the key benefits of Pinterest. And, even though there isn't one instance of explicit praise, CARMA recognizes that the site is receiving considerable implicit praise. 

I believe this mini-case study demonstrates anew the continuing need for human involvement in media analysis.  Like CARMA's CEO Albert Barr wrote last week, there is a place for automated media analysis offerings, but we need to make sure that at least a sample of an organization's coverage is assessed by analysts so that all the nuances of communication and the context in which stories appear are taken into account.