The Neutral Problem

By Chris Scully, VP of Research at CARMA International

One of the big flaws I see when examining sentiment results derived from automated media analysis services is the overabundance of stories classified as being neutral. In my 11 years at CARMA, I've looked at results from a number of automated media analysis offerings (even ones that use natural language processing), and I've seen them classify 60 percent, 70 percent, or 80 percent of stories as being neutral. 

Such a high frequency of neutral attention just doesn't comport with what I see every day when I examine my clients' traditional and social media coverage. In fact, when analysts use CARMA's favorability rating methodology, it is the rare entity that sees even 40 percent of its coverage being neutral. For instance, across all of CARMA's work in the United States during 2011, we determined that 26.1 percent of stories were neutral, compared to 56.3 percent being favorable and 17.6 percent being unfavorable.  We saw a similar breakdown in 2010 for the U.S. media coverage we analyzed, when 25.5 percent of stories were neutral, 51.8 percent were favorable, and 22.7 percent were unfavorable. 

So when automated offerings return results that say that more than half of all stories were neutral, count me as a skeptic. 

A major factor in this overabundance of neutral stories in automated media analysis offerings is how the news media discusses features and benefits of products and services. In news reports on products and services (as opposed to opinion pieces, such as product reviews, op-eds, and editorials), journalists typically use neutral language in describing the offering's features and benefits, and they frame this discussion as factual information about the product or service itself. Such discussion is not designed to be a commentary on the quality of the product or service, but rather simply to report its functionalities.

Please take a quick look at this recent Chicago Tribune news story that served as a primer on how to use Pinterest, the latest hot thing in social media, to demonstrate this concept. Don't worry, it's short (less than 400 words).  

The news article began by outlining the site's purpose in neutral language, saying, "Pinterest is a virtual pin board that allows you to organize and share the things that inspire you." It then stated without tonality that Pinterest could be viewed by non-members, but that people had to join the site if they want to create their own material. In even-handed language, it continued by giving directions on how to create new pin boards based on one's interests and hobbies, describing how to post new material, and explaining the site's culture and etiquette. It ended by suggesting that users should be on watch for classes on how to use the site, including ones geared to helping small businesses use the site to market to a wider audience. 

In essence, none of this discussion contained words or phrases that I believe to be overt praise of Pinterest, nor did it contain verbiage that I consider to be a consequential criticism of the site.  I view it as a straightforward reporting of the facts, with little to no sentiment expressed explicitly. 

I did, however, notice some words or phrases that an automated offering might classify as being positive or negative for Pinterest.  Here's a table outlining what I saw for each category and where in the story that language appeared:

Positive Words/Phrases (Location in Story)                  Negative Words/Phrases (Location in Story)

1.  Inspire (first paragraph)                                                1.  Sarcastic (fifth paragraph)

2.  Have Fun (fourth paragraph)                                         2.  Still Needs Work (sixth paragraph)

3.  Spread Virally (fifth paragraph)                                      3.  Be Patient (sixth paragraph)

4.  Sweet (fifth paragraph)                                                 4.  Addictive (eighth paragraph)

5.  Be Nice (seventh paragraph)                               

6.  Superstar (ninth paragraph)

While these words or phrases can be blatantly positive or negative, my assessment is that, in most cases, they were directed at users of Pinterest and not the site itself. I consider all of the positive words and phrases to be aimed at users, with three of the four negative words and phrases ("still needs work," "be patient," and "addictive") being directed at Pinterest itself. So, if an automated system analyzed a story according to these key words, only three of the ten words or phrases with overt sentiment would have been applied accurately to the assessment of how favorably Pinterest was presented in this article. 

Without any explicit praise of the site, and given the near equal amount of language that software might classify as being positive and as being negative, my experience suggests that an automated media analysis offering would categorize this article as being neutral or as being slightly positive. (I invite any purveyor of an automated offering to analyze this story with your software and report back to me your sentiment result and how it was derived.  I will post each response to the CARMA.com blog.) 

However, if we imagine ourselves as being on Pinterest's PR team, we all would agree that this Tribune piece is a home run. Despite having not a single word of direct, explicit praise of the site, this report is just fantastic for Pinterest, as it's a veritable checklist of why someone should use the site. I bet that Pinterest's PR folks did back flips in joy after reading this, and I'd also be willing to bet a lot of money that Pinterest's web traffic from the Chicago-area surged after this article appeared. 

In contrast to how automated offerings likely would analyze the article, CARMA's favorability score for this article aligns with the reaction of the Pinterest PR team. We classify this article as being highly favorable and give it a 75 rating in our favorability rating system. CARMA gave it such a strong score because we recognize that, despite the neutral language, the article repeatedly highlights the key benefits of Pinterest. And, even though there isn't one instance of explicit praise, CARMA recognizes that the site is receiving considerable implicit praise. 

I believe this mini-case study demonstrates anew the continuing need for human involvement in media analysis.  Like CARMA's CEO Albert Barr wrote last week, there is a place for automated media analysis offerings, but we need to make sure that at least a sample of an organization's coverage is assessed by analysts so that all the nuances of communication and the context in which stories appear are taken into account.

The Continuing Need for Human-Based Media Analysis

By Albert J. Barr, Chairman & CEO, CARMA International, Inc.

Using computers to analyze media coverage is useful and amazing. Technology really has come a long way since I got into the media analysis business in 1984. 

Back then, the challenge was how to convert information captured from an actual hard copy of a newspaper article into data that we could enter into a computer database. To accomplish this, I designed a system where all of our analysts used typewriters with paper forms. The fields being researched were typed on 8 1/2 x 11 inch sheets of paper that were printed with orange ink. The data gleaned from each article was typed, of course, in black. There was enough room on one sheet of paper for three articles.

image by Mustafa Khayat

To save time, and automate the process, we fed thousands of these forms into a scanner. The scanner could read the typed data but it could not see the orange ink that had the field names and boxes (media name, favorability, issues, messages, etc.) I can remember clients visiting our offices in Washington and being wowed at this creative, state-of-the-art idea while they watched the scanner input information from thousands of researched articles, thus replacing human typists for data entry. 

We've advanced light years since then. In the early days, I don't think you could count the number of media analysis companies in the U.S. on one hand. Since the Internet, however, and tremendous advances in computer hardware and software, this has become a large and highly competitive business. 

At CARMA International, we believe in technology. Computers can handle huge amounts of information efficiently. There is no way that humans can keep up with that kind of pace. 

However, I believe that while computers are fast and relatively accurate, they still can't pick up sarcasm and all kinds of nuances that appear in media coverage. This is why I believe a strong need still exists for some kind of human intervention both in measuring and interpreting what all this coverage means to companies, governments, and all organizations who need to know what's been said about them. 

I believe, at least for now, there has to be some form of compromise between using computers to digest millions of bits of information and humans to help analyze and interpret their meaning.

A good way to do this is to use the same approach that survey research firms have been using ever since they started polling. If you are using an automated service to "analyze" thousands, or even hundreds of thousands of mentions in both traditional and social media, it still makes sense to get a good statistical sample from this base and have real people analyze, measure, and interpret the sample. This way you can get dynamic results along with professional advice about what's being said, emerging trends, and a much more accurate measure of media sentiment. 

George Fueschel, an IBM technician and instructor in New York, coined the term "GIGO," garbage in, garbage out. Wikipedia says the term "is used primarily to call attention to the fact that computers will unquestionably process the most nonsensical of input data ('garbage in') and produce nonsensical output ('garbage out'). It was most popular in the early days of computing, but applies even more today, when powerful computers can spew out mountains of erroneous information in a short time."

Quality Control is our mantra at CARMA. I would strongly advise anyone using computers for media analysis to include a serious element of human intervention. It's insurance so organizations don't waste precious funds on information that may prove to be of little to no use.

The Human Element

By Elizabeth Ballard, Senior Analyst

Two things stood out to me as a measurement professional while reading a recent study on the radicalization of young men and the different factors that lead some to violence while others remain non-violent. The first was how the researchers utilized “grounded theory methodology,” a research process that involves the generation of theory from data (and not vice versa). The second was the importance of human interaction between the researchers and their subjects.

image by mazeo

The researchers first established a system of categories for the life histories, politics, activities, and religious beliefs of all the subjects. The researchers then interviewed each participant, assigning the various categories according to how the subjects responded to questions and engaged in conversation. This allowed the researchers to recognize patterns and develop theories about how environment, education, religious affiliation, community involvement, and family life influenced the radicalization of the men. Most importantly, the structured human-centric research was effective in gauging nuance, sarcasm, personality, and other unique traits and characteristics, which provided for a deeper understanding of the causes and motivations for the behavior of the subjects. The research led the academic team to gain critical insights into why some radical youth turn to violence and others are satisfied with non-violent protest.

This process mirrors what we do at CARMA in media measurement. We are given a massive load of information from across the media landscape, and all of this information must be analyzed, categorized, and transformed into data that can provide insight into trends, correlations, and opportunities for our clients.  Like the researchers in the radicalization and violence study, we must do this without being prescriptive; otherwise, we will be blind to patterns and outcomes that we have not anticipated. We should not set out looking for any specific answer, but instead allow general themes and associations to emerge from the data in order to reach and provide meaningful conclusions. 

Both the above study and our approach put a premium on human involvement, as human insight is critical to identifying aspects of behavior unrecognizable to a machine. While many are satisfied with automated analysis solutions, it is important to recognize that computers cannot distinguish sarcasm or metaphor, for example, and cannot split unforeseen hairs as the researchers do between violent and non-violent radicals. I doubt the study would have been as illuminating if the human component were absent from the process, as the subjects would have been statistics on a page without the nuanced perspective that comes from human engagement. The ability to identify these things is of great significance, particularly with the growing influence and dominance of social media where unfettered opinions run rampant.

Hot News: Blogging Tanks Among Inc. 500

By Angie Jeffrey, APR

Now here is a fascinating turn of events!  Would you believe that the number of the Inc. 500 companies maintaining corporate blogs has dropped from 50 percent in 2010 to 37 percent in 2011?  Reported by David Strom in ReadWriteWeb, this finding came from a five-year longitudinal study done by the University of Massachusetts at Dartmouth under the direction of Nora Barnes. The study also found, not surprisingly, that companies in the Advertising/Marketing industry are more likely to blog than those in Government Services and Construction.

In comparison, while the use of blogging remained pretty stable among the Fortune 500 from 2010-2011, that group always has had a much lower level of blog use than the Inc. 500 (which is the fastest-growing set of private American companies tracked by Inc. magazine). Barnes stated that, “The use of blogging may have peaked as a primary social media tool in the US business world.”

Additional research showed that three-fourths of the Inc. 500 companies are using social media tools such as LinkedIn and Facebook, and 90 percent of the respondents see them as important for improving brand awareness and company reputation, generating web traffic, generating leads, and supporting customers.

Chart is from The 2011 Inc. 500 Social Media Update: Blogging Declines As Newer Tools Rule, Page 5

I’m not in the least surprised.  For years, I have wondered how long companies would embrace corporate blogging, since it involves such heavy staff time commitments.  For a firm like CARMA, in a vertical industry niche, blogging is still worth the investment, especially since we’re doing it as a team.  But is it worth it for your company?  Take a look at the study and tell us what you think!

Thanks to AMEC and IPR

By Sonia LaFountain, Chief Operating OfficerPhoto by djg0333

I recently had the opportunity to attend meetings in New York with the North American Chapter of the Association for the Measurement and Evaluation of Communication (AMEC) and the Institute or Public Relations Commission on Measurement and Evaluation (IPR).  Thank you to both organizations for inviting me to sit in and learn about all the exciting work that is being done to educate and disseminate media measurement and evaluation best practices. The focus of this work  is to define, and in some cases to standardize, best practices that will surely benefit professionals around the globe. It is important for measurement to be flexible enough that it works for each individual company or organization, but universal in method so that it can be replicated and understood by all.

AMEC is a member organization focused on providing a forum for the sharing of information, knowledge, and best practice in the areas of measurement and evaluation of communication globally. AMEC provides a number of key business benefits to its members, who are bound by a Code of Practice to maintain the highest standards of professionalism. Visit AMEC's website For more information on them or how you or your organization can qualify to join. 

The IPR Commission is made up of researcher-practitioners and thought-leaders in public relations research, measurement and evaluation.  This is a diverse group drawing from corporations, public relations agencies, research firms, and academia.  Their core mission is to develop and promote standards and best practices for research, measurement and analytics that contribute to ethical, strategic and effective public relations. You’ll find a wealth of free white papers and other information on the IPR Commission’s website.