Reflections on the IPR/PRSA North American Summit on PR Measurement

By Chris Scully, VP of Research of CARMA International

The most interesting question raised at last month's North American Summit on PR Measurement was "why don't more people in the PR industry try to demonstrate how their media outreach efforts affect tangible business or organizational outcomes?". In other words, in a world where many people in PR still use a widely discredited measure like ad equivalency (AVE), purportedly to demonstrate the return on an organization's investment (ROI) in media outreach, why don't more people utilize approaches that are widely espoused by media measurement experts and that actually demonstrate the ROI of PR efforts?  How come just about everyone (including us here at CARMA) on the measurement/analysis side of the PR industry advocates for connecting media performance to business or organizational results to show true ROI, but so few PR practitioners have adopted this practice?

For the unaware, two established approaches show how media outreach contributes to an organization's overall business outcomes. The best method for accomplishing this is through marketing mix modeling, whereby metrics from an organization's advertising, marketing, and media outreach campaigns are placed in complex models and compared against changes over time in that organization's business results, such as sales figures, brand awareness, or customer satisfaction data. Such an undertaking establishes the unique contribution of PR to these business results, as well as that of advertising and marketing. This is akin to the Holy Grail of PR measurement and evaluation, whereby the true return on investment in PR is demonstrated. However, it's an involved process, and, thus, quite expensive.

The other option for finding the ROI of media outreach is employing simple regression analysis.  This approach looks to see if a statistical correlation exists between changes over time in an organization's media performance and business results. When compared to marketing mix modeling, regression analysis is rather easy to employ (that is, once you have the media performance metrics and the business results). We at CARMA offer this service under the name of CARMA Connect.  Regression analysis has a limitation: it cannot establish causality. For those of you who have taken a statistics class, this should be familiar, as I'm sure your instructor intoned many times that "correlation doesn't necessarily mean causation". For the rest of you, what this means is that, even if a strong statistical correlation exists between how an organization performs in the media and how it fares in its business results, you cannot say that it's definitive proof that the media coverage caused the changes in the business results.

However, we at CARMA believe media performance must in some way affect business results or other organizational outcomes. After all, without giving it much thought, we each can think of hundreds of instances where a news story affected our purchasing decisions or altered how we thought about a company. As a result, CARMA believes that, if a strong correlation is established between media performance and business results, it's a good indication that the media performance helped cause the changes in the business results.

Now, let's get back to the matter at hand and explore why more PR practitioners don't use either marketing mix modeling or regression analysis. When this topic was broached during the Making Strategic and Effective Use of Output AND Outcome Measurement Platforms session at the IPR/PRSA conference, panelists and audience members alike offered their opinions. 

(I hope you'll forgive me, but I wasn't taking detailed notes about who said what, so I won't be able to attribute any of these theories to specific people, nor did I write down the verbatim comments. As a result, I'm going to try to capture the general sentiments expressed.)

Here are some thoughts that were put forward:

1.  People in PR hate/are frightened by math . . . That's why they're in PR and not engineering.
2. The necessary data required for such approaches is in silos, with the PR metrics hosted in the communications department and the other needed data in the hands of the marketing folks, or other executives, and never the twain should meet . . .
3. Lack of awareness or understanding about marketing mix modeling and/or regression analysis.

I agree with my colleagues at the conference who expressed these ideas, as they certainly play a significant role in why more PR practitioners don't link their media outputs to business outcomes. However, one key reason for the limited adoption of linking outputs to outcomes was omitted from the discussion. In my mind, the practice of linking outputs to outcomes isn't more widespread because not enough senior executives – CEOs, Company Presidents, Managing Directors, etc. – put their PR teams on the hook to improve business or organizational results. As a consequence, people in the PR industry don't need to engage in exercises like marketing mix modeling or regression analysis, as they don't need to show their senior executives the effect of PR outreach on overall business results.

I believe that almost everyone in business, government, and the non-profit worlds operates under the assumption that good media attention will lead to improved outcomes. Therefore, the thinking goes, if a new product gets a good review, then it will lead to more sales.  If a company's environmental initiatives receive positive media attention, then it will strengthen the company's image as being environmentally responsible.  So on and so forth. 

(Now, certainly some CEOs have asked their PR teams to test these assumptions, but I bet that many PR teams responded that it wasn't possible to do so, largely due to reasons 1-3 cited above.)

Moreover, as a resulting of this thinking, most senior executives will be satisfied with requiring their PR teams to demonstrate only that their efforts led to good media attention. Or, in a worst case scenario, senior executives will demand AVE metrics. Once the PR team demonstrates they've fostered good media attention or passed along the ad value equivalencies of the coverage, the CEO is happy and so too is everyone else.

The IPR/PRSA conference also coincided with the release of Moneyball, Brad Pitt's movie detailing how in the early 2000s, Oakland A's general manager Billy Beane used innovative thinking to better enable his poorly funded baseball team to compete against wealthier franchises.  The movie, of course, was based on Michael Lewis's seminal 2003 book of the same title. 

(Please know that I'm going to be simplifying things a little bit here over the next few paragraphs so as not to confuse those who are unfamiliar with baseball and its minutia.)

What innovation did Billy Beane employ in Moneyball (though, this is explained far better in the book than in the movie)? That's right, he and his staff used regression analysis to determine which outputs from individual players contributed the most to positive overall team outcomes. In particular, Beane and his colleagues identified which performance metrics for hitters (outputs) best correlated to their team scoring runs (outcomes) and which performance metrics for pitchers best correlated to their team preventing runs from being scored.

At that time, most Major League Baseball teams thought the most important skill for a hitter was the ability to get base hits. Through regression analysis, Beane and his staff saw that, while the ability to get base hits is a really important, what's even more important is that hitters be able to get on base without creating outs, such as by walking frequently in addition to getting base hits. 

(In fairness, many besides Billy Beane (Bill James being the most famous among them) understood at that time the relative importance of a player's ability to get on base over his ability to get base hits; it's just that most of these people, unlike Billy Beane, were not employed by Major League Baseball teams in the early 2000s.)

Furthermore, Beane saw that the skill of getting on base was not being appreciated by his competitors throughout Major League Baseball, and this was evident from players with this skill being neither paid as much nor sought out as much by rival teams as other hitters with less important skills.

As such, the Oakland A's were able to exploit this market inefficiency and acquire players with the most important skills, with the added bonus being that such players cost the Oakland A's considerably less than worse players with overvalued skills.  As a result, the Oakland A's were able to compete with the sport's landmark franchises like the New York Yankees and the Boston Red Sox. By employing regression analysis, Billy Beane was able to make smarter decisions about how to utilize his limited financial resources, and the Oakland A's were able to thrive for several years.

The most important lesson that PR professionals can draw from Moneyball and the tale of Billy Beane, as well as from the sage words of my colleagues throughout the media measurement industry, is that linking outputs to outcomes and testing the validity of widely held assumptions is not only possible to do, but it also produces extremely valuable results. With regression analysis, PR professionals can gain knowledge that others lack and use these findings to refine their media outreach strategies and tactics to improve overall results and outcomes for their organization.

For instance, PR professionals should be using regression analysis to test whether it's more important to their organization's overall business results that its media coverage be favorable overall or that it reach a larger audience. And, they should use regression analysis to test whether certain key messages are more effective than others at improving sales results, retaining customers, or raising brand awareness.  If PR practitioners engage in these analyses, just like Billy Beane, they might learn that what everyone assumes is true, actually isn't true (or, rather, it's not true to the degree that everyone thought it was).