Johanna Blakley
Managing Director of the Norman Lear Center, based at the Annenberg School for Communication and Journalism at the University of Southern California
Food, Inc. box office
“Food, Inc.” Box Office Mojo, IMDb.com, Inc.
Food, Inc. awards
“Food, Inc. (2008): Awards.” IMDb, 2019.
The Media Impact Project — also housed within the Lear Center — collects, develops and shares methods for measuring impact. It was founded as part of our effort to expand to platforms beyond TV. When we first started, we knew that we wanted to work with Participant Media because of their incredible presence in this space, so we were thrilled when they asked us to measure the impact of one of their titles.
Food, Inc. was the perfect candidate for many reasons. It prompted viewers to make simple behavior changes in their everyday lives and it encouraged them to effect broader social change by engaging local school administrators and policymakers. It had an extensive social impact campaign, the Hungry for Change outreach effort organized by Participant’s digital arm, and it had a director, Robert Kenner, who had been traveling with the film and engaging communities of action across the US.
There was already a real belief that this film had an impact on those who saw it, but it was our job to prove it. We sought to understand whether there was any shift in the knowledge, attitudes, and behaviors of viewers that could be attributed to their viewing of Food, Inc. When you consider some of our results — like the fact that 84% of viewers said “this film changed my life,” or that 80% of viewers indicated a willingness to join a movement to reform agribusiness — you can start to build an incredibly strong case for the film’s impact. If you take a look at the matched control group findings, that case becomes even more compelling.
Controlling for self-selection bias
Media effect studies have long suffered from not having control groups. I really think that they should be instituted as a best practice in all media impact evaluations, and our study of Food, Inc. is a clear example of their value and importance.
When setting out to study the film’s impact, we knew that we would likely be working with a highly self-selected sample. Respondents to the survey answered our call because they were already signed up to Participant’s email newsletter, or had engaged with Participant on social channels. Naturally, this population would be more likely to say that a film like Food, Inc. had changed their lives, so finding a control group was essential.
Media Impact Project survey for Food, Inc.
The Media Impact Project survey offered a wealth of qualitative and quantitative data on the impact of Food, Inc. 20,000 respondents took the survey.
Blakley, Johanna, et al. Changing Appetites & Changing Minds: Measuring the Impact of Food, Inc. Media Impact Project, USC Annenberg Norman Lear Center.
http://www.mediaimpactproject.org/uploads/5/1/2/7/5127770/food-inc-summary-report.pdf
The Lear Center developed an innovative survey instrument that would assess the impact of Food, Inc. on its viewers while considering these issues of self-selection bias. We used a method called propensity score matching (PSM) to help determine whether the different results that we saw between viewers and non-viewers were associated with watching Food, Inc., rather than pre-existing differences between these two groups. Simply put: PSM allowed us to say, with a much greater degree of confidence, whether the differences between viewers and non-viewers in knowledge, behaviors and attitudes were attributable to the film.
Sure enough, the differences we uncovered between viewers and non-viewers (or the exposed group and the control group) suggested that with Food, Inc. there was a very, very powerful media intervention. We found that it was more likely that people who had seen the film were shopping at farmers’ markets and eating healthy food, were more knowledgeable about genetically modified foods and sustainable agriculture, and were more engaged in organized efforts around tackling issues in agribusiness.
The self-reported impact
This is not to say, however, that nothing can be learned from self-reported changes in knowledge or attitudes. Over two-thirds of respondents said they learned “a lot” or “very much” about issues ranging from agricultural policies to food safety. 79% of viewers agreed that Food, Inc. explained what they could do to help solve the problems addressed in the film, and 80% said that, having watched the film, they would consider joining a social movement to reform agribusiness. This last statistic became even more impressive when we noted that only 37% of respondents actually considered themselves to be politically active.
Food, Inc. media exposure and preferences
Blakley, Johanna, et al. Changing Appetites & Changing Minds: Measuring the Impact of Food, Inc. Media Impact Project, USC Annenberg Norman Lear Center.
http://www.mediaimpactproject.org/uploads/5/1/2/7/5127770/food-inc-summary-report.pdf
The importance of “scaffolding”
Documentary is clearly a powerful tool for motivating social change, but it may not be enough on its own. You also need a supportive infrastructure outside of your media intervention — a social impact campaign, for example. This “scaffolding” will offer ways to carry on the conversation after watching the film, channel audience energies and bring people together to take action, both online and off. Without this, you’ll be left with a motivated but disappointed population — eager to act, but unsure of what to do. This scaffolding should already be in place before audiences see the film.
As part of the Hungry for Change campaign, Participant’s digital team built a robust website where people could find out more about the huge environmental impact of the food industry, from pesticide usage to factory farming. There were links to petitions they could sign to combat these issues, and detailed information about the steps they could take to reduce their own tacit support of an industrialized food system. There was also information about how to arrange local screenings of the film and downloadable discussion guides to help stimulate conversation in the community. These two resources in particular helped to build a strong grassroots activism component around the film.
Theorizing change, measuring impact
Measuring impact is never going to be easy, but I think there are three categories of data that, if collected, can allow us to make very convincing arguments about what a piece of media can lead people to do.
The first is content analysis. Starting off with a clear understanding of the content in a medi intervention is key. Whether it’s a single documentary film or 32 episodes of scripted TV, we carefully analyze the content, including the language used, factual information shared, calls to action, as well as the tone and narrative framing.
Food, Inc. viewers’ political views
Blakley, Johanna, et al. Changing Appetites & Changing Minds: Measuring the Impact of Food, Inc. Media Impact Project, USC Annenberg Norman Lear Center.
http://www.mediaimpactproject.org/uploads/5/1/2/7/5127770/food-inc-summary-report.pdf
The second is exposure and engagement data. If the media is available on a digital platform, we can gather data about who was exposed to the content and how they interacted with it. Did they watch the whole thing? Did they share it? We always encourage content creators to provide opportunities for people to take further action on the same platform, such as donating or joining a mailing list, so that we can draw a clear connection between exposure and taking action.
The third is survey data. If we want to know whether media content produced changes in awareness, knowledge, attitudes, and offline behavior then we need to gather survey data. Ideally, these surveys can be administered on the same digital platform where people consumed the media content. Then we have a very clear picture of what content people consumed, how they engaged with it and what potential impact it had.
Food, Inc. viewers’ self-reported changes in knowledge and attitudes
Blakley, Johanna, et al. Changing Appetites & Changing Minds: Measuring the Impact of Food, Inc. Media Impact Project, USC Annenberg Norman Lear Center.
http://www.mediaimpactproject.org/uploads/5/1/2/7/5127770/food-inc-summary-report.pdf
It’s clear that much of the data we need to measure impact is out there, and we have the chance to do this work better than ever before. But there’s a growing (and completely warranted) concern around the misuse and abuse of data that could stymie our progress on this front. I think it will be difficult for academics, institutions, and organizations to win back people’s trust in this regard, but we have to — especially because we’re finally waking up to how crucial an impact media is having on our society.
Propensity Score Matching
By and large, the majority of people choose the films they see based on personal taste and interests. As such, researchers looking to study the impact of a given title will almost always have to deal with a certain bias. The research done at USC’s Lear Center was no different. To try and overcome this bias, however, they developed a unique survey instrument: propensity score matching (PSM).
“In this study, propensity score matching (PSM) techniques were used to control for self-selection bias among survey respondents. PSM helped identify factors that predict the likelihood of a person seeing Food, Inc. The Lear Center’s research team performed a statistical analysis of survey responses from all the respondents who watched Food, Inc., and determined what personal characteristics increased their likelihood — or propensity — to see the film.
We used these characteristics to generate a propensity score for survey respondents who had seen the film, and those who had not. People with all 17 of these characteristics received the highest score and those with the fewest received the lowest. However, just because someone has a high score does not mean that they have seen the film — it just makes it more likely that they have seen it. Therefore, a person with a very high score may not have seen the film yet, and a person with a very low score may have seen it despite themselves (for instance, a teacher may have screened the film in a class where students with low scores saw it).
Once scores were assigned, we created two groups: people who had watched Food, Inc. and those who had not. Next we compared the range of scores in each group and then performed ‘one-to-one matching,’ which allowed us to use an automated method to remove subjects from each group until both groups were composed of the same number of respondents with the same range of propensity scores (e.g., each person who saw the film was paired with a person who did not see the film, but was equally likely to see the film based on their propensity scores). The salient difference between the two groups was whether or not they had viewed Food, Inc.
This method allowed us to create something similar to an experimental study design where subjects are randomly assigned to a control group and a treatment group. Here, the “treatment” group is comprised of those who had seen Food, Inc., and the “control” group is comprised of those who had not seen the film but were equally likely to. By making these groups completely parallel, we were able to examine whether differences in knowledge and behavior are attributable to exposure to the film.”
Blakley, Johanna, et al. Changing Appetites & Changing Minds: Measuring the Impact of Food, Inc. Media Impact Project, USC Annenberg Norman Lear Center.
http://www.mediaimpactproject.org/uploads/5/1/2/7/5127770/food-inc-summary-report.pdf
