New Evidence for Cost-Effective Vegan Outreach
Earlier, I published a Vegan Outreach Cost-Effectiveness Calculator. The concept involves advertising animal-welfare based reasons to go vegan, such as handing out Vegan Outreach’s “Compassionate Choices” pamphlet [PDF], Farm Sanctuary’s “Something Better” pamphlet [PDF], or by buying Facebook ads that point to WhosAgainstAnimalCruelty.Org as The Humane League does.
By looking at survey results about how many people stop or reducing eating meat because of this outreach, and combining this with data on how long people tend to stay vegetarian and how much this reduces the demand for meat, etc., a simple version of the calculator estimates that one can prevent a year of animal suffering with $0.07 to $76.04. A more complex version of the calculator taking into account more variables guesses $0.02 to $12.26 per year.
This data was rather weak and uncertain because it was primarially based on only one survey, the “Fall 2011 HiddenFaceOfFood.com Facebook Ads Survey” [PDF], which I analyze here and on the calculator page.
However, now I’m proud to announce that with the writing of “The Powerful Impact of College Leafleting (Part 1)” and The Powerful Impact of College Leafleting: Additional Findings and Details (Part 2), Nick Cooney has released new survey data on vegetarian outreach.
The Survey: Method and Results
Unlike the previous survey, which was of Facebook users watching videos linked by Facebok ads, this study was of pamphlets. As Cooney writes:
Early in the fall semester, staffers from The Humane League visited the main campuses of two large state schools on the East Coast, the University of Delaware and the University of Maryland. They distributed thousands of leaflets outside the dining halls of each school. The leaflets distributed were an equal mixture of Farm Sanctuary’s Something Better leaflet and Vegan Outreach’s popular Compassionate Choices leaflet.
About two months later, they returned to campus with surveys to see how much students’ diets had changed. They stood outside the dining halls and asked students passing by if they would take a survey. Students did not know what the survey was about prior to stopping and agreeing to take the survey. After agreeing, only those who actually received a leaflet earlier that semester were allowed to take the survey. Nearly 500 surveys were completed.
And the results were pretty good:
Quite simply, the results were phenomenal. About 1 out of every 50 students who received a leaflet indicated they became vegetarian or pescatarian as a result. Just as importantly, 7% of students (1 in 14) said they now eat “a lot less” chicken, a lot fewer eggs, and a lot less dairy as a result of getting the leaflet. 6% eat a lot less fish, and 12% eat a lot less red meat.
Furthermore, about 1 in 5 students said they shared the leaflet with someone else who then began to eat less meat.
And the key take away:
What does all this mean for animals? After accounting for social desirability bias (people over reporting changes in their diet), the results suggest that for every 100 leaflets you distribute on a college campus, you’ll spare, by a conservative calculation, a minimum of 50 animals a year a lifetime of misery. That’s one animal spared for every two leaflets you distribute!
Where Are We Now?
My immediate thought was that I had to plug in this new data to the Calculator and see where the range fell. When I plugged in the new consumption data and noted the response bias as 0%, and tweaked a few other things to match Nick Cooney’s projections*, the estimates now point to $0.01 to $16.48 to prevent a year of suffering, or 0.08 to 29.6 animals saved per pamphlet to use Cooney’s preferred metric.
You’ll note that the optimistic end of that range is significantly better than Cooney’s conservative guess of one per two leaflets, or 0.5 animals saved per pamphlet. But this is because, I think, his estimate doesn’t include people maintaining vegetarianism for more than one year, influencing others (ripple effect), and fish.
So where does this put our data now? With a second source we’re more robust — we now have data from both Facebook ads and pamphlets, and this data converges well toward each other, as we would expect it would if the data is accurate. We’re also in a a significantly better position with regard to response bias now. Most generally, we’re better off for having a replication.
Some of the fine grain details, while important, are still basically guesses. The most staggering example is that we still don’t really know what it means for someone to reduce their meat intake “significantly” — is this a 40% decrease or a 10% decrease? What about just by “a little bit”? Could this be nothing in the face of desirability bias?
I’d still take the case of guarded optimism here. But it’s worth noting that even if you assume the worst however with the lower bounds and furthermore throw out all but those who say they eliminated entirely, it’s still only $54.06 to prevent a year of suffering, which is on par with GiveWell’s Against Malaria Foundation (assuming you value animal suffering in the same ballpark as human suffering, which I do).
In conclusion, we need more studies and replications. We need more data on some of the other factors. We also need more people to review my calculator, to make sure I didn’t make a mistake in my theory or programming. …But we’re getting the methodologies down. Things are tightening up. And the more we look, the more it seems to be that Vegan Outreach is meeting all of our expectations and more, becoming more and more clearly the best bang for our buck that we can currently quantify.
-
*I had previously had the value for increased at most pessimistic would be 25%, but switched it to 30% to match with Nick Cooney. Likewise, I had previously guessed that a significant decrease would be at most optimistic 25%, but switched it to be 40% to match.
-
I now blog at EverydayUtilitarian.com. I hope you'll join me at my new blog! This page has been left as an archive.
On 21 Jan 2013 in All, Optimal Philanthropy. 6 Comments.
21 Jan 2013, 4:46 pm
Because of design flaws in the survey, I’m not sure whether it should make me any more confident about the effectiveness of leafleting. The following would make me take the survey more seriously.
* preregistration and a commitment to publish the results on the blog regardless of what they were.
* a control group. Do college students in general perpetually report reducing their meat consumption? Maybe.
* survey administration by people not associated with THL. Obviously THL volunteers have a desired outcome, which could affect how they asked the questions.
* a preleafleting survey. Before leafleting, ask students how many times they consumed various animal products in the last week. Ask them the same questions after the leafleting. It seems that these questions would be less ambiguous than those that asked whether consumption was reduced “a little” or “a lot.” The questions would also be less susceptible to inaccurate in respondents’ memories of what they were eating two months ago.
* enabling comments on the CCC blog so I could post this there instead of here, where fewer people associated with THL will read it. I’m suspicious of the content on any blog that doesn’t allow comments.
Some of these measures would be expensive, but a couple would be free. In any case, these concerns should be mentioned somewhere in the blog, but most of them aren’t.
They surveyed 500 students. Let’s say that half of them received a leaflet. That means that 5 out of 250 students reported becoming vegetarians because of a leaflet. That is not an impressive sample size.
21 Jan 2013, 10:57 pm
As the study notes, “only those who actually received a leaflet earlier that semester were allowed to take the survey.” So all 500 students surveyed received a leaflet.
23 Jan 2013, 5:44 am
Agree with Patrick Brinich-Langlois, there’s a real problem with people wanted to conform with what they regard as good behaviour. I think this was a real problem for supermarkets in the U.K. When people were asked questions like “would you buy more organic food if we stocked more?”, or “do you prefer factory, free range, barn or organic eggs” people would report their own “goid” behaviour, but ther buying habits did not change. Later when asked “What does the ‘little red tractor”/ freedom foods/ soil association mark mean, few could answer. I think a number of people will attempt to go vegan/vegetarian, and those who are already leaning that way are more likely to spend time with your leaflettiers, so you have a form of confirmation bias to overcome plus you have yto establish the base rate of peopl going vegan/vegtarian
23 Jan 2013, 5:01 pm
I think it should make you somewhat more confident in the effectiveness of leafeleting. The data is not worthless, and contributes to a cumulative case with the research on Facebook ads (which is far more troublesome from a design point of view), self-reporting of leafleters, and some intuition.
Of course, I’m not suggesting there should be a wild swing in your probability estimates nor am I suggesting the case is closed — far from it, actually. More work needs to be done!
~
This is true of nearly every study, but it would be a big help. It’s easy to think that those administering these surveys could be biased to hide bad results. So far I haven’t seen anything to indicate that, and they could always make up data if they were that nefarious. But it would be better.
~
I think this would be very helpful. It also could be very helpful for helping control more for social desirability bias too. It would also be helpful to get them to fill out the Crowne and Marlowe Social Desirability Scale while we’re at it.
~
As for more on desirability bias, I think it’s worth noting that there isn’t really any reason to suspect a non-response or selection bias involved with the actual surveying in this leafleting example. Also, the calculator has a built in adjustment for desirability bias.
~
The survey was not administered verbally, and question wording has been disclosed [PDF]. The wording doesn’t seem too troublesome, though the survey is very obviously slanted toward decreasing, with multiple categories for decreasing and only one category for increasing.
~
This would certainly get us some kind of information, like more precise values. However, it wouldn’t allow us to track whether or not their consumption changes were stable over time, which is the advantage of returning months later and asking about global change. It’s also worth pointing out that respondents are unlikely to forget whether or not they have gone entirely vegetarian (though this still could be a result of desirability bias).
I think this methodology could be a good supplement to the other methods, however.
~
I was a bit thrown off by that too.
~
As Pablo mentioned, all 500 people received a leaflet. 500 isn’t a super impressive sample size, but it is large enough to be worthy of study, and is in line with the sample sizes of many other research in the social sciences.
12 Apr 2013, 3:05 pm
It’s only “on par” if you think the suffering of humans and animals are about equivalent.
12 Apr 2013, 3:12 pm
Right. It would be impossible for this kind of statement to be objective, as it’s always relative to a value system. I’ve updated the post to clarify in light of this.