• In defense of WEIRD science studies

    Not to be confused with weird science studies, WEIRD is an acronym that stands for “Western, Educated, Industrialized, Rich, and Democratic” and it refers to the most common demographic group studied by social scientists. There has long been controversy about how the data of such studies should be interpreted: is a finding an accurate description of human behavior, or is it merely an accurate description of undergraduate American college students who comprise the subject pool of many studies?

    The volume on the discussion was greatly heightened by an important 2010 paper by University of British Columbia researchers Joseph Henrich, Steven J. Heine, and Ara Norenzayan (pdf here). Titled The weirdest people in the world?, their paper reported some eye-opening statistics:

    • 68% of subjects in top psychology journals came from the United States
    • 96% of subjects were from WEIRD countries
    • 73% of first authors were at American universities, 99% at western universities
    • 96% of psychological samples come from countries with only 12% of the world’s population

    The authors go on to cite several basic and uncontroversial conclusions from research taken to be representative of “people” and then adduce data to show that they are not. Specifically, they cover visual perception of one kind of illusion, fairness and cooperation in economic decision-making, and spatial reasoning.

    The main thrust of the paper is that social scientists in general have a tendency to over-generalize from their data and that cross-cultural comparative data is not as esteemed as it should be. For example, consider this article in Psychology Today called Why Are People Bad at Evaluating Risks? Note the language “people” not “Americans” or “undergraduates” and this is not mere media hype. The abstract does not note anything about the group being studied, as if not relevant even though the subjects are all American undergraduate students.

    Overall, I agree with Henrich et al. Quite so, I am in the evolutionary anthropology business and as Henrich noted in the paper,

    More than other researchers in the social sciences, evolutionary researchers have led the way in performing systematic comparative work, drawing data from diverse societies. This is not because they are interested in variation per se (though some are), but because they are compelled, through some combination of their scientific drive and the enthusiasm of their critics, to test their hypotheses in diverse populations.

    Henrich did his graduate work at my own department at UCLA. I am not contesting the broad strokes, largely because there are 28 scholarly replies included in the paper already. There are some interesting criticisms though, such as:

    • The authors condemn the overgeneralizing of WEIRD populations while simultaneously lumping all non-WEIRD populations together as if they were the same (or as if WEIRD popuations are)
    • Variance in data outcomes between WEIRD and non-WEIRD cultures may be a result of methodological limitations, or differences in perception of the task (See Dan Sperber’s reply); neither disproves universality of conclusions
    • WEIRD populations are not, strictly speaking “weird”, but only seem that way because research has been tailored to the culture of origin and thus naturally fits poorly elsewhere (See Bennis and Medin)

    Nonetheless, while I am not refuting the basic claims I do want to contextualize the paper for the non-academics. WEIRD studies are important and necessary, and they always will be for several very good reasons. The first is that they are often intrinsically valuable, whether or not they answer ultimate questions about pan-human traits.

    The Dictator Game
    The dictator game is an experiment based on game theory in which one player is allowed to divide a sum of money (endowment) and give part to a second player (generally the players are strangers). They can keep all of it or none. The second player has no say, and can merely accept or reject whatever is offered. This is one of the measures that varies with culture which Henrich et al. point to as evidence that over-reliance on WEIRD populations is a big problem.

    If this is all you read or know, it sounds like researchers are myopic and ethnocentric. In fact, even purely WEIRD studies using the dictator game have been eye-opening, and not due to the presumption of universality but to the opposite conclusions. Prior to such research, it was assumed that people, as rational actors, would always simply maximize their profit. After all, why give any sum to some stranger you don’t know and will not see again? Among Americans, you see people giving 30-50% of the endowment to a stranger.  This has always been taken to mean that people have cultural values in play, namely concern for reputation and fairness. Further DG studies, even among WEIRD subjects, has attempted to suss out what social cues impact behavior. While non-WEIRD subjects differ, they almost certainly differ for the same reasons: local cultural values and social cues that change how the game is contextualized. The WEIRD studies created important groundwork for further cross-cultural research which brings me to the next point.

    Research pragmatics
    A good cross-cultural study is difficult, time-consuming, and expensive. That means no sane researcher will dive into one (or be able to get grant money for) without having some solid indicators that there is an effect to be studied. How is that accomplished? Partially with existing literature, but also with relatively cheap, easy studies of undergraduates or other samples of convenience. Right now, I am working on a large cross-cultural study involving some 20 countries and several hunter-gatherer  societies. It is a follow-up on pilot research conducted on WEIRD subjects. Everyone involved knows that it may generalize well, or not at all, and even failing to generalize might be interesting if there is a pattern to the data from different kinds of societies. But without the pilot WEIRD study, it would not have been worth anyone’s time, energy, and money to try to find out. It would simply be too risky, and maybe foolhardy. This is part of how social science research works.

    Intended purpose
    One should be cautious about condemning a bit of research merely because it has WEIRD subjects. One reason is the above: it might just be groundwork for further study. But it could also be a deliberate study of that one specific population, and not any others. Sociologists, for example, really are focusing on their own society and not pan-human traits (usually). Cultural anthropologists and ethnographers also often wish to describe one particular culture without necessarily commenting on humanity in general. This can all be true even if they are careless with language in their papers. In my own published work (and coming submissions) there is always a consideration of cultural and social factors and, if possible, discussion of them within said work. This might not be obvious, though, if you never read the actual papers or if the media distorts them as it is want to do.

    Category: Critical ThinkingEvolutionary Psychology

  • Article by: Edward Clint

    Ed Clint is an evolutionary psychologist, co-founder of Skeptic Ink, and USAF veteran.
    • Good article. I think there is still probably an over-inclination to fund studies of same-on-same, due to laziness, inertia, unrecognized blinders, selfishness, etc. — not to mention the issues of ease and expense that you brought up. So certain populations without access to the networks commonly used for grant-funding will be under-served. Meaning that the claim “oh, well THIS is actually the population we’re interested in, we’re not looking for broader applications” is at times worthy of suspicion. Why is THAT the population you’re ALWAYS interested in?
      This works both ways, of course. Richard Herrnstein’s career-long preoccupation with poor urban black populations was hardly due to concern for his fellow man, but in trying in anyway he could to prove: “buh.buh.buh, they’re STOOPIDER,I just FEEL it!” Why did he never investigate IQ differences between blondes and redheads? So if too many well-heeled fair-haired ivory tower theoreticians fetishize upon, oh, what’s trendy now, The Mapuche, let’s say…well, that should raise eyebrows as well. Imperialism dressed up as ethnography.
      Just as an aside, I was hired to review about 150 recent projects from the country’s premier thinktank a decade back, in order to identify patent-worthy methodologies, findings, software, etc. I’m sure my Non-Disclosure is long expired, so I will simply say what one of them was about: “Have The Unwed Pregnancies Of Monica, Rachel, & Phoebe Changed The Contraception Usage Rates Among American Teenage Girls Aged 13-17?” Until I learned of the acronym from your column this afternoon, I would never have known what to call such a study. Now I do. WEIRD. Thank you, Ed.
      And for any Department Heads reading this comment: find the post-doc who wrote that grant proposal and HIRE HER!!!

      • “Why did he never investigate IQ differences between blondes and redheads?”
        I don’t know, but if you replace “blondes and redheads” with “tall and short people” the answer would be “he did”. I don’t have an inkling what his motives for comparing black and white IQ scores were and I would cede that such studies have troubling ethical implications, but to suggest that this was his sole obsession, as you seem to do, isn’t true.

        • I’ll stick by sole obession. I never said his sole area of research. Just his lifelong repeated area. And I do have an inkling as to his motives. (C’mon, try real hard to guess! The same one that makes every lifelong Holocaust denier say, before anything else, “I’m not anti-semitic, i’s just that….”)
          He didn’t earn the name Herr Herrnstein among my undergrad classmates for nothing.

          • James

            From what I remember, The Bell Curve was never really refuted though. Too, psychologists admit that there exists a B-W IQ gap (of around 15 points) on culturally fair tests that has been there for 100 years and hasn’t closed.

            • To what degree do you acknowledge that it has narrowed?

            • James

              I don’t know. William Dickens and James Flynn looked at some studies and said it’s narrowed 5.5 points between 1972-2002. Others (the usual suspects) disagreed and replied.

    • Edward Gemmer

      Really interesting stuff. I also think it is worthwhile to not get too excited about the WEIRD, because it feeds into the idea that we are special snowflakes who are somehow especially different from other people.

      • James

        They are different. The E part, anyway. People who go to college have lower impulsivity, higher motivation and higher IQs.

    • Laura Alberti

      I get the point of the post, but I am not sure that it contextualizes the arguments of the WEIRD article for nonacademics as it says it would, because it touches only very tangentially to the core issues of scientific expertise and power-knowledge dynamics. The WEIRD article speaks about the tendency in behavioral research– and especially cognitive psychology experiments — to overgeneralize and to universalize results based on the statistical methods used, but with little or no consideration of who actually gets top participate in these studies. From what I remember the point of the WEIRD article is that we cannot assume that findings in psychology are universal while only studying one small part of the world population, and not that WEIRD research is not useful and should be abandoned. If we take discourse seriously, until findings are replicated to some extent with participants that are not always Western undergrad, to make authoritative expert statements and predictions about universal human behavior is inaccurate and problematic, but that does not mean that the research is not useful.

      • Yes, I do not mean to imply Henrich or anyone else is making an argument against doing WEIRD research; He called for reform and perspective which is just fine. However, I did feel these points are worth mentioning because some people do get a negatively skewed view of researchers as a result, because they aren’t academics and because these points are not (and I think don’t need to be) mentioned. For example, I have seen studies dismissed because they are WEIRD-based and no other reason- not called out as over-reaching in the conclusion, but dismissed as irrelevant or bad science.