Harvard's Implicit Psychopathology Problem

Ulrich Schimmack

January 31

A popular children's song in America has the lyrics "When your happy and you know it, clap your hands."

Since the 1960s, hundreds of surveys with millions of people have relied on self-reports of happiness to study wellbeing. While critics have challenged the validity of these reports (Schwarz & Strack, 1999), hundreds of studies have shown that these reports have some validity (Diener, Lucas, Schimmack, & Helliwell, 2009). For example, they show moderate correlations with informant ratings (Schneider & Schimmack, 2009; Zou et al., 2013) and national averages correlate r = .8 with average median income across nations (Gallup World Poll, 2008-2023).

At the same time, self-reports are not 100% valid. They are influenced by response styles and socially desirable responding. To address these concerns, it would be beneficial to have alternative measures of happiness that do not rely on self-reports.

Over the past 20 years, social psychologists have popularized the use of implicit measures. Implicit measures assess content in people's memory without asking for the information directly. The most popular implicit measure is the Implicit Association Test (IAT). In this task, respondents work on two independent classification tasks that are easy to do on their own. For example, they have to press one button for words related to them and another button for words not related to them (me / not me). In the other task, they have to classify words related to happiness (smiling) and sadness (crying). The critical trials mix the stimuli on the screen and pair responses on the same key. Most people find it easier to do the task when me is paired with happy and not me is paired with sad than in the reverse order. The crucial question is whether differences in reaction times on this task provide a valid measure of individual's happiness. For example, somebody may "clap their hand," report high happiness on a survey items, but the IAT shows faster responses in the "me/sad vs. not me/happy" than in the "me/happy vs. not me/sad" condition.

Walker and Schimmack (2008) examined the convergent validity of the happiness IAT with self-reports and informant reports. The use of informant reports was crucial because the IAT is only useful if it provides information about happiness that self-reports do not provide. If the happiness IAT merely confirms the information of self-reports (i.e., people who know how happy they are and report it accurately also show faster responses in the "happy-me" condition). However, Walker and Schimmack replicated moderate agreement between self-ratings and informant ratings and found equally low correlations with the happiness IAT. These results suggested that the happiness IAT cannot be used to improve the measurement of happiness.

The Harvard Implicit Association Test

In the United States, Harvard is a brand, although the brand has lost a bit of its luster lately. Nevertheless, many people may associate Harvard with excellence in science and assume a test administered by Harvard is scientifically valid (I did not administer a Harvard - Science IAT to test this hypothesis).

Visitors of the Harvard IAT website are not provided with any information about the validity of their test scores. This is problematic because people may assume that their test scores are valid without a warning that their test scores can be biased by measurement error. This is particularly problematic when the test is used for the assessment of mental illnesses like depression or suicidal ideation. I have shown that the suicide IAT lacks evidence of validity (Schimmack, 2021). However, I was not able to find information about validity of the depression IAT, that is essentially a happiness-sadness IAT (me/not me, happy/sad). I emailed the Project Implicit team to ask for information about validity of the Harvard IAT as a measure of clinical depression.

Here is the answer:

Hello, and thank you for your question.
We would advise contacting the Project Implicit Health team: https://implicit.harvard.edu/implicit/user/pih/pih/thescientists.html
The Project Implicit Services Team

I then contacted Dr. Teachman, who was listed first on the Project Implicit Health website. Dr. Teachman sent me two articles. One of them was written by Dr. Teachman and colleagues and published in 2019 (Teachman et al., 2019).

To summarize the article, there is essentially no evidence that the depression IAT measures clinical depression or even just normal variation in happiness.

1. Disclosure Statement

All scientists, including myself, have biases. It is therefore important to make readers aware of potential conflicts of interest. Teachman does point out that she has a stake in IAT research on mental health. This makes it more likely that she would present ambiguous evidence in a more positive light.

2. The summary statement

makes it clear that the IAT is a procedure that can be more or less valid. For example, I showed that it works well for the measurement of political orientation, but not for self-esteem (Schimmack, 2021). Teachman et al. point out that it works better for anxiety than for depression. Yet, they also point out that measures have methodological limitations.

3. Major Depressive Disorder: Empirical Evidence

The review of the empirical evidence starts with the observation that there is lack of convergent validity.

Then a couple of studies are mentioned that show the expected relationship.

These inconsistent findings are used to claim that there is convergent validity between IAT scores and clinical diagnoses, while pointing out that the evidence is mixed. What is lacking is some quantitative information about the amount of convergent validity between IAT scores and clinical diagnoses.

A valid measure of depression should be sensitive to changes in depression. However, this evidence is also mixed.

This section ends with the conclusion that "measures of implicit cognition are useful for understanding the phenomenology of MDD" (p. 135). Call me crazy, but I do not see how this conclusion follows from the mixed evidence reviewed in this section.

Conclusion

Project Implicit uses the Harvard brand to lure people to their website to take tests to obtain data without paying participants. Participants may assume that they are provided adequate compensation because they receive feedback about themselves from scientific tests of depression and other mental health problems. These visitors are not provided with information about the validity of tests and warned that test scores may be invalid due to measurement error. Moreover, the Project Implicit researchers do not even have information about the validity of their tests and ignore valid criticism of the IAT as a measure. While this may not be a problem when the IAT is used to diagnose whether somebody likes Pepsi or Coke, it is a serious problem when visitors take a mental health IAT and receive false information that they suffer from implicit depression or have suicidal ideation outside of their awareness. It is clear why Project Implicit likes to sell their questionable tests with the Harvard brand, but it is less clear why Harvard would want to further taint its brand with pseudo-scientific tests.