As we look around the globe, one fact becomes immediately apparent— remarkable diversity, as well as similarity, characterizes humanity. The term diversityoften refers to the varieties of human characteristics found within a culture, but also sometimes implies similar variety across cultures. Just within the United States, the population reflects a strikingly diverse tapestry composed of variations in gender, race, ethnicity, sexual orientation, language, ability, income, culture, and other elements of diversity. We live in a multicultural and international community with multi-layered, intersecting threads representing potential areas of study and understanding. Researchers may elect to study not only one element of diversity but also how this element may differ across cultures and/or interact with other elements of diversity. For example, one might choose to study gender differences within a culture, differences between two or more cultures related to gender, or the intersection of gender and sexual orientation either within or across cultures. Regardless, it is imperative that researchers conduct their investigations in a methodologically valid and culturally appropriate manner. It is not enough simply to include some element of diversity as a variable to study. Rather investigators must take care to insure that the methods chosen to research that variable are adequate, appropriate, and ethical. Failure to conduct cross-cultural or diversity research with an eye toward appropriate methodology may result in faulty results and erroneous conclusions.
Despite the spectrum of diversity as one looks around the globe, psychological research and study within the U.S. has failed to reflect that diversity. According to Louis Kincannon, Director of the United States Census Bureau (2007a),
About one in three U.S. residents is a minority. To put this into perspective, there are more minorities in this country today than there were people in the United States in 1910. In fact, the minority population in the U.S. is larger than the total population of all but 11 countries. (para. 1)
The U.S. Census Bureau (2007b) also reported that the minority population is not localized in one region. Indeed, “nearly one in every 10 of the nation’s 3,141 counties has a population that is more than 50 percent minority” (para. 1). Despite these figures, several researchers have noted a paucity of published research focusing on racial or ethnic minorities within psychology journals (e.g., Bernal, Trimble, Burlew, & Leong, 2003; Carter, Akinsulure-Smith, Smailes, & Clauss, 1998; Graham, 1992, Sue, 1999). According to a number of investigators, some researchers simply fail to report the racial and ethnic background of study participants (Buboltz, Miller, & Williams, 1999; Delgado-Romero, Galván, Maschino, & Rowland, 2005; Munley et al., 2002). Delgado-Romero and colleagues (2005) reported that only 57% of examined counseling articles published between 1990 and 1999 reported race and ethnicity. The results of their analysis revealed that White and Asian Americans were overrepresented, whereas African Americans, Hispanics, and Native American Indians tended to be underrepresented as compared to the U.S. population. Finally, it is also important to note that much of the traditional psychological curriculum has historically neglected issues of diversity (Guthrie, 1998). For example, despite the fact that approximately 45% of articles indexed in PsycINFO© included authors from outside the United States (Adair, Coelho, & Luna, 2002), introduction to psychology, life-span developmental psychology, and social psychology textbooks included little of this research (Woolf, Hulsizer, & McCarthy, 2002). Ultimately, it would seem that Betancourt and López’s (1993) assertion that “the study of culture and related variables occupies at best a secondary place in American (mainstream) psychology” (p. 629) is as true today as it was almost two decades ago.
Fortunately, there has been a recent push to increase both an awareness of and research concerning diversity, cross-cultural issues, and international research within psychology. In January 2000, the Council of National Psychological Associations for the Advancement of Ethnic Minority Interests (CNPAAEMI) produced the Guidelines for research in ethnic minority communities. Five ethnic minority associations within the American Psychological Association (APA) collaboratively worked to create a document reflecting issues relevant to research with African American, Asian American, Hispanic, and American Indian populations. In 2003, the APA published the Guidelines on multicultural education, training, research, practice, and organizational change for psychologistsand in 2005 and 2008 hosted Education Leadership Conferences devoted to diversity and international issues, respectively. Indeed, the APA (2009) vision statement now identifies the Association as
A principal leader and global partner promoting psychological knowledge and methods to facilitate the resolution of personal, societal and global challenges in diverse, multicultural and international contexts. (para. 9)
In 2005, the APA Working Group on Internationalizing the Undergraduate Psychology Curriculum, in concert with the American Council on Education (ACE), published a set of recommended learning outcomes for psychology courses (Lutsky et al., 2005). The report noted that psychology has long been international both as a science and as a profession with early psychologists working in such diverse countries as Argentina, India, Japan, Mexico, New Zealand, and Russia. Goal 2 of the report focused on research methods and highlighted that “Students should be aware of research methods and skills necessary for international research competence” (p. 3). The report presented five learning outcomes in relation to research methods, specifically that students will develop: (1) the ability to access and read journals published outside the United States; (2) the ability to identify and appreciate the range of research methodologies used by psychologists around the globe (e.g., ethnographies, observations); (3) an awareness of ethical issues that may be of concern in other countries (e.g., protection from harm); (4) an understanding that many constructs may have different meanings in different cultures; and (5) an appreciation of the dangers associated with making broad generalizations based on small and potentially unrepresentative samples.
In this chapter, we focus on a number of issues related to the role of diversity in understanding research methods. It is important to note that we do not discuss all issues related to diversity concerns and research. Rather, we seek to introduce the topic by providing representative examples which highlight the complexity involved in conducting research responsibly—particularly research that involves diverse populations. Moreover, much of what we discuss in each section below describes research focusing on differences between groups (e.g., differences between two cultures). However, we would like to highlight that this does not negate the need for research to be more inclusive of diverse populations in non-diversity focused research. It is important that researchers endeavor to be more inclusive of diverse populations in all of their work.
Marsella (2001) suggested that many psychologists are unwilling to “accept a very basic ‘truth’—that western psychology is rooted in an ideology of individualism, rationality, and empiricism that has little resonance in many of the more than 5,000 cultures found in today’s world” (p. 7). Indeed, as we saw in Chapter 1, psychology textbooks and journals are filled with psychological concepts, theories, and research findings that, at the surface, appear to be applicable to all humans. Yet, there is a great deal of cross-cultural as well as intra-cultural variability. Moreover, there is growing awareness that the use of homogeneous samples (e.g., introductory psychology students within the U.S.) has painted a somewhat distorted picture of psychological phenomena (Graham, 1992; Sears, 1986). Unfortunately, although homogeneous samples may increase the internal validity of a study, it is often at the expense of the ability to generalize to a range of diverse populations. All of the above highlights concerns related to the topic of research validity, specifically construct, internal, and external validity.
Construct validity typically refers to the degree to which a variable has been operationally defined so that it captures the essence of a hypothetical construct under study. For example, does a particular depression scale adequately measure depression or is an IQ test a good measure of intelligence? If the answer is “yes,” then the measure has good construct validity. If the answer is “no,” then any research conclusions drawn from the use of the measure are, at best, limited. An examination of construct validity is necessary in any cross-cultural or diversity research.
When conducting cross-cultural research, experimenters must monitor the degree to which their methodology is equivalent across cultures and groups. In fact, Matsumoto (2003) suggested that researchers should strive to create the perfect cross-cultural study. To that end, Brislin (2000) asserted that investigators need to develop an awareness of three sources of nonequivalence—translation, conceptual, and metric. Translation equivalence is necessary when conducting research on one population using experimental measures developed and standardized with another population. For example, if a researcher plans to use a standardized questionnaire developed in the U.S., translation equivalence needs to be achieved prior to its use in another country. So, how should the researcher go about translating a scale or experimental measure? According to the Guidelines for research in ethnic minority communities (CNPAAEMI, 2000), the proper translation of experimental measures involves the use of the back-translation method. First, a bilingual translator converts the measure to the target language. Next, a second translator converts the translated measure back to the original language. Finally, a researcher compares the original version to the back-translated version to examine any existing differences. The process is repeated until translation equivalence is achieved.
Conceptual equivalence is the degree to which theoretical concepts or constructs are the same between two cultures. Unfortunately, researchers can introduce conceptual nonequivalence in a variety ways, even through something as seemingly innocuous as demographics. Although it is often useful to control for certain demographic variables, such as age, sex, and religious orientation, this may become problematic when conducting cross-cultural research. For example, a demographic question such as “age” would seem relatively innocuous. Yet, for the Ju/’hoansi, also known as the !Kung of the Kalahari, this would be not be a useful question, as the Ju/’hoansi use a culturally specific age categorization system as opposed to thinking of age in chronological years (Hames & Draper, 2004). The inclusion of an indigenous (local) investigator or collaborator may serve to increase the validity of cross-cultural research because this inclusion can better enable the research team to ask the right questions (CNPAAEMI, 2000).
Finally, researchers need to be concerned with metric equivalence—the ability to compare the specific scores on a scale of interest across cultures. For example, would a researcher interpret a score of 25 on the Beck Depression Inventory (Beck, Ward, Mendelson, Mock, & Erbaugh, 1961) the same way across cultures? Would individuals living under repressive regimes, where government intrusion is a daily occurrence, score differently on the paranoia scale of the Minnesota Multiphasic Personality Inventory (MMPI-2; Butcher et al., 2001)? Additionally, metric equivalence may not exist within a culture when studying diverse populations. For example, researchers have rarely standardized measures of assessment used in research, particularly those related to personality assessment and psychopathology, for use with a disabled population (Elliott & Umlauf, 1995; Pullin, 2002). In addition, disabled individuals may score differently than others on the Hypochondriasis Scale of the MMPI-2 (Butcher et al., 2001), with seeming obsession over body functions. Varying cultural norms about time, differences in physical ability, and age may all impact completion of timed tests of intellectual ability.
Inadequately standardized measures that lack a representative normative group threaten construct validity by introducing the possibility that individuals are evaluated on performance as opposed to ability. Unfortunately, researchers rarely design studies to accommodate participants who are differently abled. Tasks designed to measure cognitive, personality, or other psychological abilities may instead be measuring primarily noncognitive or nonpsychological variables in research inclusive of participants with a range of abilities. For example, research on memory using computers to present stimuli may be more challenging for some older adults due to an increased incidence in susceptibility to glare from cataracts. Research participants of any age who are in ill health may fatigue more easily and perform poorly on outcome measures. Thus, differences found between participants may reflect visual or health differences as opposed to memory differences. Unless the researcher takes into account the impact of disability, age, or other factors, the resulting research conclusions may be biased and inaccurate.
Internal validity reflects the confidence with which we can draw cause and effect conclusions from our research results. Of primary concern for researchers is the ability to eliminate, minimize, or hold constant all extraneous or confounding variables. Unfortunately, studies examining differences between various elements of diversity (e.g., gender, culture) are by their very nature quasi-experimental and hence limited in relation to cause and effect conclusions. Indeed, this issue may contribute to the paucity of diversity research within psychology. In a provocative article, Chang and Sue (2005) suggested that the current scientific paradigm, with its focus on experimental designs, has introduced a bias in psychology. Specifically, they asserted that mainstream journals systematically exclude multicultural research, thus devaluing existing multicultural research and impeding the growth of research in this area. Ideally, experimental research should be high in both internal validity and external validity. Unfortunately, it is very difficult to achieve both in experimental designs, and extremely problematic when using alternative designs (e.g., quasi-experimental, correlational, observational, ethnographic). Chang and Sue argued that the importance psychologists have placed on determining causality has led researchers to value internal validity over external validity.
Chang and Sue (2005) further asserted that the emphasis psychology places on causality (internal validity) has led to a host of additional research problems, which only serve to devalue external validity and the very nature of multicultural research. The problems they discussed included: (a) overuse of college students as research participants; (b) willingness to assume research conducted on one population (e.g., White, middle-class, U.S. citizens) can be generalized to other groups or contexts; (c) disregard for research seeking to explore cross-cultural differences as opposed to explaining such differences; (d) the tendency of journal reviewers to insist that researchers add a White control group when conducting research on ethnic minority groups; and (e) the formation of aggregate non-White populations to obtain a large enough sample size. The last issue is especially problematic due to the implied assumption of homogeneity among the non-White population, when in fact non-White groups may differ widely. Although the exclusion of diverse participants may enhance internal validity, the resultant homogeneous sample does not reflect the diversity of human experience.
External validity is the degree to which research results can be generalized beyond one’s sample. Often psychology focuses on the universality of human cognition and behavior. Researchers often refer to these universal concepts as etics. Concepts thought to be specific to a particular culture are called emics. Matsumoto (1994) suggested that, “most cross-cultural psychologists would agree that there are just as many, if not more, emics as there are etics” (p. 5). Many psychology textbook authors present traditional descriptions and recommended treatments for posttraumatic stress disorder (PTSD) as universal, or etics. Yet, PTSD symptoms and treatment are frequently culture-specific (Atlani & Rousseau, 2000; Bracken, 1998). The primary lesson of external validity is that researchers should exercise great care when extrapolating beyond their sample and this is particularly true when conducting cross-cultural or diversity research.
One of the primary issues at the core of external validity in relation to cross-cultural research is that variability exists within cultures. For example, the Guidelines for research in ethnic minority communities (CNPAAEMI, 2000), in relation to issues of using Hispanics or Asian/Pacific Islanders in research, stated that above all else, researchers need to be aware of the diversity of these populations. These broadly defined groups vary with respect to acculturation, language, race/ethnicity, beliefs, socioeconomic status, and educational background. Indeed, they do not represent homogenous cultures. Anyone critiquing research involving Asian/Pacific Islander or Hispanic participants must consider all these issues. These same issues are also fundamental to research guidelines associated with Native American Indians. In addition to research design concerns, investigators need to be prepared for small sample sizes. In fact, researchers may end up dealing with a single and consequently small population, given the fact that each tribe is a distinct cultural entity.
The issue of external validity also applies to diversity research within cultures. For example, Eichler (1991) cited the problem of overgeneralization in gender research—the propensity to extend concepts, theories, and research developed from studies involving participants of one gender to all individuals. Examples of overgeneralization ranged from the use of sexist language to the tendency of researchers to generalize the results of a study to all individuals when in fact the initial sample was composed only of men. Hyde (1996) also asserted that researchers often employ a “female deficit model” when drawing conclusions—the tendency to frame the results in such a fashion that women’s behavior is seen as deficient or non-normative. Research conducted in highly patriarchal cultures may magnify this issue.