2.13 Skeptical thinking is the path to an improved cross-cultural psychology.

So many aspects of human thinking are influenced by cognitive processes seeking easy solutions. Cross-cultural psychology is complex and difficult and requires students and researchers willing to cultivate a skeptical mind. The many problems of bias discussed above is partly the consequence of looking for simplistic answers to the most complex science of all that of understanding human behavior in the cultural context. At the foundation of many biases in cross-cultural psychology is the inability of the researcher to observe his/her own evaluative biases in how research is conducted and carried out. After completion of the research there are also many possibilities for bias in interpretation. Although we are trained to think analytically, and look for differences even when these are minimal, we should be skeptical about generalizations. Most findings apply to specific circumstances and time. An obvious evaluation question is to ask what other factors may explain our findings. This is especially salient when cultural variables are inferred and not measured directly.

Although we may find differences in comparative studies it is well to remember that the participating groups are heterogeneous along pertinent dimensions. Therefore any conclusions pertain only to the average responses required by statistical techniques that do not take into account this within group variability. Although a society may be low on average education, it still may have an educated class that has a strong influence on developments within the cultural group. A random sampling of such a society would deemphasize other problems like whether the participants were required to attain a certain educational level to understand the survey questions or research protocol.

As we can see the complexity of cross-cultural research create many issues that undermine the validity of research results. The obvious problems of linguistic equivalence can be disposed with little difficulty using proper translation procedures. Other problems like equivalence in understanding the experimental constructs the same way in different cultures present more difficult issues in comparative research. Validity ultimately involves the correct interpretation of the results. A pertinent issue related to validity is the extent to which we have sampled issues that are truly salient in the cultures investigated. Issues significant in the society of the investigator are not necessarily salient in other cultures? Cultural comparative studies must be approached from a variety of angles including the use of alternate methods, samples, and procedures, and the researcher must recognize the continual need for replication.

Summary

The value of cross-cultural psychology depends on the reliability and validity of the research. It is well to keep in mind that the complexity of the discipline offers many opportunities to bias the results. The most significant issue in comparative research is equivalence along several criteria whether the research protocol is based on surveys, attitude scales, interviewing, experiments under cultural conditions, or qualitative studies. Linguistic equivalence is possible by following proper procedures including establishing initially a bi-lingual committee to work toward a consensus about the use of language meaning in the research protocol. The back-translation procedure further ensures that the same meaning is conveyed to the groups surveyed. Psychometric equivalence seeks evidence to ensure that the same construct is measured in comparative research. Criteria are established by comparing whether the same preference order in response categories and item difficulty is found in the groups surveyed. The structure of the psychological assessment can also be established by means of item analysis. An important equivalence issue is whether the comparative research is based on the same theoretical framework in the cultures that are compared. The psychometric structure can also be examined by means of factor analysis and other statistical techniques.

Comparing equivalent samples is essential to cross-cultural research. This is not easily achieved as factors not known about the culture may affect responses such as dissimilar education and literacy. In that regard we must remember that cross-national samples do not have the same meaning as cross-cultural samples as one nation may contain many cultural groups. Heterogeneity in cultural values confounds many national comparative studies. Matched samples create other ambiguities since they may create mismatching on other salient variables. Convenience sampling is not random, but may be useful in developing measurement instruments. Sampling that is based on theoretical considerations is called systematic when such samples are drawn with proper demographic controls. However, random sampling is the only approach considered representative of the population studied and if large enough represent the cultural group with little error. Equivalent samples can also be built using similar demographic variables. Non-equivalence is however ubiquitous in cultural and cross-cultural studies that utilize approximations to construct methodological equivalence. In survey research equivalence can be promoted post hoc by identifying non-equivalent items and eliminating these from the analysis.

Cross-cultural studies are complex as they require different levels of inferences. Low level inference is possible when the researcher constructs item samples directly from the domain of interest as occurs in the building of attitude scales utilizing statements from the universe of all possible items. Medium level inferences are not directly observed but believed to be determining factors of behavior. Factors considered medium level include personality traits where the aim of the assessment is to capture the essence of the domain. The ambiguities that occur at this level call into question whether the construct measured is in fact comparable across cultures. High level inferences concern more complex domains not easily specified, and which are not easily captured by measurement techniques. At this level inferences are typically made post hoc without empirical evidence such as occur in evaluations of research on constructs like adaptation or intelligence.

Useful cross-cultural studies have been carried out at the ecological level reporting average statistics that point to underlying dimensions responsible for culturally related behaviors. For example studies that report on the presence of individualism in a culture have been related to the rate of heart disease. However, cross-cultural comparative research seeks to ascertain differences on psychological assessments. We must however remember there is more to culture than the psychological world and survey results in any event measure only a small sampling of the domain of interest. Not measured are differences in the natural world, the ecological context, or childrearing procedures, that all have the potential to significantly influence behavior. Examining specific cultural context may explain differences in psychological attributes, and psychological survey results are often explained at the contextual level. There are however problems in interpretation when results at one level are explained by inference to another level. A step forward is to measure specific aspects of culture through the construction of scales and surveys.

The possibility of bias exists in any psychological assessment. Survey or scale items may be poorly constructed or inadequately translated. Such item error can usually be corrected by statistical testing. However, if the entire methodology is problematic the only solution is to return to the drawing board and select alternative research approaches. Bias in psychological assessments may occur due to differences in respondents familiarity with the construct measured, due to poor sampling of the concepts, and due to the influence of social desirability. Research has also demonstrated differences between individual versus group enhancement in varying cultures that also might bias the respondents answers. Response sets also influence responses in cross-cultural comparative studies, and the cultural blinds of the researcher produce implicit bias in construct and methodological selection.

All disciplines in psychology are interested in establishing cause and effect relationships. However, the special problems of cross-cultural psychology make that objective difficult. Most studies in cross-cultural psychology are correlational as it is impossible to assign cultural respondents randomly to experimental treatment. However, even in correlational research the constructs must be operationalized and measured or inferences about culture cannot be made.

Researchers in the field have often taken antagonistic positions about the value of qualitative versus quantitative research. Qualitative research is often employed for in-depth studies of a single culture, but in the context of cross-cultural research can be seen as exploratory and hypothesis generating for use in quantitative studies. The argument in favor of qualitative studies is that cultural reality is socially constructed and therefore only a relationship between the researcher and the culture (represented by informants) can assess that reality. Typically qualitative research is associated with behavior in the natural world, and best carried out when the observer works in a participant-observer relationship. Theory development in qualitative research is based on an inductive process utilizing multiple sources. However, both approaches are not antagonistic, but rather complementary despite contentions centered on the validity of the results produced.

Quantitative methods include surveys that test for psychological differences between cultures on established psychological assessments. Surveys can be open-ended but questions with fixed categories are easier to quantify. Interviews that use standardized questions can also be useful if the researcher is careful in not reinforcing certain responses. Although telephone and Internet surveys are easy to conduct the researcher must be careful to avoid bias derived from differences in socio-economic status measured by access to these means of communication. Meta-analysis creates a summary statistic and an overall integration of the results. However, bias may be introduced as the focus is only on statistically significant studies ignoring perhaps the larger pool of insignificant results.

It is also possible to design studies that test for the affect of cultural context variables where cultural populations are selected in advance as known to have characteristics of the domain of interest. Replications in other cultures of these results can be seen as evidence of validity as well as the presence of universal psychological characteristics. In one study the mindsets of the participants were primed for individualistic or collectivistic responses with predictable results.

All cross-cultural research deals with the issue of validity. The best answer to these concerns is to utilize all available approaches, demonstrate conceptual clarity, and replicate studies over time and cultural contexts. A critical look at the field cautions that the researcher's cultural frame may introduce bias in selection of domain and methodology. Also respondents are more sensitive to their own cultural frames and that may produce biases in responses. At the same time globalization and new means of communications call into question the permanence of any cultural values. Finally, the reporting of only statistical significant findings may introduce bias as summaries like those found in meta-analysis do not take into account the possible broader pool of insignificant results. Further any study represents only a fragment of the relevant psychological domain or the cultural context of interest. Consequently skeptical thinking is required in order to improve validity since cross-cultural psychology is both difficult and complex.

Chapter 3