2.7 Inferences from statistical tests on cross-cultural comparisons.

In all fields of psychology, including cross-cultural research, inferences are drawn from statistical methods like analysis of variance. These techniques allow us to determine that the differences between cultural groups are statistically significant, and have not occurred by chance. However, the fact that an observed difference does not occur by chance often leads to the unwarranted assumption that the difference is important and meaningful. Tests of mean differences do not provide estimates of the degree of difference, nor the amount of variance that is accounted for by the variables. In most cases, even when there are large and meaningful differences, the within group variance is probably greater than the between group observation.

Another faulty assumption is the idea that because there are statistically significant differences between cultural groups that most members of groups represented in the study differ in the direction indicated by the mean values. However, even with large mean differences a considerable overlap between groups is possible in the variable studied. It is important to evaluate the degree to which differences are meaningful and not just statistically significant. A test for the amount of variance accounted for by the research variables is possible with correlational techniques, and other effect size statistics (Matsumo, Grissom and Dinnel (2001). Scholars in Western societies in particular value the search for significant differences consistent with individualistic culture, and less often look for similarities between cultural groups as not nearly as interesting. One outcome of the “difference obsession” is the focus on statistical tests of significance, even when differences are small and not meaningful.

2.8 Experimental versus correlational studies.

Most studies in cross-cultural psychology are correlational in nature not properly allowing the researcher to make cause and affect inferences. For example comparisons between cultural groups cannot be called experimental since they contain really correlational information. However, the research reported in the literature often treat observed differences as cause and effect relationships, when a third variable or other factors may be responsible for the results. To use a simple example suppose differences are found between two cultures on the rate and quality of innovation. Judging the results as cause and effect might lead to the conclusion that one culture is better at innovating. However, it may be that essential nutrition in childhood is missing in one culture impairing intellectual development, or perhaps there are climatic or other factors that play a role independent of any overall generalization summarized in “culture”.

Strictly speaking cause and effect conclusions are only justified in experimental studies where control and experimental groups receive equivalent treatments. That is very difficult to achieve in most cross-cultural comparative studies. In discussing the results of correlational studies it is well to remember that these studies are exploratory in nature and don’t directly examine cause and effect. However, correlational studies are useful in exploring relationships and may allow us to develop further hypotheses about the domain of interest. Over time and through replication work we may get to understand more about what it is in the culture that produced the initial correlation. Matsumoto and Yoo (2006) suggested that attributing reasons for cultural differences without specific evidence from experimental work contribute to cultural attribution errors. For example the multiple studies in individualism and collectivism have often been used to draw inferences about specific societies in the Western world and Asia. However, unless these constructs are actually operationalized and measured and found present in the samples surveyed, and unless it can be shown that these constructs are responsible for the observed differences then the attributed difference is not verified.