late the existence general neural mechanisms that are common to both auditory and visual systems that could analyze temporal patterns of interspike intervals to infer the fundamental period from these autocorrelation-like neural representations.

In the experiment, the observers’ responses at F0 were slightly affected (about 10%) by phase (Fig. 13.3), and some responses were seen at multiples of F0 with the random-phase stimuli. (i) This difference may be explained simply by the difference of the waveform. There is a periodic structure in the in-phase condition, but such a periodicity disappeared in the condition of the random phase. (ii) A second possibility is that such a phase effect could be explained by the ACF if there is nonlinear distortion of the stimulus waveforms. The ACF would not identical for in-phase and random-phase stimuli if there is substantial nonlinearity in the visual-brain system.

13.1.1 Conclusions

1.The most frequently perceived flicker rate for complex visual waveforms corresponds to the fundamental frequency, even when there is energy at the fundamental in the power spectrum (the “missing” fundamental is perceived for periodic visual flicker patterns as well as for sounds).

2.Even though the fundamental periodicity is not readily apparent in the waveform plot of the random-phase flicker stimulus, the flicker fundamental is still perceived (again in a manner similar to auditory pitch perception).

3.These phenomena can be explained by assuming a process that detects periodic peaks in the ACF of stimuli for a certain range of fundamental frequencies.

13.2 Spatial Sensations

To understand the mechanism of our visual system, an attempt is made here to characterize the information utilized for texture perception. The spatial ACF analysis of a two-dimensional pattern provides useful measures for representing the three salient spatial sensations of texture: contrast, coarseness, and regularity. The validity of the ACF analysis was examined by use of the calculated factors to describe the subjective scale value of each sensation collected for various kinds of natural textures. When a texture has harmonic structure, for example, the first major peak in the estimated spatial autocorrelation function corresponds to the spatial period of the texture. Results show that these spatial sensations are strongly related to factors extracted from the spatial ACF.

The underlying problem is how does the human visual system characterize salient spatial sensations of texture? Several previous studies have focused on texture properties either through the study of perception or through computational models. However, the relationship between perceptual and computational properties remains unclear. To understand the mechanism of our visual system, it is important to know how the information related to sensations of the textural property may be extracted.

This section contributes to an understanding of the operation of the human visual system, as well as to computational vision.

Historically, the general texture model that was proposed by Julesz (1962) mainly drove approaches. The deficiency of the structural methods is that they are incapable of capturing the randomness-like natural textures. In the decade of the 1970 s, two conflicting prediction models of hierarchical feature detection and multiple spatial frequency channels were discussed (de Valois and de Valois, 1980). Textural features were discussed by first-order and second-order statistics of the image, such as average and variance, or a co-occurrence matrix (Haralick, 1979). Since the 1980 s, the study of texture models has been based on random field statistics (Cross and Jain, 1983; Mao and Jain, 1992) such as the Markov random field and autoregression models. These models can describe texture by using a small number of parameters. These parameters, however, are not based on the mechanism of the human visual system, and therefore they can hardly represent multiple textural property salience to the visual sensations.

More recent texture models are motivated by the psychophysical and physiological findings regarding the early stages of the human visual system. These texture models are based on the psychophysical evidence that two textures are often difficult to discriminate when they produce a similar distribution of responses in a bank of orientation and spatial-frequency selective linear filters (Turner, 1986; Malik and Perona, 1990). In most of the articles (for example, Heeger and Bergen, 1995; Zhu et al., 1998), the texture is parameterized by the first-order statistics, for example, histograms of a set of appropriately chosen multiscale and multiorientation filter outputs. The application of these sets of parameters to the texture synthesis has been successful in generating a new texture that matches the appearance of a given sample. The main disadvantage of this method is the difficulty in synthesizing periodic or structured textures. Another problem that remains to be solved is how to select a set of features to best characterize the images and how to integrate the output of many filters for further processing.

On the other hand, Tamura et al. (1978) measured human discrimination performance with texture features and compared these measures with computational results. From the description in the literature and from observations of the Brodatz (1966) texture album, six features were chosen that correspond with visual perception: contrast, coarseness, regularity, roughness, directionality, and line-likeness. The emphasis was on developing computational analogues of these attributes, but they only modified already developed computational features and combined several features to have a close relationship to a specific property. As a result, this approach failed to clarify how visual properties are related to particular physical properties. Amandasun and King (1989) carried out a similar study. Rao and Lohse (1996) developed a classification method for perceptual texture. Based on psychological similarity judgments, they constructed a three-dimensional space for texture classification. The three orthogonal dimensions were repetitive versus nonrepetitive; high-contrast and nondirectional versus low-contrast and directional; and granular, coarse, and low-complexity versus nongranular, fine, and high-complexity. A similar experiment conducted by Cho et al. (2000) suggested that the dimensionality of

directions. In this study, it was selected as M = N = 256. For computational efficiency, the ACF is computed as an inverse fast fourier transform (FFT) of the image power spectrum. Usually, the ACF is normalized as

M−1 N−1

x=0 y=0

The denominator in Equation (13.2) is the maximum value of the ACF representing the power of the image. Thus, the normalized ACF has the maximum value of 1.0 at the origin, x = y = 0.

The ACF contains the same information as the power density spectrum. We use the ACF here because it allows us to most simply and directly obtain the major factors involved in spatial texture perception. The spatial ACF reflects the distribution of spatial intervals in the image, that is, the magnitudes for each interval length, as opposed to frequency, in spite of the formal interchangeability of two descriptions. Remarkably, the spatial ACF of the pattern has a dominant periodicity that corresponds to the reciprocal of its fundamental frequency. Even when the fundamental frequency component is subtracted from the stripe pattern, due to the missing fundamental, we can still see the fundamental periodicity (Henning et al., 1975). The frequency domain analysis fails to explain this percept, because the fundamental frequency component is absent. Instead, the ACF of this missing fundamental pattern maintains a dominant periodicity at the fundamental. The perceptual mechanism of the pattern is still under discussion (Badcock and Derrington, 1989; Hammett and Smith, 1994), however, it implies that the ACF is effective for detecting the spatial periodicity of patterns. The first autocorrelation model developed to account for the pitch of the missing fundamental phenomenon was therefore originally formulated as a “duplex” model (Licklider, 1951). Licklider’s time-delay neural network architecture was similar in many respects to the Jeffress (1948) model of binaural crosscorrelation that had been proposed 3 years earlier. Licklider used a network of delay lines and coincidence counters arranged along the axes of frequency and delay to compute both a central spectrum and a central global temporal autocorrelation representation.

From the calculated ACF, four factors may be extracted from the ACF in a similar manner as that described in Chapter 5. As shown in Fig. 13.5, the ACF decays from the origin outwards. To simplify the calculation, we only considered two onedimensional ACFs along the x and y axes from the origin. This is because the most natural texture is considered to be isotropic, in which all orientations occur with the same probability. Consequently, the ACF could be assumed to be circular symmetric. Even for the anisotropic texture, at least one of our directional ACFs can hold the periodic structure. The four factors determined are

1.(0,0): the autocorrelation at the origin, x = y = 0.

2.δ1: the displacement of the first maximum peak in the ACF.

3.φ1: the amplitude of the first maximum peak.

ACFs along the x and y axes are extracted from the two-dimensional ACF. Then, the first maximum peak value in both directions is figured out. From the chosen ACF, the displacement and the amplitude of the maximum peak are picked out as δ1 and φ1 (see Fig. 13.5b). The value of effective spatial duration δe is determined by the point at which the normalized spatial ACF decays to 10% of its zero-lag maximal value, as shown in the figure. This is done by exponential fit to normalized ACF peaks above 0.1. When the given ACF has no peaks above 0.1, only the initial decay rate was fitted.

The value of (0) corresponds to the energy of the image. The displacement and the amplitude of the maximum peak (δ1 and φ1) are related to the periodicity of the image. The value of δ1 is a reciprocal of the fundamental frequency, and φ1 is the strength of the harmonic components. The value of δ1 represents the spatial pitch, and the value of φ1 represents the strength of the spatial pitch. The image width (cm) and the distance (cm) calculate the visual angle in degrees.

Materials applied are shown in Fig. 13.6a (Brodatz, 1966, Tamura et al., 1978, Fujii et al., 2003). Adding 12 botanical textures with a high degree of similarity between them and less harmonic structure (Fig. 13.6b), we discuss the spatial sensations in relation to multiple factors extracted from their spatial ACFs (Fujii and Ando, unpublished data).

According to the multiple regression analysis, the scale value (SV) of each spatial sensation obtained by the PCT was approximately described by the spatial factors extracted from the ACF of the spatial signal of gray level, so that

where c1, c2, . . ., c6 are coefficients. As a result of analyses of measured data, coefficients were obtained: c1 = 0.85, c2 = 0.25, c3 = 0.43, c4 = 0.37, c5 = 0.55, and c6 = 0.74. Figure 13.7 shows relationships between the measured SV (vertical axis) and those (horizontal axis) calculated by Equations (13.3) through (13.5). The correlation coefficients between the measured SV and calculated SV are r = 0.91 (contract), 0.88 (coarseness), and 0.74 (regularity).

Thus far, we have clearly shown that the ACF analysis provides useful measures for representing the textural property of contrast, coarseness, and regularity. Multiple regression analysis revealed that perceived contrast is affected by (0) and δ1. The simplest explanation of this observation might be the contrast sensitivity function of the human visual system (Campbell and Robson, 1968). Human spatial contrast sensitivity is best at a mid-spatial frequency range around 2–5 c/deg. Perceived regularity was strongly related to the period δ1and height of the peak φ1. As for the random texture, however, the estimated ACF does not have a periodic structure. In this case, the initial decay rate of the ACF was important to represent the texture coarseness as well as regularity. It seems that there are two distinct mechanisms in

Fig. 13.7 Relationships between estimated and measured scale values for visual contrast, coarseness, and regularity. Scale values were estimated using Equations (13.3)–(13.5). Scale values and correlations between estimated and measured values for (a) contrast (r = 0.91), (b) coarseness (r = 0.74), and (c) regularity (r = 0.88)

the image. That is, the peak frequency is detected in the presence of strong peak, and the mean frequency is detected in their absence. In the context of our analysis, the peak frequency and mean frequency in the spectrum could be comparable with the maximum peak and the initial decay rate of the ACF. For the similar purpose as our study, Liu (1997) measured the strength of perceived regularity by use of spectrum decomposition. The image was decomposed into the deterministic (periodic) and nondeterministic (random) components in the frequency domain.

What we have done here is just to show how the ACF factors can represent texture sensations and to a certain degree how human subjects may be sensitive to the same parameters captured by the ACF. There is convincing evidence that such a

Table 13.1 Autocorrelation representations in temporal and spatial vision. Temporal and spatial sensations in relation to factors extracted from the temporal ACF and the spatial ACF

1X: Factors to describe each sensation.

2X: Attention should be paid due to the fact that factors of φ1 and τe are mutually related.

mechanism also exists for visual processing (Uttal, 1975; Ben-Av and Sagi, 1995). Whatever the mechanism, it is possible that humans do have access to the correlation structure as extracted by the ACF. It is remarkable that the correlation model in vision can be applied both to explanation of temporal and spatial sensations.

Thus, the so-called seamless correlation model may exist in the brain for auditory and visual sensations. Table 13.1 summarize results of the temporal and spatial sensations of the visual field, which are described in relation to the temporal and spatial factors extracted from the ACF analyzed.