Chau Chemometrics From Basics to Wavelet Transform

last element is 0. A risk value for every coefficient, rk , is then computed by

The coefficient that has the minimum rk is selected as the threshold value for the scale j :

It should be noted that the absolute value of the coefficient is used for the thresholding operation.

2. Visually Calibrated Adaptive Smoothing Method. This method is generally called the VISU method. The threshold value is computed

where n is the number of the elements in the coefficient vector. This value can be used for hardor soft-thresholding operation. For methods 1 and 2, the thresholding sometimes is performed only on the coefficients in the index interval [2J + 1, n] (the wavelet or detail coefficients part) where J is the predefined largest decomposition scale. J must be smaller than log2 (N) when the length of the signal is N.

3. Hybrid Thresholding Method. This method is called HYBRID method because it is a soft-thresholding method where in some cases the VISU threshold εA = (2 log nj )1/2 is used, and in other cases SURE threshold

where J is the largest decomposition scale, wj and nj are the coefficients of the scale j and its length, respectively. If e < p, then soft VISU thresholding is performed, if not, the smaller value in εA and εB is used to perform the soft-thresholding operation. In this method, the thresholding is performed on the coefficients of each scale as in the SURE method.

4.Median Absolute Deviation (MAD) Method. The MAD method is a soft-thresholding method. It is also applied to the individual scales. The threshold used for each scale j is the same as in VISU method,

i.e., ε = (2 log n)1/2, but the coefficients must be normalized by wj /sj , where sj = median(|wj |)/0.6745, before the thresholding is performed.

5.MINMAX Thresholding Method. This method finds the optimum threshold for each scale according to the mean-square error between the

Figure 5.14. Plots of a simulated clean chromatogram (a), noisy chromatogram with random noise of SNR = 20 (b), and the denoised results by hard- (c), soft- (d), SURE (e), VISU (f ), HYBRID (g), and MINMAX (h) thresholding methods, respectively.

where V is the simulated chromatogram, t is retention time, n is the number of peaks, and cj , t0,j , and W1/2,j are the concentration, position, and full width at half height of the peak j , respectively. Curve (b) in Figure 5.13 shows a noisy chromatogram generated by adding random noise in curve (a). The SNR calculated by the ratio of the maximum of the signal and the maximum of the noise is 100. Curve (b) in Figure 5.14 is generated in the same way, but the SNR is 20.

Curves (c)--(h) in Figures 5.13 and 5.14 are the denoised results achieved by different thresholding methods. The Symmlet4 (L = 8) filter and the decomposition level J = 5 were adopted in the calculation. Curves

(c)and (d) in Figure 5.13 are obtained, respectively, by hardand softthresholding methods with the threshold ε = 0.003 × [max (w) − min (w)], which is determined by trial and error. In Figure 5.14, curves (c) and

(d)are obtained in the same way as in Figure 5.13, but the threshold ε = 0.015 × [max (w) − min (w)] is adopted. The thresholds for all the other thresholding methods are determined by the abovementioned equations. Soft thresholding is adopted for the HYBRID method, while hard thresholding operation is adopted for the other methods. In order to compare the denoised results with the original simulated signal, RMS values between curves (a) and (c)--(h) are calculated, respectively. These values are 0.0064, 0.0081, 0.0055, 0.0065, 0.0092, and 0.0060 for the results in

Figure 5.13 (SNR = 100), and 0.0288, 0.0325, 0.0287, 0.0289, 0.0355, and 0.0304 for the results in Figure 5.14 (SNR = 20). It can be seen that all the methods can give acceptable results, but the distortion will increase with the noise level.

Computational Details of Example 5.4

1.Generate the noisy signal with 1024 data points; see curve (b), Figure 5.13 using the Gaussian equation.

2.Make a wavelet filter---Symmlet4.

3.Normalize the data to noise level 1.

4.Set resolution level J = 5 (10 − 5).

5.Perform forward WT to obtain the wavelet coefficients.

6.Perform thresholding to the wavelet coefficients to suppress the smaller coefficients, construct the signal by applying inverse WT to the suppressed coefficients, and display the denoised results, respectively, by

6.1.Hard thresholding with manually determined threshold [Fig. 5.13, (c)].

6.2.Soft thresholding with manually determined threshold [Fig. 5.13, (d)].

6.3.SURE (hard) thresholding [Fig. 5.13, (e)].

6.4.HYBRID (soft) thresholding [Fig. 5.13, (f)].

6.5.VISU (hard) tresholding [Fig. 5.13, (g)].

6.6.MINMAX (hard) thresholding [Fig. 5.13, (h)].

Note: The only difference between Figures 5.13 and 5.14 is the SNR of the noisy signal.

From the denoised results curves (c) and (d) in both Figures 5.13 and 5.14, it can be seen that both the hard and soft thresholding by manually determined threshold ε can give satisfactory results. But these results are obtained by trial and error, which is tedious and time-consuming. Among the results of SURE, VISU, HYBRID, and MINMAX, whose thresholds are determined automatically by the abovementioned equations, the SURE method result is the best. It is clear that curves (e) in both Figure 5.13 and 5.14 are clean and the least distorted. Although curves (f) and (g) are also clean enough, there are obvious distortions, especially for the last small peak in the shoulder. From curve (h) in the two figures, it is clear that the MINMAX is not a good method for denoising of the chromatogram, because

there is still noise in the denoised result, especially when the SNR is low in Figure 5.14.

5.2.2.Smoothing

According to the principle of the WT smoothing, the procedure of smoothing with WT can be summarized as follows:

1.Apply a WT to noisy signal fnoisy and obtain the wavelet coefficients vector w.

2.Remove those elements that are thought to represent high-frequency fluctuation in w.

3.Apply the inverse transform to the suppressed w to obtain the smoothed signal fsmoothed.

It can be seen that the only difference between denoising and smoothing is the second step. For denoising, coefficients thought to be small enough are suppressed, but for smoothing, the coefficients representing high-frequency information are removed. According to the principles of WT, the detail coefficients at low scale are generally attributed to high-frequency components.

In order to remove the coefficients corresponding to the high-frequency component, a threshold for cutting off those coefficients must be provided for smoothing. Unfortunately, it is not so easy to determine a frequency threshold as in denoising. Therefore, the threshold value is again determined by trial and error.

As indicated in Figure 5.3, 5.7, and 5.9, it is difficult to determine a cutoff position from the figure. Therefore, an improved algorithm was proposed for DWT calculation. It can be described by

where the hj and gj are obtained by inserting 2j −1 − 1 zeros into the every adjacent element of h and g in Equation (5.1) and (5.2). Subsequently, the length of the filter L will be doubled. By this algorithm, both cj = {cj ,k }

Figure 5.16. Comparison of the simulated clean chromatogram (a) and the smoothed result by WT with the improved algorithm (b).

Computational Details of Example 5.5

1.Generate the noisy signal with 1024 data points [Fig. 5.14 (a),(b)] using the Gaussian equation.

2.Make a wavelet filter---Daubechies4.

3.Set resolution level J = 5.

4.Perform WT to obtain the c and d components with the improved algorithm.

5.Display Figure 5.15.

6.Perform smoothing by replacing the d1 − d4 with zeros and constructing the signal with inverse WT.

7.Display Figure 5.16.

8.Display the RMS between the original signal and the smoothed signal.

We can also propose a method to estimate an approximate scale threshold for deciding the coefficients at which scale should be set to zero using the characteristics of Equations (5.32) and (5.33) and the concept of the Nyquist critical frequency in sampling theory. The Nyquist critical frequency is defined by

where S is sampling time interval. For a signal with the Nyquist critical frequency fc , the frequency of the cj and dj calculated by Equations (5.32)

and (5.33) will match the following inequality:

2−j fc ≤ fdj ≤ 2−j +1fc (5.36)

fcj > 2−j fc

Then, if we take the peak of an analytical signal (e.g., a chromatogram) as a periodic signal, following the definition of the Nyquist critical frequency fc , the frequency of the peak can be defined by

where Wb denotes the width of the peak. Because the aim of the smoothing is to remove high-frequency fluctuation and retain the chromatographic signal, the frequency of the coefficients to be removed, fcut, should be much higher than that of the analytical peak:

where Wth denotes the estimated maximal width of noise, then rearrange the Equation (5.38) and perform a logarithm on both sides, we obtain

We can estimate an approximate scale threshold j th by Equation (5.41); all the detail coefficients at the scale lower than the j th should be set to zero in the reconstruction calculation. The difficulty in using the method is that the parameter Wth must be estimated by experience.

Example 5.6: Data Smoothing of an Experimental Chromatogram.

Figure 5.17 shows an experimental chromatogram measured by an reverse-phase HPLC and postcolumn reaction detection with arsenazo III. The sample is composed of six rare-earth ions (Lu, Yb, Tm, Er, Ho, and

Figure 5.17. Experimental chromatogram with great level noise of a mixed rare-earth solution sample.

Yb). The chromatogram recorded between 1.505 and 11.811 min sampled every 0.005 min is shown. The data length is 2048. The noise is caused by the strong absorption of the postcolumn reaction agent and the pulse of the mobile phase. Therefore, the noise is an oscillation as shown in the enlarged part of the figure. It is impossible to denoise such a chromatogram by other commonly used denoising methods. But we can easily smooth the chromatogram by using the WT smoothing method mentioned above.

At first, we can determine a scale threshold by examination of the enlarged part of the chromatogram. It is easy to find that the maximal width

Figure 5.18. Comparison of the smoothed results of the noisy chromatogram by WT with general (a) and the improved (b) algorithms.