Appendix C: The DIALANG scales

items, statements, etc. is determined statistically and a scale is constructed of them.) The calibration was based on a sample of 304 subjects (complete test design) who also took a number of DIALANG tests in Finnish. The SA-statements were presented to them either in Swedish (for 250 subjects whose mother tongue was Swedish) or in English. In addition, most subjects could consult the Finnish language version of the statements.2 The data was analysed with the OPLM programme (Verhelst et al. 1985; Verhelst and

Glass 1995).3 The results of the analysis were very good: over 90% of the statements could be scaled (i.e. they ‘fitted’ the statistical model used). The three self-assessment scales which were constructed on the basis of the calibration of the statements were very homogeneous, as indicated by the high reliability indices (Cronbach’s alpha): .91 for reading, .93 for listening and .94 for writing.4

Similar calibration studies will be carried out when the other 13 languages are piloted, following the approach developed by the Data Analysis Group. They will show to what extent the excellent results of the first study can be replicated and whether there is any tendency for some statements to be consistently better than the others, for self-assessment purposes.

Although the first calibration study is only one study, it is important to note that it tells about the quality of more than one language version of the SA statements in DIALANG. This is because most of the learners studied could choose any, even all, of the three versions (Swedish, English or Finnish) when completing the self-assessment part, although most of them probably relied on the Swedish one. Because of the careful translation procedure, we can safely assume that the SA statements are largely equivalent across the languages – an assumption which will obviously be tested as part of the other calibration studies.

Additional evidence for the quality of the DIALANG self-assessment scales – and for the CEF scales – was obtained by Dr Kaftandjieva by correlating the difficulty values of the statements in this study with the values for the same statements obtained by North (1996/2000) in a different context. The correlation was found to be very high (.83), or even .897, if one strangely behaving statement is excluded.

Document C1 presents the 107 self-assessment statements for reading, listening and writing which survived the calibration study based on Finnish data. The statements in each table are ordered in terms of difficulty from the easiest to the hardest. Statements which were not taken from the Framework are in italics.

Other DIALANG scales based on the Common European Framework

In addition to the self-assessment statements, DIALANG uses two sets of descriptive scales which are based on the CEF. The scales concern reading, writing and listening:

2The study was conducted in the Centre for Applied Language Studies at the University of Jyväskylä, which was the Coordinating Centre of the Project in 1996–1999, by the Working Group for Data Analysis consisting of Fellyanka Kaftandjieva (chair), Norman Verhelst, Sauli Takala, John de Jong, and Timo Törmäkangas. The Coordinating Centre in DIALANG Phase 2 is Freie Universität Berlin.

3OPLM is an extension of the Rasch model, which allows items to differ in their discrimination. The difference between it and the two-parameter model is that discrimination parameters are not estimated but inputted as known constants.

4The global data-model fit was also quite good (p=.26) when the statements were calibrated together. The statistical

fit for skill-based calibration was also good (p=.10 for Reading, .84 for Writing and .78 for Listening).

229

Appendix C: The DIALANG scales

•the more concise version accompanies the test score,

•the more extensive version is part of Advisory Feedback.

Concise scales

DIALANG uses the concise overall scales for reading, writing and listening to report scores on the DIALANG system. When learners get feedback on their performance, they are given a result on the CEF scale, A1 to C2, and the meaning of this score is described using these reporting scales. These were validated in the DIALANG context by asking 12 expert judges to assign each statement to one of six levels. These overall reporting scales were then used by the expert judges to assign each item in the DIALANG tests of Finnish to a CEF level. The scale is based on Table 2 of the CEF; the descriptions were slightly modified in the same way as SA statements. These scales are presented in Document C2.

Advisory feedback

The Advisory Feedback section of the assessment system uses scales which contain more extensive descriptions of proficiency in reading, writing and listening. The section provides the users with more detailed accounts of what learners can typically do with the language at each of the skill levels. The learners can also compare the description for a particular level with the descriptions for adjacent levels. These more detailed scales are also based on the scales on Table 2 in the CEF, but the descriptors were elaborated further with the help of other sections of the CEF and also other sources. These scales are presented in Document C3.

Readers interested in the results of the empirical studies reported here will find more detailed information about them in Takala and Kaftandjieva (forthcoming); for further information about the system in general and the feedback it provides, consult Huhta, Luoma, Oscarson, Sajavaara, Takala and Teasdale (forthcoming).

References

Huhta, A., S. Luoma, M. Oscarson, K. Sajavaara, S. Takala, and A. Teasdale (forthcoming). DIALANG – A Diagnostic Language Assessment System for Learners. In J.C. Alderson (ed.) Case Studies of the Use of the Common European Framework. Council of Europe.

North, B. (1996/2000). The Development of a Common Framework Scale of Language Proficiency Based on a Theory of Measurement. PhD thesis. Thames Valley University. Reprinted 2000: New York, Peter Lang.

Takala, S. and F. Kaftandjieva (forthcoming). Council of Europe Scales of Language Proficiency: A Validation Study. In J.C. Alderson (ed.) Case Studies of the Use of the Common European Framework. Council of Europe.

Verhelst, N., C. Glass and H. Verstralen (1985). One-Parameter Logistic Model: OPLM. Arnhem: CITO.

Verhelst, N. and C. Glass (1995). The One-Parameter Logistic Model. In G. Fisher and I. Molenaar (eds.) Rasch Models: Foundations, Recent Developments and Applications. New York: Springer-Verlag. 215–237.

230