Clinical Analysis of Errors

Note: Orthographic processing has been defined as “the ability to form, store, and access orthographic representations” (Stanovich & West, 1989, p. 404). In simple terms, it involves how we perceive and remember the visual look of words, letters, and numbers. Early orthographic processing tasks typically measure visual perception (e.g., visually matching letters, numbers, and symbols), or a person’s sensitivity to the orthographic regularities in the language (e.g., Which looks like a word? ulpk or klup). Other tasks measure the outcomes of successful orthographic learning, such as fluent word recognition or orthographic choice tasks (e.g., Which is a word? rane or rain) (Castles & Nation, 2006).

The error analysis procedures provide examiners with more specific information about a student’s performance than can be obtained from composite or subtest standard scores or comparisons. Some uses for the error analysis data are described by Kaufman and Kaufman (2014a) as:

•Obtaining a more precise level of diagnostic information than subtest standard scores

•Determining the concentration of skill problems

KTEA™-3 105

(Continued)

Interpretation of Skill Status Based Upon Achievement Level

There are two basic types of error classification methods: item-level error classification and within-item classification. In the item-level error classification, each item is classified according to the process, concept, or skill that it assesses. If an item is incorrect, the error type is automatically assigned. For example, each of the Reading Comprehension and Listening Comprehension items are classified as either literal or inferential. Thus, if an incorrect response is given to Reading Comprehension Form A Item 29, it is automatically classified as a “literal error,” whereas an incorrect response to Item 30 is automatically counted as an “inferential” error. Written Expression items are classified under the following item-level classifications: task, structure, word form, capitalization, and punctuation. Oral Expression items are classified along similar categories: task, structure, and word form. In contrast to item-level classification’s automatic error assignment, within-item classification requires judgment on the part of the examiner to determine the error type. That is, the specific details of the student’s response will lead examiners to select which types of errors were made. For example, on Letter & Word Recognition, examiners must evaluate the incorrect pronunciation of a word to determine if an error was made in categories such as short vowel, long vowel, silent letter, prefix, su x, and more. Multiple error categories may be marked for a single item in the within-item classification. The Don’t Forget box lists the types of error analysis that are used for the KTEA-3 subtests.

106 ESSENTIALS OF KTEA™-3 AND WIAT®-III ASSESSMENT

Don’t Forget

.....................................................................................................................

How to Score “DK” or “NR” Responses for Error Analysis

Scoring “Don’t Know” or “No Response” items, which are both scored 0 points, differs depending on whether item-level or within-item classification methods are used. For item-level classification, include DK or NR responses as an error within the relevant skill categories. For within-item classification, DK or NR responses are not included in the error analysis because no error information is available.

Types of Error Analysis Methods for KTEA-3 Subtests

Subtests with errors categorized according to item-level error classifications are automatically assigned an error category when an item is incorrect. Subtests with errors categorized according to within-item error classification are assigned an error category or categories according to the examiner’s qualitative analysis of the student’s incorrect response.

Because within-item error analysis has a large number of error categories, the grids for classifying and counting errors on these four subtests are too large to be included in the record form. Thus, separate error analysis worksheets are provided on the KTEA-3 Flash Drive for tabulating errors on these subtests.

General Procedures for Using the Error Analysis System

This section outlines the general procedures that must be followed in order to utilize the KTEA-3 error analysis methods. Error analysis may be scored using Q-global or hand scoring methods. As shown in Figure 2.7, after raw scores are entered, Q-global gives a preview of the subtest standard scores to allow the user to decide which subtests to include for error analysis. Subtests with a standard score below 85 are highlighted to indicate that error analysis is recommended. Figure 2.8 displays how within-item errors are entered in Q-global, using the Spelling subtest as an example. Rapid Reference 2.8 lists the procedures for hand scoring error analysis. The Caution box alerts examiners to the importance of utilizing “Items Attempted”

KTEA™-3 107

Figure 2.7 Error Analysis in Q-global: Recommended Subtests

Source: Kaufman Test of Educational Achievement–Third Edition (KTEA-3). Copyright © 2014 NCS Pearson, Inc. Reproduced with permission. All rights reserved.

22222222111111111

Figure 2.8 Within-Item Error Analysis Entry in Q-global

Source: Kaufman Test of Educational Achievement–Third Edition (KTEA-3). Copyright © 2014 NCS Pearson, Inc. Reproduced with permission. All rights reserved.

KTEA™-3 109

(Continued)

calculate this number by counting the number of white boxes in the skill column between item 1 and the “Last Item Administered,” even if the examinee was not administered all of the earliest items.

7.Determine Skill Status. Compare the student’s number of errors with the average number of errors and then circle the appropriate skill status (W for Weakness, A for Average, S for Strength). Use the following procedures:

If the examinee made more than the “Average Number of Errors,” then mark that error category as a Weakness (circle W).

If the examinee made fewer than the “Average Number of Errors,” then mark that error category as a Strength (circle S).

If the number of errors made by the examinee was equal to or within the range of the “Average Number of Errors,” then mark that error category as Average (circle A).

1In the norms tables, the “average” is defined as the number of errors made by the middle 50% of students.

C A U T I O N

........................................................................................................22222222111111111

How to Interpret and Report “Items Attempted”

A student’s skill status is determined by comparing the number of errors made in a category to the normative average. So how is “Items Attempted” used or interpreted? Items Attempted refers to the number of opportunities a student had to demonstrate a specific skill. This number allows the examiner (and others) to interpret error analysis results within the context of the number of opportunities there were to demonstrate an error. The following statement exemplifies how error analysis data may be reported within the context of the number of Items Attempted: “Student A made six short vowel errors in 15 opportunities, whereas the average number of errors made by the reference group was 0.” Without this information, it would be difficult to gauge the severity of the skill deficit, or the frequency with which the error is made.

Error Analysis in Reading Comprehension and Listening Comprehension

Items from Reading Comprehension (with the exception of the early items) and Listening Comprehension are divided into literal and inferential comprehension, as well as narrative and expository comprehension. Literal versus inferential comprehension is an important distinction for describing the level of comprehension demands required to respond to an item correctly. Literal comprehension requires recognizing or recalling ideas, information, or events that are explicitly stated in an oral or written text. In contrast, inferential comprehension requires the generation of new ideas from those stated in the text. Students derive inferences from relating di erent concepts

110 ESSENTIALS OF KTEA™-3 AND WIAT®-III ASSESSMENT

presented in the text or combining information with previously acquired knowledge. Sometimes inferences require students to evaluate the writer’s or speaker’s viewpoint. Literal questions do not require the student to go beyond the viewpoints of the writer or speaker and are usually paraphrased portions of the text.

New to the KTEA-3 is the addition of narrative and expository error categories. These error categories were provided to help examiners determine whether a student tends to have di culty comprehending one passage type over the other, or both. In simple terms, narrative passages tell a story for the purpose of entertainment or enjoyment, whereas the primary purpose of expository passages is to convey information.

Rapid Reference 2.9

.....................................................................................................................

Using the Error Analysis System for Reading Comprehension and Listening Comprehension

Types of Errors Recorded: Each item is classified as either literal or inferential; and either narrative or expository.

Items for Which Errors Are Recorded: Error analysis is based on the final item set used for scoring.

111111111

Error Analysis in Written Expression and Oral Expression

Students are required to communicate their ideas in words for both the Written Expression and Oral Expression subtests. The skills needed to communicate orally are similar to those needed to communicate in written form. However, oral communication is typically more spontaneous and natural, and writing is more deliberate and structured. Di culty in both oral and written communication can be caused by a variety of errors, which the error analysis procedures attempt to quantify.

Rapid Reference 2.10

.....................................................................................................................

Using the Error Analysis System for Written Expression and Oral Expression

KTEA™-3 111

To quantify students’ communication ability, error analysis examines several aspects of communication:

•How well the writing or speech adheres to the task demands to communicate in a comprehensible and functional manner (pragmatics)

•How well-constructed the student’s sentences are (syntax)

•Appropriateness of the word forms (grammar)

•Correct use of words (semantics, scored under Word Form)

•Mechanics (capitalization and punctuation for written expression)

When contrasting the errors made in Written Expression with those from Oral Expression, examiners can di erentiate language structure problems from writing problems. Some examinees have deficits in their basic knowledge or ability to express language, which will likely be evident in errors on both tests. Other examinees with intact oral language skills may exhibit deficits that are specific to written expression.

Errors Analysis in Phonological Processing

The particular aspects of sound awareness and manipulation that a student has and has not mastered will be evident in the errors that are made on the Phonological Processing subtest. The development of skills tapped in this subtest—rhyming, sound matching, blending, segmenting, and deleting sounds—are important precursors to reading. Teachers can use information about deficits in certain areas of phonological

111111111

awareness when trying to teach early reading skills.

Phonological processing involves skills that allow students to manipulate phonemes heard in spoken language. Examinees with poor phonological processing may be unable to identify sounds in words, manipulate sounds in words, perceive a word as a sequence of sounds, or isolate beginning, medial, or final sounds. These subskills are assessed using error analysis.

Rapid Reference 2.11

.....................................................................................................................

Using the Error Analysis System for Phonological Processing

Types of Errors Recorded: Rhyming, sound matching, blending, segmenting, and deleting sounds

Items for Which Errors Are Recorded: Error analysis is based on the items administered within each section.

Error Analysis in Math Concepts & Applications

This subtest contains two related sets of abilities: concepts and applications. Concepts are the basic ideas and relationships on which the system of mathematics is built.

112 ESSENTIALS OF KTEA™-3 AND WIAT®-III ASSESSMENT

Acquisition of math concepts is hierarchical, which requires students to master basic concepts before more advanced concepts can be learned. Applications involve using these concepts and skills to solve actual and hypothetical problems (e.g., reading graphs). If a student has not yet mastered a certain concept or skill, he or she will not be able to apply that concept to solve an actual or hypothetical problem.

Rapid Reference 2.12

.....................................................................................................................

Using the Error Analysis System for Math

Concepts & Application

Types of Errors Recorded: Number concepts, addition, subtraction, multiplication, division, tables and graphs, time and money, geometry, measurement, fractions, decimals and percents, data investigation, algebra, multistep problems, word problems

Items for Which Errors Are Recorded: Error analysis is based on all administered items until the ceiling is reached.

Most Math Concepts & Applications 111111111items are associated with one primary skill, but a few items are associated with two primary skills (i.e., Geometry, Measurement). Many items are associated with one or two primary skills as well as a secondary skill: namely, Multistep Problems or Word Problems. Thus, if an examinee responds incorrectly to an item, then the item is marked as an error in one or two of the primary 13 categories (such as addition, multiplication, division, etc.). However, the final two skill categories (Multistep Problems, Word Problems) may be marked as an additional error along with the one or two primary error categories.

Error Analysis in Math Computation

The error analysis system for Math Computation is unique in that it uses both item-level and within-item classifications. Error analysis for Math Computation provides information about nine skill areas (such as addition, multiplication, fractions) and about 10 specific processes (such as regrouping, converting to common denominators, or placing decimal points) in which the student has made errors. Useful information is gleaned from understanding students’ skill deficits (item-level error analysis), but even more instructionally relevant information may be revealed from the process errors that students make (within-item error analysis). Understanding the reason that a student missed an item (e.g., because they added when they should have subtracted, because they made an error when regrouping) can help determine how and where to provide additional remedial instruction for that student.

KTEA™-3 113

Rapid Reference 2.13

.....................................................................................................................

Using the Error Analysis System for Math Computation

Where Error Analysis Is Recorded (Hand Scoring): Error Analysis Worksheet on Flash Drive

Types of Errors Recorded: Item-level errors: addition, subtraction, multiplication, division, fraction, decimal, exponent or root, algebra

Within-item errors: wrong operation, factor computation, regrouping addition, regrouping subtraction, subtract smaller from larger, add or subtract numerator and denominator, equivalent fraction/common denominator, multiply/divide fraction, mixed number, incorrect sign, uncodable

Items for Which Errors Are Recorded: Error analysis is based on all administered items until the ceiling is reached.

The first two error categories of the specific processes (listed under Within-Item Errors) are Wrong Operation and Fact or Computation Error. These two error categories can be applied to all items. However, if an error is better described by another type of error category, then the Wrong Operation category111111111 should not be marked. For example, if the examinee solves a problem involving division by a fraction by dividing the numerators and denominators instead of inverting and multiplying, then you should mark the Multiply/Divide Fraction category instead of the Wrong Operation category.

Error Analysis in Letter & Word Recognition, Nonsense Word Decoding, and Spelling

The reading of words requires that students connect speech sounds to letter patterns. Three subtests tap this skill in slightly di erent ways. Nonsense Word Decoding assesses a student’s ability to apply decoding and structural analysis skills to nonsense words composed of typically occurring letter patterns. Spelling requires students to relate speech sounds that they hear to letter patterns that they write. Letter & Word Recognition taps a student’s ability to read words with both regular and irregular letter patterns.

The error analysis system for Letter & Word Recognition, Nonsense Word Decoding, and Spelling is made up of categories of predictable skill categories, which generalize across most words, and unpredictable skill categories, which are not generalizable to other words. A separate error category is provided for words with unpredictable patterns for two reasons. First, separating out errors on unpredictable patterns gives a clearer indication of the student’s problem areas in the decoding or spelling of predictable patterns, and secondly, a weakness in the skill category of unpredictable patterns may suggest a deficit in orthographic processing.

KTEA™-3 115

(Continued)

C A U T I O N

........................................................................................................

Di erences Between Skill Categories for

Letter & Word Recognition, Nonsense

Word Decoding, and Spelling

•All three subtests include a category for Consonant Blends, but in Nonsense Word Decoding and Spelling this category is divided22222222111111111 into Initial and Medial/Final consonant blends because these subtests include a large number of instances of each subtype.

•Spelling error analysis does not contain the Wrong Vowel or Misordered Sounds categories. Writing a wrong vowel is categorized as a short or a long vowel error.

•All three subtests include Prefixes and Word Beginnings and Suffixes and Inflections categories. They are merged into one category in Nonsense Word Decoding because that subtest has few prefixes.

•In Spelling, if a student inserts a grapheme (smaller than a syllable) that is not in the sound of the word, count this is a Nonphonetic error.

Note: Adapted from the KTEA-3 Scoring Manual (Kaufman & Kaufman, 2014d, p. 47).

The last category in Letter & Word Recognition, Nonsense Word Decoding, and Spelling is Whole Word Error. Generally, if a student’s response is significantly different from the stimulus word, then it is usually better to mark the item as a Whole Word Error rather than trying to identify numerous specific errors. There are several times when it is best to mark a student’s response in the Whole Word Error category (Kaufman & Kaufman, 2004a, p. 50):

1.The response is unclassifiable, showing little or no correspondence to the stimulus. If the response contains several incorrect parts, mark Whole Word Error instead of trying to identify them. Record UN (unclassifiable) next to the word when recording errors.