Imagine Learning
Imagine Language & Literacy Benchmark

Descriptive Information

Please provide a description of your tool:

The Imagine Language & Literacy Placement and Benchmarking system includes subtests addressing print concepts, phonological awareness, phonics and word recognition, reading comprehension, oral vocabulary, and grammar. Student performance data is provided by subtest. Data collected through the Imagine Language & Literacy Placement and Benchmarking system serve multiple purposes. First, Placement Test results are used to determine developmentally-appropriate starting points for each student across several literacy and language development curricula. Second, Placement and Benchmark Test results provide point-in-time skill-level estimates that are independently useful for Universal Screening. Third, when data are collected for both the Placement and Benchmark Tests (or multiple administrations of the Benchmark Test), differences in performance can be used to estimate scaled student growth. Those data can then be compared across administrations to illuminate changes in students’ reading proficiency levels.

The tool is intended for use with the following grade(s).

Preschool / Pre - kindergarten
Kindergarten
First grade
Second grade
Third grade
Fourth grade
Fifth grade
Sixth grade
Seventh grade
Eighth grade
Ninth grade
Tenth grade
Eleventh grade
Twelfth grade

The tool is intended for use with the following age(s).

0-4 years old
5 years old
6 years old
7 years old
8 years old
9 years old
10 years old
11 years old
12 years old
13 years old
14 years old
15 years old
16 years old
17 years old
18 years old

The tool is intended for use with the following student populations.

Students in general education
Students with disabilities
English language learners

ACADEMIC ONLY: What skills does the tool screen?

Reading

Phonological processing:

RAN
Memory
Awareness
Letter sound correspondence
Phonics
Structural analysis

Word ID

Accuracy
Speed

Nonword

Accuracy
Speed

Spelling

Accuracy
Speed

Passage

Accuracy
Speed

Reading comprehension:

Multiple choice questions
Cloze
Constructed Response
Retell
Maze
Sentence verification
Other (please describe):

Listening comprehension:

Multiple choice questions
Cloze
Constructed Response
Retell
Maze
Sentence verification
Vocabulary
Expressive
Receptive

Mathematics

Global Indicator of Math Competence

Accuracy
Speed
Multiple Choice
Constructed Response

Early Numeracy

Accuracy
Speed
Multiple Choice
Constructed Response

Mathematics Concepts

Accuracy
Speed
Multiple Choice
Constructed Response

Mathematics Computation

Accuracy
Speed
Multiple Choice
Constructed Response

Mathematic Application

Accuracy
Speed
Multiple Choice
Constructed Response

Fractions/Decimals

Accuracy
Speed
Multiple Choice
Constructed Response

Algebra

Accuracy
Speed
Multiple Choice
Constructed Response

Geometry

Accuracy
Speed
Multiple Choice
Constructed Response

Other (please describe):

Please describe specific domain, skills or subtests:

BEHAVIOR ONLY: Which category of behaviors does your tool target?

Internalizing
Externalizing
Internalizing and Externalizing

BEHAVIOR ONLY: Please identify which broad domain(s)/construct(s) are measured by your tool and define each sub-domain or sub-construct.

Acquisition and Cost Information

Administration

BEHAVIOR ONLY: What type of administrator is your tool designed for?

General education teacher
Special education teacher
Parent
Child
External observer
Other

If other, please specify:

What is the administration setting?

Direct observation
Rating scale
Checklist
Performance measure
Questionnaire
Direct: Computerized
One-to-one
Other

If other, please specify:

Does the tool require technology?

Yes

If yes, what technology is required to implement your tool? (Select all that apply)

Computer or tablet
Internet connection
Other technology (please specify)

If your program requires additional technology not listed above, please describe the required technology and the extent to which it is combined with teacher small-group instruction/intervention:

What is the administration context?

Individual
Small group   If small group, n=
Large group   If large group, n=
Computer-administered
Other

If other, please specify:

ACADEMIC ONLY: What are the discontinue rules?

No discontinue rules provided
Basals
Ceilings
Other

If other, please specify:

the test uses subtest level discontinuation criteria, which consists of varying percent correct thresholds to indicate mastery.

Are norms available?

No

Are benchmarks available?

Yes

If yes, how many benchmarks per year?

3

If yes, for which months are benchmarks available?

Administration 1: August-October, Administration 2: December-February, Administration 3: April - June

BEHAVIOR ONLY: Can students be rated concurrently by one administrator?

If yes, how many students can be rated concurrently?

Training & Scoring

Training

Is training for the administrator required?

Yes

Describe the time required for administrator training, if applicable:

Less than 1 hr of training

Please describe the minimum qualifications an administrator must possess.

No minimum qualifications

Are training manuals and materials available?

Yes

Are training manuals/materials field-tested?

Yes

Are training manuals/materials included in cost of tools?

Yes

If No, please describe training costs:

Can users obtain ongoing professional and technical support?

Yes

If Yes, please describe how users can obtain support:

Educators can access Imagine Learning’s customer support services to address any technical issues via telephone or email from 6am-6pm MT, Monday-Friday. Local representatives are also available to provide assistance. Varying support packages are listed under the pricing section.

Scoring

How are scores calculated?

Manually (by hand)
Automatically (computer-scored)
Other

If other, please specify:

Do you provide basis for calculating performance level scores?

No

What is the basis for calculating performance level and percentile scores?

Age norms
Grade norms
Classwide norms
Schoolwide norms
Stanines
Normal curve equivalents

What types of performance level scores are available?

Raw score
Standard score
Percentile score
Grade equivalents
IRT-based score
Age equivalents
Stanines
Normal curve equivalents
Developmental benchmarks
Developmental cut points
Equated
Probability
Lexile score
Error analysis
Composite scores
Subscale/subtest scores
Other

If other, please specify:

Does your tool include decision rules?

Yes

If yes, please describe.

Decision rules for the Imagine literacy Placement test, which is used as Benchmark 1, and all subsequent Benchmark tests is listed below. (see document attachment in reliability section)

Can you provide evidence in support of multiple decision rules?

Yes

If yes, please describe.

Using field test data, student subtest score patterns were observed along the difficulty continuum. The continuum of difficulty determines the subtest administration flow and was established by subject matter expert design. The score threshold for each subtest was determined by its correlated success rate, beyond guessing, on the following subtest. In other words, when a student reaches a subtest score sufficient to indicate they will be able to access the upcoming subtest they are allowed to proceed. If a student does not reach that criteria, they are not permitted to proceed, and the test terminates and issues a score. This same logic was used to determine instructional placement in the Imagine Language & Literacy instructional application. Cut points vary by subtest as documented above.

Please describe the scoring structure. Provide relevant details such as the scoring format, the number of items overall, the number of items per subscale, what the cluster/composite score comprises, and how raw scores are calculated.

Automatic scoring. Literacy scores – Scaled literacy scores are reported. They are calculated from combined subtest performance, subtests vary by student based on adaptive logic. Language scores –Scaled language scores are reported. They are calculated from combined subtest performance, subtests vary by student based on adaptive logic. Subtest scores – Raw scores are reported as the percentage correct from the presented items within each subtest.

Describe the tool’s approach to screening, samples (if applicable), and/or test format, including steps taken to ensure that it is appropriate for use with culturally and linguistically diverse populations and students with disabilities.

Imagine Language & Literacy Benchmark is a computerized adaptive multistage assessment which combines features of traditional assessments and computer adaptive tests. Students are assessed via short subtests that focus on skills that are representative of literacy skills across the elementary reading curriculum. The initial sub-test delivered to students is selected according to grade level. Successful students then attempt subtests that assess more complex, higher-order literacy skills. Unsuccessful students attempt skill subtests that assess prerequisite skills. The adaptive nature of the test ensures that content presented is at an appropriate level for most students. Screening is accomplished by comparing student performance with grade-level standards. If test performance suggests a student cannot successfully engage with appropriate content for their grade level, the student is deemed to be in need of additional intervention/instruction.

Classification Accuracy & Cross-Validation Summary

Grade	Kindergarten	Grade 1	Grade 2	Grade 3	Grade 4	Grade 5	Grade 6
Classification Accuracy Fall
Classification Accuracy Winter
Classification Accuracy Spring

Legend

Convincing evidence

Partially convincing evidence

Unconvincing evidence

Data unavailable

^dDisaggregated data available

Measures of Academic Progress (MAP) Reading Assessment

Classification Accuracy

Describe the criterion (outcome) measure(s) including the degree to which it/they is/are independent from the screening measure.

Rasch Unit (RIT) scale scores from the Measures of Academic Progress (MAP) Reading assessment (Northwest Evaluation Association, 2014), were used as the criterion measure. Performance on this measure was expected to be related to performance on the Imagine Learning Beginning-of-Year Benchmark because it includes similar measures of language and literacy skills for grades K – 6. Specifically, the Imagine Learning Beginning-of-Year Benchmark includes measures of letter recognition, phonemic awareness, word recognition, basic reading vocabulary, sentence cloze, beginning book comprehension, leveled book comprehension, and cloze passage completion. MAP Reading includes measures of word recognition, structure and vocabulary, and reading informational texts. MAP Reading is a widely-used and, like the Imagine Learning Beginning-of-Year Benchmark, is a computer adaptive assessment. MAP Reading also has good evidence of reliability and validity for use as an interim progress monitoring assessment. This formative assessment is often used to identify students who are at-risk or in need of intensive intervention and, although it measures similar constructs, is independent of the Imagine Learning Beginning-of-Year Benchmark.

Describe when screening and criterion measures were administered and provide a justification for why the method(s) you chose (concurrent and/or predictive) is/are appropriate for your tool.

Describe how the classification analyses were performed and cut-points determined. Describe how the cut points align with students at-risk. Please indicate which groups were contrasted in your analyses (e.g., low risk students versus high risk students, low risk students versus moderate risk students).

Classification accuracy analyses were used to calculate the Area Under the Receiver Operating Characteristic Curve through the pROC package (Robin et al., 2011) in RStudio Version 1.0.153. The Imagine Learning Winter 2016 Screening Logit Scores were used as the classifier and were tested against the criterion of Spring 2016 MAP Reading Scores. The logit scores used in the Imagine Learning Beginning-of-Year Benchmark were a continuous indicator of the students’ performance on the screener. Students were identified on the criterion as needing intensive intervention if they scored in the 20th percentile or below on the MAP Reading assessment. Area Under the Curve (AUC) was specified using Youden’s (1950) method to determine cut points on the logit scores for the Imagine Learning Beginning-of-Year Benchmark. This method optimizes the threshold for sensitivity and specificity using the logit score calculated by Winsteps through Rasch measurement. The cut-points determined using Youden’s (1950) method performed as expected. The cut points increased for each grade level, suggesting the test was designed so that students performed better on the test as they advanced through grade levels.

Were the children in the study/studies involved in an intervention in addition to typical classroom instruction between the screening measure and outcome assessment?

No

If yes, please describe the intervention, what children received the intervention, and how they were chosen.

Cross-Validation

Has a cross-validation study been conducted?

No

If yes,

Describe the criterion (outcome) measure(s) including the degree to which it/they is/are independent from the screening measure.

Describe when screening and criterion measures were administered and provide a justification for why the method(s) you chose (concurrent and/or predictive) is/are appropriate for your tool.

Describe how the cross-validation analyses were performed and cut-points determined. Describe how the cut points align with students at-risk. Please indicate which groups were contrasted in your analyses (e.g., low risk students versus high risk students, low risk students versus moderate risk students).

Were the children in the study/studies involved in an intervention in addition to typical classroom instruction between the screening measure and outcome assessment?

If yes, please describe the intervention, what children received the intervention, and how they were chosen.

Classification Accuracy - Winter

Reliability

Grade	Kindergarten	Grade 1	Grade 2	Grade 3	Grade 4	Grade 5	Grade 6
Rating

Legend

Convincing evidence

Partially convincing evidence

Unconvincing evidence

Data unavailable

^dDisaggregated data available

*Offer a justification for each type of reliability reported, given the type and purpose of the tool.

Because the Imagine Learning Beginning-of-Year Benchmark assessment uses a total score as a basis for screening decisions, we followed guidance from the Technical Review Committee (National Center for Intensive Intervention, 2018) and conducted model-based analyses of item quality. A model-based approach also accommodates the computerized adaptive nature of the assessment. Sijtsma (2009) and Bentler (2009) describe statistical model-based internal consistency reliability coefficients based in covariance and correlational approaches to identifying the proportion of variance that can be reliably interpreted as true and not as due to error. Bentler (2003, 2007) proposed using any statistically acceptable structural model that contains additive random errors to estimate model-based internal consistency reliability. Rasch models (Rasch, 1960; Andrich, 1988; Bond & Fox, 2015; Linacre, 1993, 1994, 1996; Wilson, 2005) are widely-used statistically acceptable structural models applying additive random errors to estimate internal consistency reliability. Use of a Rasch model-based approach to reliability is justified in the substantive relation to the fact that education is founded on the idea that future student success can be predicted from performance on tasks similar enough to those encountered in real life to be useful proxies. These tasks are assembled into curricular materials and approached from pedagogical perspectives that typically are not tailored to each individual student but are assumed to apply in a developmentally appropriate way across students as they age and progress in knowledge. Much recent work in education has focused on developing and testing measurement models and calibrating tools as coherent guides to future practice, across the full range of assessment applications, from formative classroom instruction to accountability (Wilson, 2004; Gorin & Mislevy, 2013; NRC, 2006). A simple model of this kind takes the form: ln(pni / (1-pni)) = bn - di which states that the natural logarithm ln of the odds (pni / (1- pni)) is hypothesized to be equal to the difference between the ability b of student n and the difficulty d of item i. Estimates of the ability measures b and the item difficulty calibrations d are expressed in a common unit, which makes it possible to interpret the measures relative to the calibrations in the substantive terms of the items, and vice versa. All measures and calibrations are associated with individually estimated error and model fit statistics. A Rasch measurement model-based approach to reliability (Andrich, 1982; Andrich & Douglas, 1977; Fisher, 1992, 2008; Wright & Masters, 1982) differs, then, from the statistical model-based approaches described by Sijtsma (2009) and Bentler (2009) in the way error is estimated and treated en route to arriving at an estimate of the ratio of true variation to error variation.

*Describe the sample(s), including size and characteristics, for each reliability analysis conducted.

Exhibit 3 demonstrates that the sample for the reliability analysis came from multiple regions, divisions, and states for grades K-5. For students for whom gender was known, approximately half of each grade level was male and half was female. Fewer than ten percent of each grade level was identified as English Language Learners, though ELL status data were missing for the majority of students. Inconsistent information was available for FRL, other SES indicators, disability classification, and first language.

*Describe the analysis procedures for each reported type of reliability.

A Rasch measurement model-based approach was used to assess reliability. This approach estimates error (better termed uncertainty; Fisher & Stenner, 2017; JCGM, 2008) for each individual person and item. The mean square error variance is then subtracted from the total variance to arrive directly at an estimate of the true variance. The ratio of the true standard deviation to the root mean square error then provides a separation ratio, G, that increases in linear error-unit amounts (Wright & Masters, 1982) relative to a separation reliability coefficient formulated in the traditional 0.00-1.00 form (Andrich, 1982). An intuitive expression of measurement model-based reliability is obtained in terms of strata, ranges with centers separated by three errors. Strata are estimated by multiplying the separation ratio, G, by 4, which captures 95% of the variation in the measures; adding 1, allowing for error; and dividing that total by 3, for the number of errors taken as separating the centers of the resulting ranges. Winsteps version 4.00.1 was used for all Rasch analyses (Linacre, 2017). Model fit statistics and Principal Components Analyses of the standardized residuals explicitly focus on the internal consistency of the observed responses, evaluating them in terms of expectations formed on the basis of a simple conception of what is supposed to happen when students respond to test questions. Simulations show that model-based assessments of internal consistency using fit statistics are better able to detect violations of interval measurement’s unidimensionality and invariance requirements than Cronbach’s alpha (Fisher, Elbaum, & Coulter, 2010).

*In the table(s) below, report the results of the reliability analyses described above (e.g., internal consistency or inter-rater reliability coefficients).

Results from other forms of reliability analysis not compatible with above table format:

Manual cites other published reliability studies:

No

Provide citations for additional published studies.

Do you have reliability data that are disaggregated by gender, race/ethnicity, or other subgroups (e.g., English language learners, students with disabilities)?

If yes, fill in data for each subgroup with disaggregated reliability data.

Results from other forms of reliability analysis not compatible with above table format:

Manual cites other published reliability studies:

No

Provide citations for additional published studies.

Validity

Grade	Kindergarten	Grade 1	Grade 2	Grade 3	Grade 4	Grade 5	Grade 6
Rating

Legend

Convincing evidence

Partially convincing evidence

Unconvincing evidence

Data unavailable

^dDisaggregated data available

*Describe each criterion measure used and explain why each measure is appropriate, given the type and purpose of the tool.

Rasch Unit (RIT) scale scores from the Measures of Academic Progress (MAP) Reading assessment (Northwest Evaluation Association, 2014), were used as a criterion measure to demonstrate concurrent and predictive validity of the Imagine Learning Beginning-of-Year Benchmark assessment. Performance on this external measure was expected to be related to performance on the Imagine Learning Beginning-of-Year Benchmark because it includes similar measures of language and literacy skills for grades K – 6. Specifically, the Imagine Learning Beginning-of-Year Benchmark includes measures of letter recognition, phonemic awareness, word recognition, basic reading vocabulary, sentence cloze, beginning book comprehension, leveled book comprehension, and cloze passage completion. MAP Reading includes measures of word recognition, structure and vocabulary, and reading informational texts. MAP Reading is widely-used to identify students who are at-risk or in need of intensive intervention and, like the Imagine Learning Beginning-of-Year Benchmark, is a computer adaptive assessment. MAP Reading also has good evidence of reliability and validity for use as an interim progress monitoring assessment.

*Describe the sample(s), including size and characteristics, for each validity analysis conducted.

See Exhibits 5-7 for sample characteristics for each validity analysis conducted. Across all grade K-5 samples used to examine predictive and concurrent validity, sample sizes ranged 641 to 914; sample sizes for grade 6 ranged from 115 to 159. Multiple regions, divisions, and states were represented for each grade level K-5; the 6th grade sample represented one region, division, and state. The proportion of male and female students across all samples was roughly equal. English Language Learners (ELLs) represented between a quarter and a third of all samples, except for Grade 6 samples in which ELLs represented between 4 and 12 percent of all students. In all samples, most students were Hispanic, with the next most prevalent race/ethnicity categories being White/non-Hispanic and Black/African American. Data indicating eligibility for free/reduced price lunch, student disability status, and services received were not consistently available.

*Describe the analysis procedures for each reported type of validity.

Three bivariate correlation analyses were conducted for each grade level. The first provides evidence of predictive validity using winter 2016 Imagine Learning Beginning-of-Year Benchmark assessment data to predict scores on the spring 2017 MAP Reading assessment data. Two analyses were used to demonstrate concurrent validity by relating Imagine Learning Beginning-of-Year Benchmark assessment data and MAP Reading assessment data (separately, using winter 2016 and spring 2017 data). The Pearson correlation coefficient was computed for each correlation and a 95% confidence interval around each coefficient was calculated.

*In the table below, report the results of the validity analyses described above (e.g., concurrent or predictive validity, evidence based on response processes, evidence based on internal structure, evidence based on relations to other variables, and/or evidence based on consequences of testing), and the criterion measures.

Results from other forms of validity analysis not compatible with above table format:

Manual cites other published reliability studies:

No

Provide citations for additional published studies.

Describe the degree to which the provided data support the validity of the tool.

Using the NCII lower bound threshold of .6 for the confidence intervals around the standardized estimate, analyses showed evidence of concurrent validity for all grade levels. Based on this threshold, evidence of predictive validity was available for first, second, third, and sixth grades. The lower bound of the confidence intervals for predictive validity for fourth and fifth grades were very close to the threshold (.597 for fourth grade and .594 for fifth grade).

Do you have validity data that are disaggregated by gender, race/ethnicity, or other subgroups (e.g., English language learners, students with disabilities)?

No

If yes, fill in data for each subgroup with disaggregated validity data.

Results from other forms of validity analysis not compatible with above table format:

Disaggregated validity analyses are not included because bias analysis (differential item functioning) was used to assess subgroup bias (see Technical Standard 5).

Manual cites other published reliability studies:

No

Provide citations for additional published studies.

Bias Analysis

Have you conducted additional analyses related to the extent to which your tool is or is not biased against subgroups (e.g., race/ethnicity, gender, socioeconomic status, students with disabilities, English language learners)? Examples might include Differential Item Functioning (DIF) or invariance testing in multiple-group confirmatory factor models.

Yes

If yes,

a. Describe the method used to determine the presence or absence of bias:

Differential Item Functioning (DIF) was evaluated for all items on the Imagine Learning Beginning-of-Year Benchmark using focal versus contrast groups differences by means of t-tests. Winsteps version 4.00.1 was used for all analyses (Linacre, 2017). Evaluating DIF requires close attention to what item estimate stability is desired. Lee (1992) finds that student performance across elementary grade levels changes by about a logit per year, with the rate of growth slowing as the upper grades are reached. The present data roughly corresponded with that, as the changes from K to Grade 1, Grade 1 to Grade 2, and Grade 2 to Grade 3 were all a logit or more, and the remaining three were all half a logit or more. Accordingly, as Linacre (1994) points out, items stable to within a logit will be targeted at the correct grade level. Linacre (1994) shows that to have 99% confidence that an item difficulty estimate is stable to within a logit, its error of measurement must be 0.385 or less (since that is the value obtained dividing the desired one logit range by the 2.6 standard deviations needed for 99% confidence). The sample size needed for item estimate errors of 0.385 or less was in the range of 27 to 61, depending on how well targeted the item is at the sample. An item that is too easy or hard, and so is off target, will have a higher error and will need a correspondingly larger sample size to drive down the error to the range needed for stability within a logit. The criterion of interest in the Winsteps table is then the DIF size, which should be less than 1.00 logit for samples of sufficient size.

b. Describe the subgroups for which bias analyses were conducted:

Groups tested included ELL versus non-ELL, and unknown; female versus male, and unknown; and White versus non-White. Exhibit 10 displays the number of students and percentages within each of these categories by grade level. Race/ethnicity categories were collapsed into binary categories (i.e., White and Nonwhite) because of small cell sizes for certain groups.

c. Describe the results of the bias analyses conducted, including data and interpretative statements. Include magnitude of effect (if available) if bias has been identified.

For ELL, with three groups, the hypothesis tested was that the item has the same difficulty as its average difficulty for all groups. In 98.5% of comparisons, DIF sizes of |0.77| or less were obtained. Of the six instances of DIF greater than |0.77| logits, four of them involved Phoneme Segmentation items with only four nonELL students responding. The other two items were two Cloze Passage items (items 107.1 and 107.18); both had 77 ELL students answering, both had DIF sizes of about -1.00 logit, meaning these ELL students correctly answered these two items disproportionately more frequently. However, neither item had a t-statistic greater than 1.96, so neither difference was statistically significant at a p-value less than .05. For gender, also with three groups, the hypothesis tested was again that the item has the same difficulty as its average difficulty for all groups. No DIF sizes over |0.77| logits were found and seven items had DIF sizes greater than or equal to |0.50|. All were Cloze Passage items. Males found four items more difficult than the other two groups did, and females found three items easier. Two items with DIF sizes of 0.66 disadvantaging males were statistically significant at p less than .05. For ethnicity, the hypothesis tested was whether the item has the same difficulty for the two groups. There were 3 items with DIF sizes greater than |0.76|. Two of the three have DIF measure standard errors greater than 0.385 for one of the contrasted groups, and so were not expected to be stable to within a logit with 99% confidence. The other item (Cloze Passage 104.1), however, was 1.57 logits more difficult for white students (CLASS=1) than it was for others. The DIF analysis provides a statistical approach to assessing whether the items retain their relative difficulties across different groups. This kind of invariance can also be investigated more directly, by scaling the items on subsamples of students. A series of analyses of each grade level resulted in seven separate sets of item calibrations. Of the 21 pairs of correlations, 15 were between 0.94 and 0.99, four were between 0.83 and 0.89, and the remaining two (Kindergarten vs 5th and 6th grades) were 0.78 and 0.66. Disattenuated for error (Muchinsky, 1996; Schumacker, 1996), the smallest correlation was 0.75. These results supported the overall stability of the scale across grade levels.