Acadience Reading Pre-K: PELI
PELI Composite Score

Descriptive Information

Please provide a description of your tool:

The Preschool Early Literacy Indicators (PELI®) is an individually administered assessment of essential skillsfor early literacy development in children who are 3 to 5 years of age. The PELI was developed to be used for the following primary purposes: (a) identification of preschool children who may need additional support in acquiring early literacy skills; (b) monitoring progress of children in acquisition of early literacy skills; and (c) evaluating child outcomes as a result of instruction and/or intervention. The PELI Composite Score (PCS) is a single score that provides an overall representation of a child’s early literacy and language skills necessary to be on track for beginning to learn to read. The PCS is the best score to use as an overall predictor of later early literacy success. Children who achieve scores at or above the benchmark on the PCS are likely to make adequate progress in acquiring the critical early literacy skills necessary to be on track for learning to read at the beginning of kindergarten. Conversely, children who score below the benchmark goal or below the cut point for risk are likely to need additional support to make adequate progress in their acquisition of critical early literacy skills.

The tool is intended for use with the following grade(s).

Preschool / Pre - kindergarten
Kindergarten
First grade
Second grade
Third grade
Fourth grade
Fifth grade
Sixth grade
Seventh grade
Eighth grade
Ninth grade
Tenth grade
Eleventh grade
Twelfth grade

The tool is intended for use with the following age(s).

0-4 years old
5 years old
6 years old
7 years old
8 years old
9 years old
10 years old
11 years old
12 years old
13 years old
14 years old
15 years old
16 years old
17 years old
18 years old

The tool is intended for use with the following student populations.

Students in general education
Students with disabilities
English language learners

ACADEMIC ONLY: What skills does the tool screen?

Reading

Phonological processing:

RAN
Memory
Awareness
Letter sound correspondence
Phonics
Structural analysis

Word ID

Accuracy
Speed

Nonword

Accuracy
Speed

Spelling

Accuracy
Speed

Passage

Accuracy
Speed

Reading comprehension:

Multiple choice questions
Cloze
Constructed Response
Retell
Maze
Sentence verification
Other (please describe):

Listening comprehension:

Multiple choice questions
Cloze
Constructed Response
Retell
Maze
Sentence verification
Vocabulary
Expressive
Receptive

Mathematics

Global Indicator of Math Competence

Accuracy
Speed
Multiple Choice
Constructed Response

Early Numeracy

Accuracy
Speed
Multiple Choice
Constructed Response

Mathematics Concepts

Accuracy
Speed
Multiple Choice
Constructed Response

Mathematics Computation

Accuracy
Speed
Multiple Choice
Constructed Response

Mathematic Application

Accuracy
Speed
Multiple Choice
Constructed Response

Fractions/Decimals

Accuracy
Speed
Multiple Choice
Constructed Response

Algebra

Accuracy
Speed
Multiple Choice
Constructed Response

Geometry

Accuracy
Speed
Multiple Choice
Constructed Response

Other (please describe):

Please describe specific domain, skills or subtests:

Letter naming is also assessed as a means of indexing alphabet knowledge. Additionally, this tool also screens for Vocabulary-Oral Language. This subtest targets a child’s knowledge of and ability to use words in simple sentences to convey meaning, and is assessed by looking at a child's ability to name pictures and tell about them.

BEHAVIOR ONLY: Which category of behaviors does your tool target?

Internalizing
Externalizing
Internalizing and Externalizing

BEHAVIOR ONLY: Please identify which broad domain(s)/construct(s) are measured by your tool and define each sub-domain or sub-construct.

Acquisition and Cost Information

Administration

BEHAVIOR ONLY: What type of administrator is your tool designed for?

General education teacher
Special education teacher
Parent
Child
External observer
Other

If other, please specify:

What is the administration setting?

Direct observation
Rating scale
Checklist
Performance measure
Questionnaire
Direct: Computerized
One-to-one
Other

If other, please specify:

Does the tool require technology?

No

If yes, what technology is required to implement your tool? (Select all that apply)

Computer or tablet
Internet connection
Other technology (please specify)

If your program requires additional technology not listed above, please describe the required technology and the extent to which it is combined with teacher small-group instruction/intervention:

What is the administration context?

Individual
Small group   If small group, n=
Large group   If large group, n=
Computer-administered
Other

If other, please specify:

ACADEMIC ONLY: What are the discontinue rules?

No discontinue rules provided
Basals
Ceilings
Other

If other, please specify:

PELI discontinue rules vary for each subtest. For the Alphabet Knowledge subtest, administrators are told to discontinue if a child misses three consecutive prompted letters. For Vocabulary-Oral Language, administrators should discontinue the Picture Naming section if the child does not respond or says, "I don't know" for the first three words. Additionally, the administrator should discontinue the Tell About section if the child does not respond or says, "I don't know" for the first three words. For the Phonological Awareness subtest, there is an optional discontinue rule in which the administrator can choose to discontinue if a 3-4 year old child does not give a correct response on the first five items. There is no discontinue rule for the Comprehension subtest.

Are norms available?

Yes

Are benchmarks available?

Yes

If yes, how many benchmarks per year?

3

If yes, for which months are benchmarks available?

Beginning of year (months 1 - 3 of school year), middle of year (months 4 - 6 of the school year), and end of year (months 7 - 9 of the school year).

BEHAVIOR ONLY: Can students be rated concurrently by one administrator?

If yes, how many students can be rated concurrently?

Training & Scoring

Training

Is training for the administrator required?

Yes

Describe the time required for administrator training, if applicable:

Administrator training for Acadience Reading Pre-K: PELI uses a combination of video modules, practice activities, and knowledge check assessments. Overall, the complete PELI Essential Workshop provides over four hours of in-depth training content focused on the administration and scoring of this assessment. Training for the PELI Composite Score sub tool requires approximately four to seven hours to complete.

Please describe the minimum qualifications an administrator must possess.

Administrator must have adequate training on the administration and scoring of the assessments.

No minimum qualifications

Are training manuals and materials available?

Yes

Are training manuals/materials field-tested?

Yes

Are training manuals/materials included in cost of tools?

No

If No, please describe training costs:

The Acadience Reading Pre-K: PELI Assessment Manual is included with test materials. In addition to administrator support provided within the Acadience Reading Pre-K: PELI Assessment Manual, Acadience Learning offers a variety of training options to meet different needs and at different price points. Training options include asynchronous training, and synchronous training via live online training, and onsite training (hiring a trainer to come out to the school or district). Acadience Learning staff can work with schools, LEAs, regional agencies, and SEAs to develop customized training plans to meet their unique needs. We also have an Acadience Mentor program, where individual participants or small groups of participants can become Acadience Reading Pre-K: PELI Mentors. Mentors receive access to our official training materials, which they can use to train others in their school or district. For an individual teacher subscription to the online Acadience Reading Pre-K: PELI Essential Workshop, the cost is $129. Please note: Other training options may cost more or less depending on the circumstances and the number of attendees.

Can users obtain ongoing professional and technical support?

Yes

If Yes, please describe how users can obtain support:

Acadience Learning: Customer support is available from 8:00am to 5:00pm PT, Monday through Friday by phone, email, or through Acadience Learning's website; Voyager Sopris: Customer support is available 8:00am to 6:00pm CT, Monday through Friday by phone, email, or through the Voyager Sopris website.

Scoring

How are scores calculated?

Manually (by hand)
Automatically (computer-scored)
Other

If other, please specify:

Do you provide basis for calculating performance level scores?

Yes

What is the basis for calculating performance level and percentile scores?

Age norms
Grade norms
Classwide norms
Schoolwide norms
Stanines
Normal curve equivalents

What types of performance level scores are available?

Raw score
Standard score
Percentile score
Grade equivalents
IRT-based score
Age equivalents
Stanines
Normal curve equivalents
Developmental benchmarks
Developmental cut points
Equated
Probability
Lexile score
Error analysis
Composite scores
Subscale/subtest scores
Other

If other, please specify:

Does your tool include decision rules?

Yes

If yes, please describe.

The development of PELI benchmark goals and studies evaluating their decision utility occurred in a series of studies over a two-year period. The first study included 274 three- to four-year-old children and 2,472 four- to five-year-old children from 217 preschool classrooms in 37 early childhood programs in nine states representing all four census regions of the United States. In the second study, 3,233 children participated from 106 schools located in 15 states representing all census regions of the United States. PELI benchmark goals and cut points for risk were derived by examining the predictive utility of a score on the PELI at a particular point in time, compared with later PELI measures and to criterion measures that served as outcomes. Our fundamental logic for developing the benchmark goals was to begin with an external outcome goal and work backward following systematic step-by-step procedures. We started by determining a level of performance representing adequate early literacy and language skills on each outcome measure at the end of the year. We used the benchmark goal for the beginning-of-kindergarten RCS as our external outcome goal for early literacy skills and the 40th percentile on the PPVT-4 as our external outcome goal for language skills. Next, we examined the predictive utility of the end-of-year PCS for 4- to 5-year-olds with respect to the end-ofyear external outcome goals (i.e., at or above benchmark on the RCS, 40th percentile on the PPVT-4) and used this data to specify a benchmark goal for the PCS. The primary specification for the PELI benchmark goals was to establish a level of skill where children scoring at or above the benchmark had a favorable probability (above 80%) of achieving subsequent literacy outcomes. Then, using the PCS end-of-year benchmark goal as an internal goal, we established the benchmark goals for the middle-of-year PCS. Finally, we established the benchmark goals on the beginning-of-year PCS using the middle-of-year PCS as an internal goal. Once the benchmark goals were established for the PCS, they were used to establish the benchmark goals for each individual PELI subtest using the same step-by-step procedures. The same step-by-step procedure was used for determining PELI goals for 3- to 4-year-olds using the 4- to 5-year-old beginning-of-year PCS as the starting point. Cut points for risk were derived using a similar step-by-step procedure. The primary specification for the PELI cut points for risk was to establish a level of skill where children scoring well below the benchmark had a low probability (less than 20%) of achieving subsequent literacy outcomes. In addition to the primary specifications for the benchmark goals and cut points for risk, an important secondary consideration was based on the logistic regression predicting the probability of scoring at or above the benchmark on the outcome measure based on a child’s score on the PELI. For all children with scores in the “At or Above Benchmark” range on the PELI, the overall probability of achieving subsequent early literacy goals may be greater than 80%, but for children with scores at the high end of the range, the probability is higher and for children at the low end of the range, the probability is lower. The logistic regression analysis was used to estimate the probability of achieving subsequent early literacy and language goals for children who obtained the exact benchmark score. We attempted to keep the predicted probabilities for children who obtained the exact benchmark score at 60% or higher. Diagnostic efficiency of the PELI benchmark goals and cut points for risk also was evaluated using receiveroperator characteristic (ROC) curves for the PCS, the PLI, and each subtest of the PELI with subsequent PELI assessments and with each of the outcome measures. Additional indices were calculated to provide multiple perspectives on the effectiveness of the PELI scores, including sensitivity, specificity, positive and negative predictive power, and percent accurate classification. In early childhood, we are operating from a prevention perspective. As such, our primary consideration in developing PELI benchmark goals was to identify a level of skill that is predictive of success so that we can set our goals and monitor progress toward those outcomes. Further details and analyses can be found in Chapter 8 of the PELI Assessment Manual. Research on the technical adequacy and decision-making utility of the PELI is ongoing with different samples of children and statistical procedures, and using different external outcome measures. Ongoing research studies will provide additional evidence regarding the technical adequacy of the PELI as well as provide guidance for future revisions of the PELI. Current technical reports on the PELI are available from Acadience Learning.

Can you provide evidence in support of multiple decision rules?

Yes

If yes, please describe.

Research evidence supporting the use of PELI measures for benchmark assessment three times per year is found in the following documents: •PELI Assessment Manual •Acadience Reading Pre-K: PELI Benchmark Goals and Composite Score •Kaminski, R., Abbott, M., Bravo Aguayo, K., Latimer, R., & Good, R. (2014). The preschool early literacy indicators: Validity and benchmark goals. Topics in Early Childhood Special Education, 34(2), 71-82.

Please describe the scoring structure. Provide relevant details such as the scoring format, the number of items overall, the number of items per subscale, what the cluster/composite score comprises, and how raw scores are calculated.

The PELI Composite Score (PCS) is a single score that provides an overall representation of a child’s early literacy and language skills necessary to be on track for beginning to learn to read. The PCS is a derived score that is a combination of the individual PELI subtests. To ensure that the PCS reflects each subtest equally, an improper linear model, with weights based on the standard deviations of each measure, was used. The weights are applied to the subtest scores in the following formula: PCS = 2*AK + 4*Comp + 4*PA + 3*V-OL (where AK is Alphabet Knowledge, Comp is Comprehension, PA is Phonological Awareness, and V-OL is Vocabulary-Oral Language) For the Alphabet Knowledge subtest, a child receives 1 point for each letter correctly named. There are 26 possible items in this section, making the raw score range 0-26. The Vocabulary-Oral Language subtest is broken down into two parts. In part one, Picture Naming, the child receives 1 point for each picture that is named correctly. There are 10 items for this section. In part two, Tell About, the child can receive up to 5 points based on the quality of their response. There are 5 items for this section, making the total number of items for this subtest 15 and the raw score scale between 0-35. Detailed scoring directions and examples are shown in the PELI Assessment Manual. The Comprehension subtest is also broken down into two parts. In part one, Comprehension Questions, all questions are scored on a 3-point scale, with a score of 2 being correct, a score of 1 being partially correct, and a score of 0 being incorrect. There are 9 items in this section. In part two, Shared Retell, all items are scored as 1 or 0, with a score of 1 being correct and a score of 0 being incorrect. There are 5 items in this section, making the total number of items for this subtest 14 and the raw score scale between 0-23. The PELI Assessment Manual provides a rubric and detailed examples to aid administrators in scoring this subtest. Lastly, the Phonological Awareness subtest, broken up into two parts, has 10 test items total. In the scoring rule for part 1, Word Parts, a child receives 1 point for each correct part of the word they say correctly. There are 5 items for this section. For part 2, a child receives 2 points for saying the correct first phoneme and 1 point for saying the correct first sounds (i.e., consonant blend, consonant plus vowel, or consonant blend plus vowel). There are also five items for this section.

Describe the tool’s approach to screening, samples (if applicable), and/or test format, including steps taken to ensure that it is appropriate for use with culturally and linguistically diverse populations and students with disabilities.

We recommend that the PELI be used within an Outcomes-Driven Model of educational decision-making. The Outcomes-Driven Model is a data-based decision-making model in which data are used to make decisions that promote the acquisition of early literacy skills and enhance the school readiness of all children. Within the Outcomes-Driven Model, the PELI provides data that help teachers match the amount and type of instructional support with needs of individual children. The Outcomes-Driven Model consists of five decision-making steps. Step 1. Identify need for support. This process, also known as universal screening, occurs during benchmark assessment. The purpose is to identify those children who may need additional instructional support to meet end-of-preschool early literacy goals. At a systems level, benchmark assessment also provides data regarding the performance of all children in the program with respect to the benchmark goals. Step 2. Validate need for support. The purpose of this step is to be reasonably confident in your decision that a child needs or does not need additional instructional support. Before making a decision to provide instructional support, it is important to consider additional information and knowledge about a child to validate the score. Other sources of information that may be used to validate need for support include data from other assessments, classroom observations, anecdotal records, parental reports, and/or work samples. Step 3. Plan and implement support. Our research indicates that most children who meet the benchmark goals are likely to make adequate progress when they receive instruction in a research-based core curriculum in early literacy. Children whose scores are below the benchmark are likely to need additional instruction or intervention in the skill areas in which they are having difficulty. PELI data can be used to determine specific skill areas in which a child needs additional support. Step 4. Evaluate effectiveness of support. Children who are at risk for future difficulties in learning to read should be assessed more frequently to ensure that the instruction they are receiving is effective. PELI interim progress monitoring books may be used between the regular benchmark periods to assess all critical early literacy areas. Additionally, PELI Quick Checks may be used to efficiently monitor the progress of those children who are receiving targeted instructional support in a single skill area. Monitoring with Quick Checks may occur once a month, once every two weeks, or as often as once per week. In general, children who are furthest behind and need the most intensive support should be progress monitored more frequently to allow for timely decisions regarding the effectiveness of the intervention and whether or not to make modifications to the intervention plan. Step 5. Review outcomes. By looking at benchmark assessment data for all children at the end of the year, programs serving preschoolers can ensure that the core curriculum and supplemental interventions are working for all children. The preschool program can use the end-of-year PELI data at a systems level to identify areas of instructional support that may need improvement. The Outcomes-Driven Model is depicted in Figure 1.4. The numbers in the figure correspond to the steps of the model. The model consists of two recursive processes. The benchmark assessment process occurs from beginning-of-year to middle-of-year, and repeats from middle-of-year to end-of-year. Steps 1, 2, and 5 are a part of the benchmark assessment process. The progress monitoring process, consisting of steps 3 and 4, is ongoing and repeats as needed throughout the year. The PELI Assessment Manual also provides a breakdown regarding how administrators can support diverse populations including English language learners, children from culturally diverse backgrounds, children with special needs, and children out of preschool age range. These directions include information about approved and unapproved accommodations and modifications, and examples of how and when an administrator might want to use unapproved or modified assessment procedures (see PELI Assessment Manual, pp.15-21).

Classification Accuracy & Cross-Validation Summary

Grade	Pre-Kindergarten 4/5-Year-Olds
Classification Accuracy Fall
Classification Accuracy Winter
Classification Accuracy Spring

Legend

Convincing evidence

Partially convincing evidence

Unconvincing evidence

Data unavailable

^dDisaggregated data available

Acadience Reading K-6 Well Below Benchmark

Classification Accuracy

Describe the criterion (outcome) measure(s) including the degree to which it/they is/are independent from the screening measure.

Describe when screening and criterion measures were administered and provide a justification for why the method(s) you chose (concurrent and/or predictive) is/are appropriate for your tool.

Describe how the classification analyses were performed and cut-points determined. Describe how the cut points align with students at-risk. Please indicate which groups were contrasted in your analyses (e.g., low risk students versus high risk students, low risk students versus moderate risk students).

Were the children in the study/studies involved in an intervention in addition to typical classroom instruction between the screening measure and outcome assessment?

If yes, please describe the intervention, what children received the intervention, and how they were chosen.

Cross-Validation

Has a cross-validation study been conducted?

No

If yes,

Describe the criterion (outcome) measure(s) including the degree to which it/they is/are independent from the screening measure.

Describe when screening and criterion measures were administered and provide a justification for why the method(s) you chose (concurrent and/or predictive) is/are appropriate for your tool.

Describe how the cross-validation analyses were performed and cut-points determined. Describe how the cut points align with students at-risk. Please indicate which groups were contrasted in your analyses (e.g., low risk students versus high risk students, low risk students versus moderate risk students).

Were the children in the study/studies involved in an intervention in addition to typical classroom instruction between the screening measure and outcome assessment?

If yes, please describe the intervention, what children received the intervention, and how they were chosen.

Classification Accuracy - Fall

Classification Accuracy - Winter

Classification Accuracy - Spring

Reliability

Grade	Pre-Kindergarten 4/5-Year-Olds
Rating

Legend

Convincing evidence

Partially convincing evidence

Unconvincing evidence

Data unavailable

^dDisaggregated data available

*Offer a justification for each type of reliability reported, given the type and purpose of the tool.

Reliability refers to the relative stability with which a test measures the same skills across minor differences in conditions. Three types of reliability are reported in the table below, alternate form reliability, alpha, and inter-rater reliability. Alternate form reliability is the correlation between different measures of the same early literacy skills. The coefficient reported is the average correlation among three forms of the measure. Coefficient alpha is a measure of reliability that is widely used in education research and represents the proportion of true score to total variance. Alpha incorporates information about the average inter-test correlation as well as the number of tests.

*Describe the sample(s), including size and characteristics, for each reliability analysis conducted.

The sample used for reliability assessment comes from a larger data set with 3,316 students across multiple grades. The coefficient reported is the median alternate form correlation from that dataset.

*Describe the analysis procedures for each reported type of reliability.

Alternate form reliability is reported as the average correlation among three alternate forms of the same test. High alternate form reliability coefficients suggest that these multiple forms are measuring the same construct. Coefficient alpha treats each of the three tests as separate indicators and is calculated using the alternate form reliability, where the number of tests is equal to three.

*In the table(s) below, report the results of the reliability analyses described above (e.g., internal consistency or inter-rater reliability coefficients).

Results from other forms of reliability analysis not compatible with above table format:

Manual cites other published reliability studies:

No

Provide citations for additional published studies.

Do you have reliability data that are disaggregated by gender, race/ethnicity, or other subgroups (e.g., English language learners, students with disabilities)?

No

If yes, fill in data for each subgroup with disaggregated reliability data.

Results from other forms of reliability analysis not compatible with above table format:

Manual cites other published reliability studies:

No

Provide citations for additional published studies.

Validity

Grade	Pre-Kindergarten 4/5-Year-Olds
Rating

Legend

Convincing evidence

Partially convincing evidence

Unconvincing evidence

Data unavailable

^dDisaggregated data available

*Describe each criterion measure used and explain why each measure is appropriate, given the type and purpose of the tool.

The criterion used to address the validity of the PELI was the Acadience Reading K–6 Composite Score (RCS) for benchmarking at the beginning of kindergarten. Acadience Reading K–6 assesses foundational reading skills that are appropriate for kindergarten students. The beginning of year (BOY) Kindergarten RCS consists of two measures, Letter Naming Fluency and First Sound Fluency. The PELI assesses emergent literacy skills in a storybook based approach that is untimed (i.e., not fluency-based) and is designed to be developmentally appropriate for preschoolers. The PELI Composite Score (PCS) consists of scores from PELI subtests of alphabet knowledge, vocabulary and oral language, listening comprehension, and phonological awareness. The measures are given to students are different ages (preschool-age children vs Kindergarteners). While both assessments are published by Acadience Learning, Acadience Reading and the PELI share no subtests, measures, or items, so the overlap in item samples or method variance is nonexistent.

*Describe the sample(s), including size and characteristics, for each validity analysis conducted.

The dataset used for validity includes 1,456 students with data collected from 55 districts within 16 states representing seven census regions.

*Describe the analysis procedures for each reported type of validity.

Predictive validity is the correlation between the PELI Composite Score at the middle of the year and the Reading Composite Score score at the end of the school year. This coefficient represents the extent to which the PELI Composite Score can predict later reading outcomes. Concurrent validity is the correlation between the PELI Composite Score and the RCS measure both at the end of the year. This coefficient represents the extent to which the PELI Composite Score is related to important reading outcomes.

*In the table below, report the results of the validity analyses described above (e.g., concurrent or predictive validity, evidence based on response processes, evidence based on internal structure, evidence based on relations to other variables, and/or evidence based on consequences of testing), and the criterion measures.

Results from other forms of validity analysis not compatible with above table format:

Manual cites other published reliability studies:

No

Provide citations for additional published studies.

Describe the degree to which the provided data support the validity of the tool.

The Acadience Reading Composite shares no overlapping items or measures with PELI but is still a valid measure of early literacy skills, the high correlation between the two measures suggests that the PELI Composite Score is also a valid measure of literacy skills, but one designed for assessment in Pre-K students

Do you have validity data that are disaggregated by gender, race/ethnicity, or other subgroups (e.g., English language learners, students with disabilities)?

No

If yes, fill in data for each subgroup with disaggregated validity data.

Results from other forms of validity analysis not compatible with above table format:

Manual cites other published reliability studies:

No

Provide citations for additional published studies.

Bias Analysis

Have you conducted additional analyses related to the extent to which your tool is or is not biased against subgroups (e.g., race/ethnicity, gender, socioeconomic status, students with disabilities, English language learners)? Examples might include Differential Item Functioning (DIF) or invariance testing in multiple-group confirmatory factor models.

Yes

If yes,

a. Describe the method used to determine the presence or absence of bias:

Bias was assessed using the Cleary of predictor-criterion regression. This involved regression PELI scores at the end of the year onto the PELI score at the beginning and middle of year. This process involved first regressing the end of year onto the earlier PELI score. Step two examined whether group membership predicted end of year PELI score independently of the concurrent PELI, then finally the interaction between subgroups and earlier PELI score. Bias was quantified using the change in R-squared as the effect size. The sample used for this analysis was 22,240 from 11 states. Because such a large sample size was used, trivially small effects can be significant, so inference was made using the magnitude of the effect size.

b. Describe the subgroups for which bias analyses were conducted:

Bias analyses were conducted for the following group comparisons: Gender Ethnicity ELL Status State

c. Describe the results of the bias analyses conducted, including data and interpretative statements. Include magnitude of effect (if available) if bias has been identified.

A consistent pattern emerged from the results of the analysis suggesting a lack of bias in PELI composite scores. Overall, while the Cleary model may have been significant, the model coefficients showed a small effect. Overall, subgroup member accounted for at most 1% additional variance explained, whether as a main effect or as an interaction. This result held whether the subgroup analysis focused on comparing gender, ethnicity, ELL status, or state that the student was assessed in.