IXL LevelUp™ Diagnostic for Math
Mathematics

Descriptive Information

Please provide a description of your tool:

The IXL LevelUp Diagnostic for Math is a brief and comprehensive assessment designed to determine progress toward standards for students in pre-kindergarten through twelfth grade. Districts and schools may use this assessment to identify students who are falling behind and intervene as early as possible. After students complete the diagnostic, administrators and teachers have access to key data for planning instruction, including overall and strand-level grade scores and scaled scores derived from IRT-based scores, nationally normed percentiles, and overall and strand-based achievement levels.

The tool is intended for use with the following grade(s).

Preschool / Pre - kindergarten
Kindergarten
First grade
Second grade
Third grade
Fourth grade
Fifth grade
Sixth grade
Seventh grade
Eighth grade
Ninth grade
Tenth grade
Eleventh grade
Twelfth grade

The tool is intended for use with the following age(s).

0-4 years old
5 years old
6 years old
7 years old
8 years old
9 years old
10 years old
11 years old
12 years old
13 years old
14 years old
15 years old
16 years old
17 years old
18 years old

The tool is intended for use with the following student populations.

Students in general education
Students with disabilities
English language learners

ACADEMIC ONLY: What skills does the tool screen?

Reading

Phonological processing:

RAN
Memory
Awareness
Letter sound correspondence
Phonics
Structural analysis

Word ID

Accuracy
Speed

Nonword

Accuracy
Speed

Spelling

Accuracy
Speed

Passage

Accuracy
Speed

Reading comprehension:

Multiple choice questions
Cloze
Constructed Response
Retell
Maze
Sentence verification
Other (please describe):

Listening comprehension:

Multiple choice questions
Cloze
Constructed Response
Retell
Maze
Sentence verification
Vocabulary
Expressive
Receptive

Mathematics

Global Indicator of Math Competence

Accuracy
Speed
Multiple Choice
Constructed Response

Early Numeracy

Accuracy
Speed
Multiple Choice
Constructed Response

Mathematics Concepts

Accuracy
Speed
Multiple Choice
Constructed Response

Mathematics Computation

Accuracy
Speed
Multiple Choice
Constructed Response

Mathematic Application

Accuracy
Speed
Multiple Choice
Constructed Response

Fractions/Decimals

Accuracy
Speed
Multiple Choice
Constructed Response

Algebra

Accuracy
Speed
Multiple Choice
Constructed Response

Geometry

Accuracy
Speed
Multiple Choice
Constructed Response

Other (please describe):

Please describe specific domain, skills or subtests:

The IXL LevelUp™ Diagnostic is designed to assess the math skills that are critical for success in grade-level math content. The content is organized into strands, which reflect the strands used in many states' standards across grades. The number of items a student answers from any strand is determined by the student’s rostered grade level. The following shows the strands covered and the grades in which students see the content: Numbers and Operations in Base Ten (includes Counting and Cardinality in Kindergarten): Preschool - Grade 5. Operations and Algebraic Thinking: Preschool - Grade 5. Measurement and Data: Preschool - Grade 5. Geometry: Preschool - Grade 12. Numbers and Operations - Fractions: Grades 3 - 5. The Number System: Grades 6 - 8. Expressions and Equations: Grades 6 - 8. Ratios and Proportional Relationships: Grades 6 - 7. Statistics and Probability: Grades 6 - 12. Functions: Grades 8 - 12. Algebra: Grades 9 - 12. Number and Quantity: Grades 9 - 12.

BEHAVIOR ONLY: Which category of behaviors does your tool target?

Internalizing
Externalizing
Internalizing and Externalizing

BEHAVIOR ONLY: Please identify which broad domain(s)/construct(s) are measured by your tool and define each sub-domain or sub-construct.

Acquisition and Cost Information

Administration

BEHAVIOR ONLY: What type of administrator is your tool designed for?

General education teacher
Special education teacher
Parent
Child
External observer
Other

If other, please specify:

What is the administration setting?

Direct observation
Rating scale
Checklist
Performance measure
Questionnaire
Direct: Computerized
One-to-one
Other

If other, please specify:

Does the tool require technology?

Yes

If yes, what technology is required to implement your tool? (Select all that apply)

Computer or tablet
Internet connection
Other technology (please specify)

If your program requires additional technology not listed above, please describe the required technology and the extent to which it is combined with teacher small-group instruction/intervention:

The IXL LevelUp Diagnostic is compatible with most devices and operating systems (e.g., Windows, Mac OS, and Chrome OS [including Chromebooks]). Minimum requirements include an up-to-date version of Chrome, Safari, or Edge.

What is the administration context?

Individual
Small group   If small group, n=
Large group   If large group, n=
Computer-administered
Other

If other, please specify:

ACADEMIC ONLY: What are the discontinue rules?

No discontinue rules provided
Basals
Ceilings
Other

If other, please specify:

The assessment ends for a student when one of the three stopping rule components has been met: (1) overall precision, (2) maximum number of items, and (3) maximum amount of time. The stopping rule will end the assessment if the overall precision of the student’s ability estimate reaches a prespecified level—specifically, if the ability estimate's conditional standard error of measurement (CSEM) is less than 0.3 logits. Additionally, the IXL LevelUp Diagnostic for Math will terminate when the maximum number of items is reached or the maximum time has elapsed (i.e., 60 minutes), provided the CSEM is lower than 0.4 logits.

Are norms available?

Yes

Are benchmarks available?

Yes

If yes, how many benchmarks per year?

The IXL LevelUp Diagnostic is typically administered in three distinct windows (beginning-of-year, middle-of-year, and end-of-year). There is no limit to the number of times students can take it per year. To ensure that math ability is the only construct measured, rather than a student’s ability to memorize or recall individual items, individual students will not be presented with any items they saw in the past 90 days.

If yes, for which months are benchmarks available?

Benchmarks are set relative to the beginning (August 1 – November 30), middle (December 1 – February 28), and end (March 1 – June 1) of the school year.

BEHAVIOR ONLY: Can students be rated concurrently by one administrator?

If yes, how many students can be rated concurrently?

Training & Scoring

Training

Is training for the administrator required?

No

Describe the time required for administrator training, if applicable:

Administrator training for the LevelUp Diagnostic is optional, though highly recommended. Administrator training is offered as 60-minute virtual sessions.

Please describe the minimum qualifications an administrator must possess.

No minimum qualifications

Are training manuals and materials available?

Yes

Are training manuals/materials field-tested?

Yes

Are training manuals/materials included in cost of tools?

Yes

If No, please describe training costs:

Administrators and teachers can refer to online resources including the IXL LevelUp Diagnostic for Math Technical Manual (https://www.ixl.com/materials/IXL_LevelUp_Diagnostic_for_Math_Technical_Manual.pdf), Teacher Implementation Guide (https://www.ixl.com/materials/i_guides/Teacher_Guide_IXL_LevelUp_Math_Benchmark.pdf), and Administrator Implementation Guide (https://www.ixl.com/materials/i_guides/Admin_Guide_IXL_LevelUp_Math_Benchmark.pdf) for guidance on administration and use of the Diagnostic. These are available at no additional cost. Live virtual administrator training is available as an additional purchase. These are offered at $695 for up to 50 attendees and $1095 for large audiences (50-200 attendees).

Can users obtain ongoing professional and technical support?

Yes

If Yes, please describe how users can obtain support:

Administrators and teachers can refer to online resources including the IXL LevelUp Diagnostic for Math Technical Manual, Teacher Implementation Guide, and Administrator Implementation Guide for guidance on administration and use of the Diagnostic. Users can also refer to IXL’s online help center for additional user guides and answers to frequently asked questions at www.ixl.com/help-center. IXL offers technical support via phone (855-255-6676) from 7AM to 7PM, Monday to Friday. Users may also contact IXL via email (help@ixl.com). Other than closure for major holidays, IXL’s attentive staff will respond to inquiries within one business day.

Scoring

How are scores calculated?

Manually (by hand)
Automatically (computer-scored)
Other

If other, please specify:

Do you provide basis for calculating performance level scores?

Yes

What is the basis for calculating performance level and percentile scores?

Age norms
Grade norms
Classwide norms
Schoolwide norms
Stanines
Normal curve equivalents

What types of performance level scores are available?

Raw score
Standard score
Percentile score
Grade equivalents
IRT-based score
Age equivalents
Stanines
Normal curve equivalents
Developmental benchmarks
Developmental cut points
Equated
Probability
Lexile score
Error analysis
Composite scores
Subscale/subtest scores
Other

If other, please specify:

After students complete the diagnostic, administrators and teachers have access to key data for planning instruction, including overall and strand-level grade scores and scaled scores derived from IRT-based scores, nationally normed percentiles, and overall and strand-based achievement levels.

Does your tool include decision rules?

No

If yes, please describe.

Can you provide evidence in support of multiple decision rules?

No

If yes, please describe.

Please describe the scoring structure. Provide relevant details such as the scoring format, the number of items overall, the number of items per subscale, what the cluster/composite score comprises, and how raw scores are calculated.

The IXL LevelUp Diagnostic calculates various score types according to the specific needs of teachers and administrators. Teachers and students have access to a score tied to expected student performance on grade-level content. For example, a score of 400 indicates that a student is ready for content typically taught at the beginning of 4th grade, while a score of 450 means a student is ready for content targeting the middle of 4th grade. Administrators also see a traditional linear scaled score that ranges from 0 to 400. This score provides administrators with a more finely grained tool for assessing student growth over time. Nationally-normed percentiles provide information about the typical levels of performance for an identifiable population of students or schools. For example, a student may achieve the highest test score in her class on a given assessment but still fall below the national average of students (i.e., a percentile rank < 50) at her grade level. With this information, educators can compare their students’ scores to the scores of students across the United States who completed the same assessment. Educators often use this information to tailor resources and instruction to maximize student learning and achievement. National norms may be found in the IXL LevelUp Diagnostic for Math Technical Manual, https://www.ixl.com/materials/IXL_LevelUp_Diagnostic_for_Math_Technical_Manual.pdf. The IXL LevelUp Diagnostic also classifies students as performing far below grade level, below grade level, on grade level, or above grade level. In addition, achievement levels are available for each content strand. Due to the adaptive nature of the IXL LevelUp Diagnostic, the total number of items delivered to students varies based on several factors. However, lower and upper bounds on the number of items from any content strand are set based on a student’s grade and time of year (e.g., for the Fractions strand, a student at the end of fourth grade would see a minimum of 4 items and a maximum of 10 items). The complete test constraints associated with grade are available in the IXL LevelUp Diagnostic for Math Technical Manual (https://www.ixl.com/materials/IXL_LevelUp_Diagnostic_for_Math_Technical_Manual.pdf). As a student progresses through the LevelUp Diagnostic, the algorithm updates its estimate of ability and the standard error of this ability estimate each time the student responds to an item using a conventional Bayesian expected a posteriori (EAP) estimation method. As the algorithm becomes more certain in its running estimate of student ability (as indicated by the decreasing standard error of the ability estimate), it selects items more suitable to the student’s ability estimate. That is, the algorithm chooses more difficult items as the student answers correctly and less challenging items as the student answers incorrectly. This adaptivity reduces the number of items needed to measure math ability, thus increasing test efficiency and improving the test experience.

Describe the tool’s approach to screening, samples (if applicable), and/or test format, including steps taken to ensure that it is appropriate for use with culturally and linguistically diverse populations and students with disabilities.

The IXL LevelUp Diagnostic for Math is a brief and comprehensive assessment designed to determine progress toward standards for students in pre-kindergarten through twelfth grade. Districts and schools may use this assessment to identify students who are falling behind and intervene as early as possible. Content development for the IXL LevelUp Diagnostic used a principled assessment design framework. Principled assessment design is a general framework for designing, developing, and implementing assessments to support the ongoing accumulation and synthesis of validity evidence claims (e.g., Ferrara et al., 2016). At the beginning of the process, this framework requires clearly defined assessment targets. These assessment targets drive the entire assessment development plan, and continuous focus on targets ensures that all subsequent decisions are consistent with providing evidence to support validity claims. The assessment targets for the IXL LevelUp Diagnostic are achievement level descriptors (ALDs) derived from standards from many states at various grade levels. Achievement level descriptors are statements that describe expectations of what students at specific achievement levels should know and be able to do. The IXL LevelUp Diagnostic determines progress towards standards for students in pre-kindergarten through twelfth grade. ALDs were written for each educational standard to describe the performance of students who are far below grade level, below grade level, on grade level, and above grade level. The format of the IXL LevelUp Diagnostic includes a mix of selected-response, constructed-response, and technology-enhanced questions. Where appropriate, the writers of these items applied the principles of Universal Design (Thompson et al., 2002) to reduce construct-irrelevant variance while simultaneously increasing accessibility. Item writers wrote items to target achievement level descriptors while considering important factors such as reading level, cognitive load, vocabulary, and sentence length, among others. Item writers were selected based on their relevant grade-level and subject-matter experience as ELA K–12 classroom teachers. The panel consisted of in-service educators with significant experience teaching or supervising the grade level(s) they evaluated, and with demographics matching that of elementary and middle school teachers in the U.S. Multiple rounds of internal review were then conducted to ensure all items appropriately targeted the intended achievement level descriptors while meeting the content and item writing specifications. The IXL LevelUp Diagnostic is designed to be appropriate for students with diverse backgrounds (e.g., race, ethnicity, culture, and gender) and levels of ability. The diagnostic is available in Spanish, making it accessible for Spanish-speaking students regardless of English language proficiency. It also supports assistive technologies to provide equal accessibility for all students. These include screen reader compatibility, audio support, keyboard shortcuts, browser zoom up to 200 percent, and high contrast ratios. More broadly, the IXL LevelUp Diagnostic is highly adaptive, which allows it to quickly adjust to meet students' diverse abilities and proficiency levels.

Classification Accuracy & Cross-Validation Summary

Grade	Grade 3	Grade 4	Grade 5	Grade 6	Grade 7	Grade 8
Classification Accuracy Fall
Classification Accuracy Winter
Classification Accuracy Spring

Legend

Convincing evidence

Partially convincing evidence

Unconvincing evidence

Data unavailable

^dDisaggregated data available

State assessments

Classification Accuracy

Describe the criterion (outcome) measure(s) including the degree to which it/they is/are independent from the screening measure.

The criterion measure is the end-of-year state summative math assessment administered in the following states: AR, CA, CT, KS, LA, MA, MN, NJ, OK, PA, and TX (see below for full list of assessments). The IXL LevelUp Diagnostic for Math and state math assessments are independent assessments that were developed by different organizations. The following state assessments were used as criterion measures: Arkansas Teaching, Learning & Assessment System (ATLAS); Smarter Balanced Assessment (SBA; in CA and CT); Kansas Assessment Program (KAP); Louisiana Educational Assessment Program (LEAP); Massachusetts Comprehensive Assessment System (MCAS); and the Minnesota Comprehensive Assessments (MCA); New Jersey Student Learning Assessments (NJSLA); Oklahoma School Testing Program (OSTP); Pennsylvania System of School Assessment (PSSA); and the State of Texas Assessments of Academic Readiness (STAAR®).

Describe when screening and criterion measures were administered and provide a justification for why the method(s) you chose (concurrent and/or predictive) is/are appropriate for your tool.

Describe how the classification analyses were performed and cut-points determined. Describe how the cut points align with students at-risk. Please indicate which groups were contrasted in your analyses (e.g., low risk students versus high risk students, low risk students versus moderate risk students).

Cut points on the state math assessments were determined using each state’s published cut scores and percentiles for student achievement, where available. In line with NCII TRC guidance, students with scores below the 20th percentile (or within the “Below Basic” proficiency level) were considered “at risk,” while those with scores at or above the 20th percentile (or above the “Below Basic” proficiency level) were considered “not at risk.” Cut points on the screening measure were identified as scores that maximized classification accuracy with the criterion measure. Using these values, students were classified as “at risk” if they scored below the cut point and “not at risk” if they scored at or above the cut point. Note that for each grade and at each time point analyzed here, the sample included more than 150 students from five geographical divisions as defined by the US Census Bureau: Pacific (CA), West North Central (MN, KS), West South Central (OK, TX, AR, LA), Middle Atlantic (PA, NJ), New England (MA, CT).

Were the children in the study/studies involved in an intervention in addition to typical classroom instruction between the screening measure and outcome assessment?

No

If yes, please describe the intervention, what children received the intervention, and how they were chosen.

Some students may have been involved in various interventions in their particular schools, but we do not know which interventions or which students.

Cross-Validation

Has a cross-validation study been conducted?

No

If yes,

Describe the criterion (outcome) measure(s) including the degree to which it/they is/are independent from the screening measure.

Describe when screening and criterion measures were administered and provide a justification for why the method(s) you chose (concurrent and/or predictive) is/are appropriate for your tool.

Describe how the cross-validation analyses were performed and cut-points determined. Describe how the cut points align with students at-risk. Please indicate which groups were contrasted in your analyses (e.g., low risk students versus high risk students, low risk students versus moderate risk students).

Were the children in the study/studies involved in an intervention in addition to typical classroom instruction between the screening measure and outcome assessment?

If yes, please describe the intervention, what children received the intervention, and how they were chosen.

Classification Accuracy - Fall

Classification Accuracy - Winter

Classification Accuracy - Spring

Reliability

Grade	Grade 3	Grade 4	Grade 5	Grade 6	Grade 7	Grade 8
Rating

Legend

Convincing evidence

Partially convincing evidence

Unconvincing evidence

Data unavailable

^dDisaggregated data available

*Offer a justification for each type of reliability reported, given the type and purpose of the tool.

Often, researchers evaluate the reliability of a traditional, fixed-form assessment via internal consistency measures such as Cronbach’s alpha or McDonald’s omega. However, within the context of a CAT, these measures are not appropriate. Below, we report two types of reliability that are appropriate for CAT assessments: marginal reliability and standard error of measurement (SEM; including accompanying plots).

*Describe the sample(s), including size and characteristics, for each reliability analysis conducted.

The sample for calculating marginal reliability and SEM included student records from the 2024-25 school year and part of the 2025-26 school year. The sample included students in all nine U.S. Census Bureau divisions. The table below reports the sample sizes by grade. As illustrated in the histogram plots (available from the Center upon request), this sample included students across all performance levels.

*Describe the analysis procedures for each reported type of reliability.

Marginal reliability: Although typical measures of reliability are not estimable for adaptive assessments, marginal reliability provides a method that closely approximates the traditional measures of internal consistency when the ability distribution and item parameters are known (see Dimitrov, 2003; Samejima, 1977, 1994). SEM: As mentioned, the IXL LevelUp Diagnostic uses expected a-posteriori (EAP) and maximum likelihood (ML; in final scoring) to estimate student ability and SEM to indicate the level of certainty or reliability of this estimate. It derives the SEM by calculating the standard deviation of the posterior distribution of the EAP estimate by integrating over all possible values of ability given a response pattern (see Bock & Mislevy, 1982). A student’s standard error of measurement (SEM) is an indicator of the precision of their score and describes the range in which a score may vary upon repeated testing that is due to chance. SEM is a function of the interaction between the ability of a student, the difficulty of the items, and the number of items on a test. A lower SEM indicates less error and more precision around a score. Because the CAT algorithm selects items based on a student’s estimated ability level, it can target a student more accurately and significantly decrease the SEM with fewer items than a traditional fixed-form assessment. Although the CAT algorithm targets students more accurately according to their ability, it generally performs better for students whose ability and performance are closer to the typical ability of students at or near their grade level. The figure below illustrates the distribution of standard errors across the student ability spectrum, and the table below reports the mean and median standard errors by grade with corresponding 95% confidence intervals. Although not summarized here, the Technical Manual for the IXL LevelUp Diagnostic for Math includes additional reliability information, including test-retest reliability and classification consistency.

*In the table(s) below, report the results of the reliability analyses described above (e.g., internal consistency or inter-rater reliability coefficients).

Results from other forms of reliability analysis not compatible with above table format:

Standard error of measurement (SEM) describes the range that a score may vary upon repeated testing that is due to chance. It is inversely related to marginal reliability, with lower SEM values indicating higher precision. In this sample, median SEM coefficients ranged from 0.310 in grades 7 and 8 to 0.316 in grade 4, indicating low error and high precision around students’ scores on the IXL LevelUp Diagnostic for Math across grades 3 through 8. The plots (available from the Center upon request) display SEM for each grade in the analysis.

Manual cites other published reliability studies:

No

Provide citations for additional published studies.

Do you have reliability data that are disaggregated by gender, race/ethnicity, or other subgroups (e.g., English language learners, students with disabilities)?

If yes, fill in data for each subgroup with disaggregated reliability data.

Results from other forms of reliability analysis not compatible with above table format:

Manual cites other published reliability studies:

Provide citations for additional published studies.

Validity

Grade	Grade 3	Grade 4	Grade 5	Grade 6	Grade 7	Grade 8
Rating

Legend

Convincing evidence

Partially convincing evidence

Unconvincing evidence

Data unavailable

^dDisaggregated data available

*Describe each criterion measure used and explain why each measure is appropriate, given the type and purpose of the tool.

The IXL LevelUp Diagnostic for Math Technical Manual provides validity evidence based on test content (i.e., subject-matter expert review), internal structure (i.e., unidimensionality and DIF), and relations to other variables. (For more information, see the Technical Manual here: https://www.ixl.com/materials/IXL_LevelUp_Diagnostic_for_Math_Technical_Manual.pdf). In this section, we focus on the relationship between the LevelUp Diagnostic and external variables among students in 3rd through 8th grade. In the concurrent validity analyses, the criterion measure was the NWEA MAP® Growth™ math assessment (MAP Growth), a widely used computer-adaptive assessment of math ability. In the predictive validity analyses, the criterion measures were end-of-year summative assessments from three states: the Arkansas Teaching, Learning & Assessment System (ATLAS), Massachusetts Comprehensive Assessment System (MCAS), and the Minnesota Comprehensive Assessments (MCA). All criterion measures and the IXL LevelUp Diagnostic are independent assessments developed by different organizations. Despite the assessments being separately developed, they are expected to be related as the underlying construct being measured is the same (i.e., students’ math ability).

*Describe the sample(s), including size and characteristics, for each validity analysis conducted.

For the concurrent validity analyses, the sample included students who completed both the IXL LevelUp Diagnostic for Math and MAP Growth during the middle-of-year time frame of the 2024-25 school year (December 2024 to February 2025). Sample sizes for the concurrent validity analyses ranged from 470 (Grade 4) to 1,238 (Grade 5). For the predictive validity analyses, the sample included students who completed the IXL LevelUp Diagnostic during the beginning-of-year time frame (August to November 2024) and the MAP Growth assessment during the end-of-year time frame (March to June 2025). Sample sizes for the predictive validity analyses ranged from 263 fifth-grade students to 1,027 third-grade students. The validity sample represented students across all performance levels and three U.S. Census Divisions: New England (MA), West North Central (MN), and West South Central (AR).

*Describe the analysis procedures for each reported type of validity.

All validity analyses were conducted using the Pearson product-moment correlation coefficient r. Concurrent analyses examined the relationship between students’ scores on the IXL LevelUp Diagnostic and their concurrent MAP Growth math RIT score, while predictive analyses examined the relationship between students’ scores on the IXL LevelUp Diagnostic and their subsequent state assessment scores. A positive correlation between the two variables indicates that students who have higher math ability as measured by the IXL LevelUp Diagnostic also have higher math ability as measured by the criterion assessment. For both types of validity, confidence intervals (95%) were calculated using Fisher’s r-to-z transformation.

*In the table below, report the results of the validity analyses described above (e.g., concurrent or predictive validity, evidence based on response processes, evidence based on internal structure, evidence based on relations to other variables, and/or evidence based on consequences of testing), and the criterion measures.

Results from other forms of validity analysis not compatible with above table format:

Manual cites other published reliability studies:

Yes

Provide citations for additional published studies.

An, X. (2026). Predictive Validity of the IXL LevelUp™ Diagnostic for Math: A Multi-State Study. pp 1-12. https://www.ixl.com/materials/us/research/LevelUp_Math_Validity_%28A_Multi-State_Study%29.pdf An, X. (2026). Predictive Validity of the IXL LevelUp™ Diagnostic for Math Using the State of Texas Assessments of Academic Readiness (STAAR) as a Criterion. pp 1-10. https://www.ixl.com/materials/us/research/Predictive_Validity_of_the_IXL_LevelUp_Diagnostic_for_Math_(STAAR_as_a_Criterion).pdf

Describe the degree to which the provided data support the validity of the tool.

In each grade, there was a statistically significant positive correlation between concurrent administrations of IXL’s LevelUp Diagnostic for Math and MAP Growth math. Likewise for the predictive analyses, in each grade there was a statistically significant positive correlation between scores on the fall administration of the LevelUp Diagnostic and the spring administration of state summative assessments (ATLAS, MCAS, and MCA). Coefficients ranged from .75 to .82, reflecting strong relationships between the IXL LevelUp Diagnostic and the four external assessments.

Do you have validity data that are disaggregated by gender, race/ethnicity, or other subgroups (e.g., English language learners, students with disabilities)?

No

If yes, fill in data for each subgroup with disaggregated validity data.

Results from other forms of validity analysis not compatible with above table format:

Manual cites other published reliability studies:

Provide citations for additional published studies.

Bias Analysis

Have you conducted additional analyses related to the extent to which your tool is or is not biased against subgroups (e.g., race/ethnicity, gender, socioeconomic status, students with disabilities, English language learners)? Examples might include Differential Item Functioning (DIF) or invariance testing in multiple-group confirmatory factor models.

Yes

If yes,

a. Describe the method used to determine the presence or absence of bias:

Differential item functioning (DIF) analysis investigates each item for signs of interactions with sample characteristics. An item is said to exhibit DIF when individuals with the same ability but from different groups have notably differing probabilities of answering the item correctly. DIF detection procedures help to gather validity evidence for the proposed interpretations of test scores by ensuring that scores are free from potential bias and that individual items do not create an advantage for one group over another. To examine manifest DIF, we used the residual-DIF (RDIF) method, which is an approach for detecting manifest DIF by comparing the residuals of item responses between demographic groups (e.g., male and female students; Lim, Choe, & Han, 2022; Lim & Choe, 2023). A residual is the difference between a student's actual response to an item and the expected probability of a correct response to the item conditioned on the student’s estimated ability. The RDIF statistic, per se, indicates the magnitude of DIF present in each item. Separate calculations for effect size are unnecessary. Absolute RDIF values less than 0.05 indicate little to no DIF, absolute RDIF values greater than 0.05 but less than 0.10 indicate moderate DIF, and absolute RDIF values greater than 0.10 indicate large DIF (Lim & Choe, 2023).

b. Describe the subgroups for which bias analyses were conducted:

Student gender (male vs. female) and student race/ethnicity (white vs. non-white).

c. Describe the results of the bias analyses conducted, including data and interpretative statements. Include magnitude of effect (if available) if bias has been identified.

When investigating DIF based on reported sex, out of several thousand items that received sufficient exposures from the relevant groups (i.e., male vs. female), 157 items were flagged for potential DIF. Of the items flagged, 71 (45.2%) indicated possible bias in favor of male students, and 86 (54.7%) indicated possible bias in favor of female students. When investigating for DIF based on race/ethnicity (i.e., white vs. others), only 53 items were flagged for potential DIF. Of these items flagged, 28 (52.8%) indicated possible bias in favor of white students and 25 (47.2%) indicated possible bias in favor of non-white students. However, all of the flagged items were found to be free from substantive DIF (see the following section). More importantly, all the flagged items were free from substantive DIF. It is important to distinguish between statistical DIF and substantive DIF (Penfield & Lam, 2000; Roussos & Stout, 1996). Statistical DIF refers to the statistical identification of DIF, whereas substantive DIF refers to the identification of construct-irrelevant factors responsible for the statistical DIF (i.e., potential sources of bias). It is always important to remember that statistical DIF is a detection strategy, and scrutiny of items is always warranted to identify and address substantive DIF. DIF detection methods may flag some items as being unbiased even though they are indeed biased, while some items identified as having bias may not actually be biased. Therefore, all items identified as potentially biased using the methods outlined above were reviewed by subject-matter experts to ensure that all items are free from substantive DIF.