Reliability of Performance Level Score

Full Bubble: Either (a) a model based approach to reliability was reported with at least two sources of variance or (b) at least two other types of reliability were reported appropriate for the purpose of the tool (e.g., inter-rater reliability is provided for tools that require human judgment), and evidence was drawn from at least two samples that are representative of students across all performance levels; and for each type of reliability reported the lower bound of the confidence interval around the median coefficient met or exceeded 0.70.

Half Bubble: Either (a) a model-based approach to reliability was reported with at least two sources of variance or (b) at least two other types of reliability were reported appropriate for the purpose of the tool (e.g., inter-rater reliability is provided for tools that require human judgment), and evidence was drawn from at least one sample that is representative of students across all performance levels and/or for each type of reliability reported the lower bound of the confidence interval around the median coefficient fell below 0.70 but met or exceeded 0.60.

Empty Bubble: Does not meet full or half bubble.

Validity of Performance Level Score

Full Bubble: At least two types of validity were reported that were appropriately justified* for the tool and evidence was drawn from a sample representative of students across all performance levels and for each type of validity reported the lower bound of the confidence interval around the median coefficient met or exceeded 0.60 (or was within an acceptable range given the expected relationship with the criterion measure(s)).

Half Bubble: At least two types of validity were reported that were appropriately justified* for the tool and evidence was drawn from a sample representative of students across all performance levels and one type of validity met the following criteria but the other did not: the lower bound of the confidence interval around the median coefficient met or exceeded 0.60 (or was within an acceptable range given the expected relationship with the criterion measure(s)).

Empty Bubble: Does not meet full or half bubble.

^* Appropriately justified analyses must include criterion measures that are external to the progress monitoring system and theoretically linked to the underlying construct measured by the tool.

Bias Analysis Conducted

Yes: One or more of the following three types of analyses were conducted:

1. Multiple-group confirmatory factor models for categorical item responses
2. Explanatory group models such as multiple-indicators, multiple-causes (MIMIC) or explanatory IRT with group predictors
3. Differential Item Functioning from Item Response Theory (DIF in IRT)

No: Does not meet “yes.”

Sensitivity to Student Learning: Reliability of Slope

Full Bubble: The analysis conducted was appropriate, with sufficient number and spacing of data points*, from a sample of children in need of intensive intervention, and the lower bound of the confidence interval around the median coefficient met or exceeded 0.50.

Half Bubble: The analysis conducted was appropriate, with sufficient number and spacing of data points*, from a sample of children in need of intensive intervention, and the lower bound of the confidence interval around the median coefficient fell below 0.50 but met or exceeded 0.40.

Empty Bubble: Does not meet full or half bubble.

Dash: Data were not provided.

* Sufficient number and spacing of data points is defined as at least 10 regularly collected measurements over a period of at least 20 weeks.

Sensitivity to Student Learning: Validity of Slope

Full Bubble: There is at least one appropriately justified validity analysis*, with sufficient number and spacing of data points**, from a sample of children in need of intensive intervention, and the lower bound of the confidence interval around each coefficient met or exceeded 0.40 (or was within an acceptable range given the expected relationship with the criterion measure(s)).

Half Bubble: Analyses, measures, number and spacing of data points, and sample were appropriate, but evidence was mixed, with the lower bound of the confidence interval around each coefficient falling below 0.40 or not falling within an acceptable range given the expected relationship with the criterion measure(s).

Empty Bubble: Does not meet full or half bubble.

Dash: Data were not provided.

* Appropriately justified analyses must include criterion measures that are external to the progress monitoring system and theoretically linked to the underlying construct measured by the tool.
** Sufficient number and spacing of data points is defined as at least 10 regularly collected measurements over a period of at least 20 weeks.

Alternate Forms

Full Bubble: There are at least 20 alternate forms and evidence is strong for comparability of alternate forms, and from a sample of students in need of intensive intervention.

Half Bubble: There are at least 20 alternate forms and evidence for comparability is moderate, and from a sample of students in need of intensive intervention.

Empty Bubble: Does not meet full or half bubble.

Dash: Data were not provided.

Decision Rules for Setting and Revising Goals

Full Bubble: The basis for establishing decision rules for setting and revising goals is (1) strongly evidence-based; (2) based on analysis of progress monitoring measurement collected at least weekly over the period of time that is deemed necessary for the decision rules, and (3) from a sample of students that is in need of intensive intervention.

Half Bubble: The basis for establishing decision rules for setting and revising goals is (1) moderately evidence-based; (2) based on analysis of progress monitoring measurement collected at least weekly over the period of time that is deemed necessary for the decision rules, and (3) from a sample of students that is in need of intensive intervention.

Empty Bubble: Does not meet full or half bubble.

Dash: Data were not provided.

Decision Rules for Changing Instruction

Full Bubble: The basis for establishing decision rules for when changes to instruction need to be made is (1) strongly evidence-based; (2) based on analysis of progress monitoring measurement collected at least weekly over the period of time that is deemed necessary for the decision rules, and (3) from a sample of students that is in need of intensive intervention.

Half Bubble: The basis for establishing decision rules for when changes to instruction need to be made is (1) moderately evidence-based; (2) based on analysis of progress monitoring measurement collected at least weekly over the period of time that is deemed necessary for the decision rules, and (3) from a sample of students that is in need of intensive intervention.

Empty Bubble: Does not meet full or half bubble.

Dash: Data were not provided.

Administration Format

The administration format may include individual student administration, group administration, or computer-facilitated administration.

Administration and Scoring Time

The time needed to administer and score the assessment.

Scoring Format

The scoring format may include manual scoring (i.e., hand scoring) and/or automatic scoring (i.e., computer scoring).

Rates of Improvement and End of Year Benchmarks

Rates of improvement refers to the availability of standards for minimum acceptable growth (i.e., slope of improvement or average weekly gains, by grade level). End of year benchmarks refers to the availability of benchmarks for minimum acceptable end-of-year performance.