Full Bubble: Either (a) a model based approach to reliability was reported with at least two sources of variance or (b) at least two other types of reliability were reported appropriate for the purpose of the tool (e.g., inter-rater reliability is provided for tools that require human judgment), and evidence was drawn from at least two samples that are representative of students across all performance levels; and for each type of reliability reported the lower bound of the confidence interval around the median coefficient met or exceeded 0.70.
Half Bubble:
Either (a) a model-based approach to reliability was reported with at least two sources of variance or (b) at least two other types of reliability were reported appropriate for the purpose of the tool (e.g., inter-rater reliability is provided for tools that require human judgment), and evidence was drawn from at least one sample that is representative of students across all performance levels and/or for each type of reliability reported the lower bound of the confidence interval around the median coefficient fell below 0.70 but met or exceeded 0.60.
Empty Bubble: Does not meet full or half bubble.
Academic Progress Monitoring Rating Rubric
Validity of Performance Level Score
Full Bubble:
At least two types of validity were reported that were appropriately justified* for the tool and evidence was drawn from a sample representative of students across all performance levels and for each type of validity reported the lower bound of the confidence interval around the median coefficient met or exceeded 0.60 (or was within an acceptable range given the expected relationship with the criterion measure(s)).
Half Bubble:
At least two types of validity were reported that were appropriately justified* for the tool and evidence was drawn from a sample representative of students across all performance levels and one type of validity met the following criteria but the other did not: the lower bound of the confidence interval around the median coefficient met or exceeded 0.60 (or was within an acceptable range given the expected relationship with the criterion measure(s)).
Empty Bubble: Does not meet full or half bubble.
* Appropriately justified analyses must include criterion measures that are external to the progress monitoring system and theoretically linked to the underlying construct measured by the tool.
Academic Progress Monitoring Rating Rubric
Bias Analysis Conducted
Yes:
One or more of the following three types of analyses were conducted:
Multiple-group confirmatory factor models for categorical item responses
Explanatory group models such as multiple-indicators, multiple-causes (MIMIC) or explanatory IRT with group predictors
Differential Item Functioning from Item Response Theory (DIF in IRT)
No: Does not meet “yes.”
Academic Progress Monitoring Rating Rubric
Sensitivity to Student Learning: Reliability of Slope
Full Bubble:
The analysis conducted was appropriate, with sufficient number and spacing of data points*, from a sample of children in need of intensive intervention, and the lower bound of the confidence interval around the median coefficient met or exceeded 0.50.
Half Bubble:
The analysis conducted was appropriate, with sufficient number and spacing of data points*, from a sample of children in need of intensive intervention, and the lower bound of the confidence interval around the median coefficient fell below 0.50 but met or exceeded 0.40.
Empty Bubble:
Does not meet full or half bubble.
Dash:
Data were not provided.
* Sufficient number and spacing of data points is defined as at least 10 regularly collected measurements over a period of at least 20 weeks.
Academic Progress Monitoring Rating Rubric
Sensitivity to Student Learning: Validity of Slope
Full Bubble:
There is at least one appropriately justified validity analysis*, with sufficient number and spacing of data points**, from a sample of children in need of intensive intervention, and the lower bound of the confidence interval around each coefficient met or exceeded 0.40 (or was within an acceptable range given the expected relationship with the criterion measure(s)).
Half Bubble:
Analyses, measures, number and spacing of data points, and sample were appropriate, but evidence was mixed, with the lower bound of the confidence interval around each coefficient falling below 0.40 or not falling within an acceptable range given the expected relationship with the criterion measure(s).
Empty Bubble:
Does not meet full or half bubble.
Dash: Data were not provided.
* Appropriately justified analyses must include criterion measures that are external to the progress monitoring system and theoretically linked to the underlying construct measured by the tool.
** Sufficient number and spacing of data points is defined as at least 10 regularly collected measurements over a period of at least 20 weeks.
Academic Progress Monitoring Rating Rubric
Alternate Forms
Full Bubble:
There are at least 20 alternate forms and evidence is strong for comparability of alternate forms, and from a sample of students in need of intensive intervention.
Half Bubble:
There are at least 20 alternate forms and evidence for comparability is moderate, and from a sample of students in need of intensive intervention.
Empty Bubble:
Does not meet full or half bubble.
Dash: Data were not provided.
Academic Progress Monitoring Rating Rubric
Decision Rules for Setting and Revising Goals
Full Bubble: The basis for establishing decision rules for setting and revising goals is (1) strongly evidence-based; (2) based on analysis of progress monitoring measurement collected at least weekly over the period of time that is deemed necessary for the decision rules, and (3) from a sample of students that is in need of intensive intervention.
Half Bubble:
The basis for establishing decision rules for setting and revising goals is (1) moderately evidence-based; (2) based on analysis of progress monitoring measurement collected at least weekly over the period of time that is deemed necessary for the decision rules, and (3) from a sample of students that is in need of intensive intervention.
Empty Bubble:
Does not meet full or half bubble.
Dash:
Data were not provided.
Academic Progress Monitoring Rating Rubric
Decision Rules for Changing Instruction
Full Bubble: The basis for establishing decision rules for when changes to instruction need to be made is (1) strongly evidence-based; (2) based on analysis of progress monitoring measurement collected at least weekly over the period of time that is deemed necessary for the decision rules, and (3) from a sample of students that is in need of intensive intervention.
Half Bubble:
The basis for establishing decision rules for when changes to instruction need to be made is (1) moderately evidence-based; (2) based on analysis of progress monitoring measurement collected at least weekly over the period of time that is deemed necessary for the decision rules, and (3) from a sample of students that is in need of intensive intervention.
Empty Bubble:
Does not meet full or half bubble.
Dash:
Data were not provided.
Academic Progress Monitoring Rating Rubric
Administration Format
The administration format may include individual student administration, group administration, or computer-facilitated administration.
Academic Progress Monitoring Rating Rubric
Administration and Scoring Time
The time needed to administer and score the assessment.
Academic Progress Monitoring Rating Rubric
Scoring Format
The scoring format may include manual scoring (i.e., hand scoring) and/or automatic scoring (i.e., computer scoring).
Academic Progress Monitoring Rating Rubric
Rates of Improvement and End of Year Benchmarks
Rates of improvement refers to the availability of standards for minimum acceptable growth (i.e., slope of improvement or average weekly gains, by grade level). End of year benchmarks refers to the availability of benchmarks for minimum acceptable end-of-year performance.
