Behavioral Progress Monitoring Tools Rating Rubric
Please note that the following rubrics are applied separately for each sub-scale, grade level/span, and informant targeted by the tool.
Performance Level Standards
Note: For all standards in this section, it is expected that evidence is drawn from a sample that is representative of students across all performance levels. Tools not intended for use with students across all performance levels may receive a rating of “N/A” in these categories.
Reliability
Full Bubble: At least two types of reliability are reported that are appropriate1 for the purpose of the tool, and the analyses are drawn from at least two samples representative of students across all performance levels; and the median of the estimates for each type met or exceeded 0.70.
Half Bubble: At least two types of reliability are reported that are appropriate1 for the purpose of the tool, and
-
the analyses are drawn from at least one sample representative of students across all performance levels
or - the median of the estimates for each type met or exceeded 0.60.
Empty Bubble: Does not meet full or half bubble.
1 Tests which require human judgment must report inter-rater reliability to be eligible for a Full or Half Bubble rating. Other types of reliability must include justification of appropriateness given the purpose of the tool.
Validity
Full Bubble: At least two types of appropriately justified1 validity analyses are reported, and the analyses are drawn from at least one sample representative of students across all performance levels, and the median of the estimates for both types of validity met or exceeded 0.60 (or was within an acceptable range given the expected relationship with the criterion measure(s)).
Half Bubble: One type of appropriately justified1 validity analysis is reported, and the analysis is drawn from a sample representative of students across all performance levels, and the median of the estimates met or exceeded 0.60 (or was within an acceptable range given the expected relationship with the criterion measure(s)).
Empty Bubble: Does not meet full or half bubble.
1Appropriately justified analyses must include criterion measures that are external to the progress monitoring system and theoretically linked to the underlying construct measured by the tool.
Bias Analysis Conducted
Yes: One or more of the following three types of analyses were conducted:
1. Multiple-group confirmatory factor models for categorical item responses
2. Explanatory group models such as multiple-indicators, multiple-causes (MIMIC) or explanatory IRT with group predictors
3. Differential Item Functioning from Item Response Theory (DIF in IRT)
No: Does not meet “yes.”
Growth Standards
Note: For all standards in Tab 2, it is expected that evidence is drawn from a sample of student in need of intensive intervention. Convincing evidence that children were in need of intensive intervention may include one or more of the following: students have ED label; students are placed in an alternative school/classroom; students have demonstrated non-response to moderately intensive intervention (e.g., Tier 2); or students have demonstrated severe problem behaviors (e.g., Tier 3), according to an evidence-based tool (e.g., systematic screening tool or direct observation).
Sensitive to Behavior Change
Full Bubble: The basis for assuming that the data are sensitive to incremental change is strong (e.g., the range of possible scores is sufficient to detect small changes and documentation of sensitivity to change consistent with another criterion).
Half Bubble: The basis for assuming that the data are sensitive to incremental change is moderate (e.g., the range of possible scores is sufficient to detect a change and documentation of sensitivity to change).
Empty Bubble: Does not meet full or half bubble.
Dash: Data were not provided.
Reliability and Validity: Intensive Population
Full Bubble:
At least two types of reliability are reported that meet the following criteria:
- are appropriate1 for the purpose of the tool,
- are drawn from at least two samples representative of students in need of intensive intervention, and
- the median of the estimates for each type met or exceeded 0.70.
and
At least two types of validity are reported that meet the following criteria:
- are appropriately justified2,
- are drawn from at least one sample representative of students in need of intensive intervention, and
- the median of the estimates for both types of validity met or exceeded 0.60 (or was within an acceptable range given the expected relationship with the criterion measure(s)).
Half Bubble: One or more of the following were met:
- At least two types of reliability are reported that meet the following criteria: are appropriate for the purpose of the tool, are drawn from at least two samples representative of students in need of intensive intervention, and the median of the estimates for each type met or exceeded 0.60. However, validity was not reported.
- At least two types of validity are reported that meet the following criteria: are appropriately justified1, are drawn from a sample representative of students in need of intensive intervention, and the median of the estimates for both types of validity met or exceeded 0.60 (or was within an acceptable range given the expected relationship with the criterion measure(s)). However, reliability was not reported.
-
One type of reliability was reported that met the criteria in #1
and
one type of validity was reported that met the criteria in #2.
Empty Bubble: Does not meet full or half bubble.
Dash: Data were not provided.
1 Tests which require human judgment must report inter-rater reliability to be eligible for a Full or Half Bubble rating. Other types of reliability must include justification of appropriateness given the purpose of the tool.
2 Appropriately justified analyses must include criterion measures that are external to the progress monitoring system and theoretically linked to the underlying construct measured by the tool.
Data to Support Intervention Change
Full Bubble: The data provided to support decisions about intervention change is (1) strong (2) based on analysis of progress monitoring measurement collected at least weekly over the period of time that is deemed necessary for the decision rules; and (3) from a sample of students that is in need of intensive intervention.
Half Bubble: The data provided to support decisions about intervention change has (1) moderate empirical support for targeted behavior; (2) is based on analysis of progress monitoring measurement collected at least weekly over the period of time that is deemed necessary for the decision rules; and (3) is from a sample of students that is in need of intensive intervention.
Empty Bubble: Does not meet full or half bubble.
Dash: Data were not provided.
Data to Support Intervention Choice
Full Bubble: The data provide guidance on intervention choice (e.g., a class of relevant interventions or a specific intervention), that is (1) strongly evidence-based; (2) based on analysis of progress monitoring measurement collected at least weekly over the period of time that is deemed necessary for the decision rules, and (3) from a sample of students that is in need of intensive intervention.
Half Bubble: The data provide guidance on intervention choice (e.g., a class of relevant interventions or a specific intervention), that is (1) moderately evidence-based; (2) based on analysis of progress monitoring measurement collected at least weekly over the period of time that is deemed necessary for the decision rules, and (3) from a sample of students that is in need of intensive intervention.
Empty Bubble: Does not meet full or half bubble.
Dash: Data were not provided.
Usability
Administration Format
The administration format may include individual student administration and/or small or large group administration.
Administration and Scoring Time
The time needed to administer and score the assessment.
Administration and Scoring Time
The scoring format may include manual scoring (i.e., hand scoring) and/or automatic scoring (i.e., computer scoring).
Levels of Performance
The ability of a progress monitoring tool to classify and progress monitor students according to their level of performance.
Usability Study Conducted
A usability study examines the extent to which the tool is convenient and practicable for use.
