Behavioral Progress Monitoring Tools Rating Rubric
Please note that the following rubrics are applied separately for each sub-scale, grade level/span, and informant targeted by the tool.
Foundational Psychometric Standards
Note: For all standards in this section, it is expected that evidence is drawn from a sample that is representative of students across all performance levels. Tools not intended for use with students across all performance levels may receive a rating of “N/A” in these categories.
Reliability
Full Bubble: Either (a) a model based approach to reliability was reported with at least two sources of variance or (b) at least two other types of reliability were reported appropriate for the purpose of the tool, and drawn from at least two samples that are representative of students across all performance levels; and for each type of reliability reported the lower bound of the confidence interval around the median estimate met or exceeded 0.70.
Half Bubble: Either (a) a model-based approach to reliability was reported with at least two sources of variance, or (b) at least two other types of reliability were reported appropriate for the purpose of the tool, drawn from at least one sample that is representative of students across all performance levels and/or for each type of reliability reported the lower bound of the confidence interval around the median estimate fell below 0.70 but met or exceeded 0.60.
Empty Bubble: Does not meet full or half bubble.
Validity
Full Bubble: There are at least two types of appropriately justified validity analyses* from a sample representative of students across all performance levels and the lower bound of the confidence interval around the each estimate met or exceeded 0.60 (or if not, within an acceptable range given the expected relationship with the criterion measure(s)).
Half Bubble: Analyses, measures, and sample were appropriate, but evidence was mixed, with one or more measures either not meeting or exceeding 0.60 or not within an acceptable range given the expected relationship with the criterion measure(s).
Empty Bubble: Does not meet full or half bubble.
Bias Analysis Conducted
Yes: One or more of the following three types of analyses were conducted:
1. Multiple-group confirmatory factor models for categorical item responses
2. Explanatory group models such as multiple-indicators, multiple-causes (MIMIC) or explanatory IRT with group predictors
3. Differential Item Functioning from Item Response Theory (DIF in IRT)
No: Does not meet “yes.”
Progress Monitoring with Intensive Population Standards
Note: For all standards in Tab 2, it is expected that evidence is drawn from a sample of student in need of intensive intervention. Convincing evidence that children were in need of intensive intervention may include one or more of the following: students have ED label; students are placed in an alternative school/classroom; students have demonstrated non-response to moderately intensive intervention (e.g., Tier 2); or students have demonstrated severe problem behaviors (e.g., Tier 3), according to an evidence-based tool (e.g., systematic screening tool or direct observation).
Sensitive to Behavior Change
Full Bubble: The basis for assuming that the data are sensitive to incremental change is strong (e.g., the range of possible scores is sufficient to detect small changes and documentation of sensitivity to change consistent with another criterion).
Half Bubble: The basis for assuming that the data are sensitive to incremental change is moderate (e.g., the range of possible scores is sufficient to detect a change and documentation of sensitivity to change).
Empty Bubble: Does not meet full or half bubble.
Dash: Data were not provided.
Reliability: Intensive Population
Full Bubble: A model based approach to reliability was reported with at least two sources of variance or at least two other types of reliability were reported appropriate for the purpose of the tool, and drawn from at least two samples that are representative of students in need of intensive intervention and for each type of reliability reported the lower bound of the confidence interval around the median estimate met or exceeded 0.70.
Half Bubble: A model-based approach to reliability was reported with at least two sources of variance or at least two other types of reliability were reported appropriate for the purpose of the tool, drawn from at least one sample that is representative of students in need of intensive intervention and/or for each type of reliability reported the lower bound of the confidence interval around the median estimate fell below 0.70 but met or exceeded 0.60.
Empty Bubble: Does not meet full or half bubble.
Dash: Data were not provided.
Validity: Intensive Population
Full Bubble: There are at least two types of appropriately justified validity analyses* from a sample representative of students in need of intensive intervention and the lower bound of the confidence interval around each estimate met or exceeded 0.60 (or if not, within an acceptable range given the expected relationship with the criterion measure(s)).
Half Bubble: Analyses, measures, and sample were appropriate, but evidence was mixed, with one or more measures either not meeting or exceeding 0.60 or not within an acceptable range given the expected relationship with the criterion measure(s).
Empty Bubble: Does not meet full or half bubble.
Data to Support Intervention Change
Full Bubble: The data provided to support decisions about intervention change is (1) strong (2) based on analysis of progress monitoring measurement collected at least weekly over the period of time that is deemed necessary for the decision rules; and (3) from a sample of students that is in need of intensive intervention.
Half Bubble: The data provided to support decisions about intervention change has (1) moderate empirical support for targeted behavior; (2) is based on analysis of progress monitoring measurement collected at least weekly over the period of time that is deemed necessary for the decision rules; and (3) is from a sample of students that is in need of intensive intervention.
Empty Bubble: Does not meet full or half bubble.
Dash: Data were not provided.
Data to Support Intervention Choice
Full Bubble: The data provide guidance on intervention choice (e.g., a class of relevant interventions or a specific intervention), that is (1) strongly evidence-based; (2) based on analysis of progress monitoring measurement collected at least weekly over the period of time that is deemed necessary for the decision rules, and (3) from a sample of students that is in need of intensive intervention.
Half Bubble: The data provide guidance on intervention choice (e.g., a class of relevant interventions or a specific intervention), that is (1) moderately evidence-based; (2) based on analysis of progress monitoring measurement collected at least weekly over the period of time that is deemed necessary for the decision rules, and (3) from a sample of students that is in need of intensive intervention.
Empty Bubble: Does not meet full or half bubble.
Dash: Data were not provided.
Usability
Administration Format
The administration format may include individual student administration and/or small or large group administration.
Administration and Scoring Time
The time needed to administer and score the assessment.
Administration and Scoring Time
The scoring format may include manual scoring (i.e., hand scoring) and/or automatic scoring (i.e., computer scoring).
Levels of Performance
The ability of a progress monitoring tool to classify and progress monitor students according to their level of performance.
Usability Study Conducted
A usability study examines the extent to which the tool is convenient and practicable for use.
