Reliability

Full Bubble: At least two types of reliability are reported that are appropriate¹ for the purpose of the tool, and the analyses are drawn from at least two samples representative of students across all performance levels; and the median of the estimates for each type met or exceeded 0.70.

Half Bubble: At least two types of reliability are reported that are appropriate¹ for the purpose of the tool, and

1. the analyses are drawn from at least one sample representative of students across all performance levels
   or
2. the median of the estimates for each type met or exceeded 0.60.

Empty Bubble: Does not meet full or half bubble.

¹ Tests which require human judgment must report inter-rater reliability to be eligible for a Full or Half Bubble rating. Other types of reliability must include justification of appropriateness given the purpose of the tool.

Validity

Full Bubble: At least two types of appropriately justified¹ validity analyses are reported, and the analyses are drawn from at least one sample representative of students across all performance levels, and the median of the estimates for both types of validity met or exceeded 0.60 (or was within an acceptable range given the expected relationship with the criterion measure(s)).

Half Bubble: One type of appropriately appropriately justified¹ validity analysis is reported, and the analysis is drawn from a sample representative of students across all performance levels, and the median of the estimates met or exceeded 0.60 (or was within an acceptable range given the expected relationship with the criterion measure(s)).

Empty Bubble: Does not meet full or half bubble.

¹ Appropriately justified analyses must include criterion measures that are external to the progress monitoring system and theoretically linked to the underlying construct measured by the tool.

Bias Analysis Conducted

Yes: One or more of the following three types of analyses were conducted:

1. Multiple-group confirmatory factor models for categorical item responses
2. Explanatory group models such as multiple-indicators, multiple-causes (MIMIC) or explanatory IRT with group predictors
3. Differential Item Functioning from Item Response Theory (DIF in IRT)

No: Does not meet “yes.”

Sensitive to Behavior Change

Full Bubble: The basis for assuming that the data are sensitive to incremental change is strong (e.g., the range of possible scores is sufficient to detect small changes and documentation of sensitivity to change consistent with another criterion).

Half Bubble: The basis for assuming that the data are sensitive to incremental change is moderate (e.g., the range of possible scores is sufficient to detect a change and documentation of sensitivity to change).

Empty Bubble: Does not meet full or half bubble.

Dash: Data were not provided.

Reliability and Validity: Intensive Population

Full Bubble: At least two types of reliability are reported that meet the following criteria:

1. are appropriate¹ for the purpose of the tool,
2. are drawn from at least two samples representative of students in need of intensive intervention, and
3. the median of the estimates for each type met or exceeded 0.70.

At least two types of validity are reported that meet the following criteria:

1. are appropriately justified²,
2. are drawn from at least one sample representative of students in need of intensive intervention, and
3. the median of the estimates for both types of validity met or exceeded 0.60 (or was within an acceptable range given the expected relationship with the criterion measure(s)).

Half Bubble: One or more of the following were met:

1. At least two types of reliability are reported that meet the following criteria: are appropriate for the purpose of the tool, are drawn from at least two samples representative of students in need of intensive intervention, and the median of the estimates for each type met or exceeded 0.60. However, validity was not reported.
2. At least two types of validity are reported that meet the following criteria: are appropriately justified¹, are drawn from a sample representative of students in need of intensive intervention, and the median of the estimates for both types of validity met or exceeded 0.60 (or was within an acceptable range given the expected relationship with the criterion measure(s)). However, reliability was not reported.
3. One type of reliability was reported that met the criteria in #1
   and
   one type of validity was reported that met the criteria in #2.

Empty Bubble: Does not meet full or half bubble.

Dash: Data were not provided.

¹ Tests which require human judgment must report inter-rater reliability to be eligible for a Full or Half Bubble rating. Other types of reliability must include justification of appropriateness given the purpose of the tool.
² Appropriately justified analyses must include criterion measures that are external to the progress monitoring system and theoretically linked to the underlying construct measured by the tool.

Data to Support Intervention Change

Full Bubble: The data provided to support decisions about intervention change is (1) strong (2) based on analysis of progress monitoring measurement collected at least weekly over the period of time that is deemed necessary for the decision rules; and (3) from a sample of students that is in need of intensive intervention.

Half Bubble: The data provided to support decisions about intervention change has (1) moderate empirical support for targeted behavior; (2) is based on analysis of progress monitoring measurement collected at least weekly over the period of time that is deemed necessary for the decision rules; and (3) is from a sample of students that is in need of intensive intervention.

Empty Bubble: Does not meet full or half bubble.

Dash: Data were not provided.

Data to Support Intervention Choice

Full Bubble: The data provide guidance on intervention choice (e.g., a class of relevant interventions or a specific intervention), that is (1) strongly evidence-based; (2) based on analysis of progress monitoring measurement collected at least weekly over the period of time that is deemed necessary for the decision rules, and (3) from a sample of students that is in need of intensive intervention.

Half Bubble: The data provide guidance on intervention choice (e.g., a class of relevant interventions or a specific intervention), that is (1) moderately evidence-based; (2) based on analysis of progress monitoring measurement collected at least weekly over the period of time that is deemed necessary for the decision rules, and (3) from a sample of students that is in need of intensive intervention.

Empty Bubble: Does not meet full or half bubble.

Dash: Data were not provided.

Administration Format

The administration format may include individual student administration and/or small or large group administration.

Administration and Scoring Time

The time needed to administer and score the assessment.

Administration and Scoring Format

The scoring format may include manual scoring (i.e., hand scoring) and/or automatic scoring (i.e., computer scoring).

Levels of Performance

The ability of a progress monitoring tool to classify and progress monitor students according to their level of performance.

Usability Study Conducted

A usability study examines the extent to which the tool is convenient and practicable for use.