mCLASS: Reading 3D
Text Reading & Comprehension (TRC)
Summary
mCLASS:3D - TRC is a set of screening and progress monitoring measures for grades K-6. Text Reading and Comprehension (TRC) is an individually administered assessment using leveled readers from a book set to determine a student’s instructional reading level. During this measure, students are asked to read a book and complete a number of follow-up tasks, which may include responding to oral comprehension questions, completing a retell, and/or writing responses to comprehension questions. Assessors observe and record the student’s oral reading behaviors through the administration of TRC to determine reading accuracy, reading fluency and comprehension of the text. The comprehension components help assessors determine whether the student understands the meaning of the text and the student’s instructional reading level. While the student reads from the set of leveled readers, the teacher follows along on a handheld device, recording the student’s performance as the child reads. The handheld software offers a pre-loaded class list indicating required assessment tasks, provides the teacher with directions and prompts to ensure standardized, accurate administration, and automates the precise timing requirements. Upon completion of each task, the handheld automatically calculates the student’s score and provides a risk evaluation. Student performance data are securely and immediately transferred to the Web-based mCLASS reporting system. The mCLASS:3D Web site offers a range of reports at the district, school, class, and individual student level for further analysis. The set of measures in the screening are designed to be administered at the beginning, middle, and end of year, with alternate forms of all measures available for progress monitoring in between screening windows.
- Where to Obtain:
- Amplify Education, Inc
- support@amplify.com
- 55 Washington Street Suite 800 Brooklyn, NY 11201-1071
- (800) 823-1969
- https://www.amplify.com/
- Initial Cost:
- $20.90 per student
- Replacement Cost:
- $20.90 per student per year
- Included in Cost:
- The basic pricing plan is an annual per student license of $20.90. For users already using an mCLASS assessment product, the cost per student to add mCLASS:3D is $6 per student.
- mCLASS allows for administration of the TRC assessment using mobile devices and allows teachers to easily record student responses with just a tap of a button as well as other observations noticed during an assessment for a deeper interpretation of students’ skills. It has embedded script for prompts and directions for ensuring standardized administration so all students receive the same opportunity to perform. The mCLASS Platform provides a comprehensive service for managing the staff organizational structure and student enrollment data, providing online reporting and analysis tools for users of different roles from administrators to classroom teachers, and supporting our mobile assessment delivery system. It supports the Now What Tools, which makes assessment results actionable for teachers by translating data into practical instructional support with tools for small- group instruction, item-level analysis, and parent letters. Educators and administrators can immediately access student data using reports that are designed for influencing instruction and informing administrative decisions. mCLASS is an assessment instrument well-suited for use with capturing the developing reading skills of students with disabilities, with a few exceptions: a) students who are deaf; b) students who have fluency-based speech disabilities, e.g., stuttering, oral apraxia; c) students who are learning to read in a language other than English or Spanish; d) students with severe disabilities. Use of mCLASS is appropriate for all other students, including those with disabilities and receiving special education supports for whom reading connected text is an IEP goal. For students receiving special education, it may be necessary to adjust goals and timelines; and provide accommodations as part of the administration. The purpose of accommodation is to facilitate assessment for children for whom a standard administration may not provide an accurate estimate of their skills in the core early literacy skill areas. Valid and acceptable accommodations are ones that are unlikely to change substantially the meaning or interpretation of a student’s scores. The valid and acceptable accommodations for TRC administration are available upon request
- Training Requirements:
- 4-8 hours of training
- Qualified Administrators:
- examiners must receive training in the assessment administration and scoring
- Access to Technical Support:
- Amplify’s Customer Care Center offers complete user-level support from 7:00 a.m. to 7:00 p.m. EST, Monday through Friday. Customers may contact a customer support representative via telephone, e- mail, or electronically through the mCLASS website. Calls to the Customer Care Center’s toll-free number are answered immediately by an automated attendant and routed to customer support agents according to regional expertise. Additionally, customers have self-service access to instructions, documents, and frequently asked questions on our Website. The research staff and product teams are available to answer questions about the content within the assessments. Larger implementations have a designated account manager to support ongoing successful implementation
- Assessment Format:
-
- One-to-one
- Scoring Time:
-
- Scoring is automatic
- Scores Generated:
-
- Raw score
- Developmental benchmarks
- Administration Time:
-
- 6 minutes per student
- Scoring Method:
-
- Automatically (computer-scored)
- Technology Requirements:
-
- Accommodations:
- mCLASS allows for administration of the TRC assessment using mobile devices and allows teachers to easily record student responses with just a tap of a button as well as other observations noticed during an assessment for a deeper interpretation of students’ skills. It has embedded script for prompts and directions for ensuring standardized administration so all students receive the same opportunity to perform. The mCLASS Platform provides a comprehensive service for managing the staff organizational structure and student enrollment data, providing online reporting and analysis tools for users of different roles from administrators to classroom teachers, and supporting our mobile assessment delivery system. It supports the Now What Tools, which makes assessment results actionable for teachers by translating data into practical instructional support with tools for small- group instruction, item-level analysis, and parent letters. Educators and administrators can immediately access student data using reports that are designed for influencing instruction and informing administrative decisions. mCLASS is an assessment instrument well-suited for use with capturing the developing reading skills of students with disabilities, with a few exceptions: a) students who are deaf; b) students who have fluency-based speech disabilities, e.g., stuttering, oral apraxia; c) students who are learning to read in a language other than English or Spanish; d) students with severe disabilities. Use of mCLASS is appropriate for all other students, including those with disabilities and receiving special education supports for whom reading connected text is an IEP goal. For students receiving special education, it may be necessary to adjust goals and timelines; and provide accommodations as part of the administration. The purpose of accommodation is to facilitate assessment for children for whom a standard administration may not provide an accurate estimate of their skills in the core early literacy skill areas. Valid and acceptable accommodations are ones that are unlikely to change substantially the meaning or interpretation of a student’s scores. The valid and acceptable accommodations for TRC administration are available upon request
Descriptive Information
- Please provide a description of your tool:
- mCLASS:3D - TRC is a set of screening and progress monitoring measures for grades K-6. Text Reading and Comprehension (TRC) is an individually administered assessment using leveled readers from a book set to determine a student’s instructional reading level. During this measure, students are asked to read a book and complete a number of follow-up tasks, which may include responding to oral comprehension questions, completing a retell, and/or writing responses to comprehension questions. Assessors observe and record the student’s oral reading behaviors through the administration of TRC to determine reading accuracy, reading fluency and comprehension of the text. The comprehension components help assessors determine whether the student understands the meaning of the text and the student’s instructional reading level. While the student reads from the set of leveled readers, the teacher follows along on a handheld device, recording the student’s performance as the child reads. The handheld software offers a pre-loaded class list indicating required assessment tasks, provides the teacher with directions and prompts to ensure standardized, accurate administration, and automates the precise timing requirements. Upon completion of each task, the handheld automatically calculates the student’s score and provides a risk evaluation. Student performance data are securely and immediately transferred to the Web-based mCLASS reporting system. The mCLASS:3D Web site offers a range of reports at the district, school, class, and individual student level for further analysis. The set of measures in the screening are designed to be administered at the beginning, middle, and end of year, with alternate forms of all measures available for progress monitoring in between screening windows.
ACADEMIC ONLY: What skills does the tool screen?
- Please describe specific domain, skills or subtests:
- Concepts of print
- BEHAVIOR ONLY: Which category of behaviors does your tool target?
-
- BEHAVIOR ONLY: Please identify which broad domain(s)/construct(s) are measured by your tool and define each sub-domain or sub-construct.
Acquisition and Cost Information
Administration
- Are norms available?
- Yes
- Are benchmarks available?
- Yes
- If yes, how many benchmarks per year?
- 3 (BOY, MOY, EOY as defined by district user)
- If yes, for which months are benchmarks available?
- approximately September, January, May
- BEHAVIOR ONLY: Can students be rated concurrently by one administrator?
- If yes, how many students can be rated concurrently?
Training & Scoring
Training
- Is training for the administrator required?
- Yes
- Describe the time required for administrator training, if applicable:
- 4-8 hours of training
- Please describe the minimum qualifications an administrator must possess.
- examiners must receive training in the assessment administration and scoring
-
No minimum qualifications
- Are training manuals and materials available?
- Yes
- Are training manuals/materials field-tested?
- Yes
- Are training manuals/materials included in cost of tools?
- No
- If No, please describe training costs:
- For first-time mCLASS users, a 2-day in-person training is available for $575 in-office or $4,800 onsite for 25 maximum participants
- Can users obtain ongoing professional and technical support?
- Yes
- If Yes, please describe how users can obtain support:
- Amplify’s Customer Care Center offers complete user-level support from 7:00 a.m. to 7:00 p.m. EST, Monday through Friday. Customers may contact a customer support representative via telephone, e- mail, or electronically through the mCLASS website. Calls to the Customer Care Center’s toll-free number are answered immediately by an automated attendant and routed to customer support agents according to regional expertise. Additionally, customers have self-service access to instructions, documents, and frequently asked questions on our Website. The research staff and product teams are available to answer questions about the content within the assessments. Larger implementations have a designated account manager to support ongoing successful implementation
Scoring
- Do you provide basis for calculating performance level scores?
-
Yes
- Does your tool include decision rules?
- If yes, please describe.
- Can you provide evidence in support of multiple decision rules?
-
No
- If yes, please describe.
- Please describe the scoring structure. Provide relevant details such as the scoring format, the number of items overall, the number of items per subscale, what the cluster/composite score comprises, and how raw scores are calculated.
- Raw scores are provided as the reading level of the student, categorized as a reading level A through Z. A student’s reading level is a composite of his reading accuracy and comprehension of text. Cut Points for determining reading level are provided. A student must reach the accuracy and comprehension cut points in order for a book level to be determined as the student’s reading level. Developmental benchmarks for each measure, grade, and time of year (beginning, middle, end) classify each student’s score as Above Proficient, Proficient, Below Proficient, Far Below Proficient.
- Describe the tool’s approach to screening, samples (if applicable), and/or test format, including steps taken to ensure that it is appropriate for use with culturally and linguistically diverse populations and students with disabilities.
- TRC is a set of screening and progress monitoring measures for grades K-6. Text Reading and Comprehension (TRC) is an individually administered assessment using leveled readers from a book set to determine a student’s instructional reading level. During this assessment, students are asked to read a book and complete a number of follow-up tasks, which may include responding to oral comprehension questions, completing a retell, and/or writing responses to comprehension questions. Assessors observe and record the student’s oral reading behaviors through the administration of TRC to determine reading accuracy, reading fluency and comprehension of the text. The comprehension components help assessors determine whether the student understands the meaning of the text and the student’s instructional reading level. The instructional reading level, a composite score of reading accuracy and comprehension is used to classify students in one of four proficiency levels. The skills assessed in TRC include those skills that must be mastered for any student learning to read in English. The materials were subject to multiple rounds of review by content development experts and school-based professionals to ensure they were culturally relevant and free from bias. Field testing data (including qualitative feedback from educators) also indicate lack of bias in the results. The observational administration allows for flexibility with respect to issues of linguistic diversity through professional judgment based on the student’s responses and prior knowledge of his/her speech patterns. A list of approved accommodations is available upon request.
Technical Standards
Classification Accuracy & Cross-Validation Summary
| Grade |
Kindergarten
|
Grade 1
|
Grade 2
|
Grade 3
|
Grade 4
|
Grade 5
|
Grade 6
|
|---|---|---|---|---|---|---|---|
| Classification Accuracy Fall |
|
|
|
|
|
|
|
| Classification Accuracy Winter |
|
|
|
|
|
|
|
| Classification Accuracy Spring |
|
|
|
|
|
|
|
Legend
Convincing evidencePartially convincing evidenceUnconvincing evidenceData unavailabledDisaggregated data available
DIBELS Next
Classification Accuracy
- Describe the criterion (outcome) measure(s) including the degree to which it/they is/are independent from the screening measure.
- DIBELS Next (Dynamic Measurement Group, 2010) is a set of measures used to assess early literacy and reading skills, including phonemic awareness, basic phonics, accurate and fluent reading of connected text, and reading comprehension for students from kindergarten through Grade 6. It includes the following measures: Letter Naming, Fluency (LNF), First Sound Fluency (FSF), Phoneme Segmentation Fluency (PSF), Nonsense Word Fluency (NWF), Oral Reading Fluency (DORF), and Daze — a maze task. An overall composite score is calculated based on a student’s scores on grade-specific measures to provide an overall indication of literacy skills. DIBELS Next is considered an appropriate criterion measure given the strong reliability and validity evidence demonstrated by various studies (please refer to the DIBELS Next technical manual for details:Good, Kaminski, Dewey, Walin, Powell-Smith, & Latimer, 2013). DIBELS Next was selected as the criterion measure as the Composite Score is a powerful indicator of overall reading skill. DIBELS Next serves a similar purpose to TRC: to provide an indicator of risk or proficiency with grade appropriate reading skills, and to measure growth in reading skills over time. While both DIBELS Next and TRC are available within the mCLASS platform, they are completely separate assessments.
- Describe when screening and criterion measures were administered and provide a justification for why the method(s) you chose (concurrent and/or predictive) is/are appropriate for your tool.
- Describe how the classification analyses were performed and cut-points determined. Describe how the cut points align with students at-risk. Please indicate which groups were contrasted in your analyses (e.g., low risk students versus high risk students, low risk students versus moderate risk students).
- DIBELS Next is used as the criterion measure for this classification analysis. The performance on TRC at the middle of the year is used to predict students’ achievement level on DIBELS Next at the end of the year. Classification rates and ROC Curves are calculated in R software (R Development Core Team, 2017) using the “pROC” and “ROCR” packages. The TRC cut points are intended to reflect the grade-level expectations encompassed in the Common Core State Standards (CCSS). The cut point for indicate a level of skill below which the student is unlikely to achieve subsequent reading goals without receiving additional, targeted instructional support. The cut points are different depending on the specific grade and benchmark period. A panel of early reading and literacy experts established the cut points for the TRC performance levels (“Far Below Proficient”, “Below Proficient”, “Proficient”, and “Above Proficient”) using the ID Matching standard setting method. A student who is in the “Far Below Proficient” category shows an incomplete understanding of grade-level test and might need intensive support. Evidence for those cut points has been provided in prior submissions. For DIBELS Next, three composite score ranges were established to determine a student’s overall reading performance level. A student who scores at level one, Well Below Benchmark (scores below the cut point for risk) shows an incomplete understanding of grade-level text and might need intensive support; a student who scores at level two, Below Benchmark (scores below the benchmark goal and at or above the cut point for risk) might need strategic support but cannot make prediction of outcome; while a student who scores at level three, At or Above Benchmark (scores at or above the benchmark goal), comprehends text at or beyond grade level. Therefore, for this analysis, students assigned to the “risk” category were those performing “Well Below Benchmark” and need intensive intervention
- Were the children in the study/studies involved in an intervention in addition to typical classroom instruction between the screening measure and outcome assessment?
-
Yes
- If yes, please describe the intervention, what children received the intervention, and how they were chosen.
- The sample was selected from the mCLASS data system user base and represents 173224 students from 140 districts across the country. Therefore, specific interventions and instruction provided are unknown
Cross-Validation
- Has a cross-validation study been conducted?
-
No
- If yes,
- Describe the criterion (outcome) measure(s) including the degree to which it/they is/are independent from the screening measure.
- Describe when screening and criterion measures were administered and provide a justification for why the method(s) you chose (concurrent and/or predictive) is/are appropriate for your tool.
- Describe how the cross-validation analyses were performed and cut-points determined. Describe how the cut points align with students at-risk. Please indicate which groups were contrasted in your analyses (e.g., low risk students versus high risk students, low risk students versus moderate risk students).
- Were the children in the study/studies involved in an intervention in addition to typical classroom instruction between the screening measure and outcome assessment?
- If yes, please describe the intervention, what children received the intervention, and how they were chosen.
Classification Accuracy - Winter
| Evidence | Kindergarten | Grade 1 | Grade 2 | Grade 3 | Grade 4 | Grade 5 | Grade 6 |
|---|---|---|---|---|---|---|---|
| Criterion measure | DIBELS Next | DIBELS Next | DIBELS Next | DIBELS Next | DIBELS Next | DIBELS Next | DIBELS Next |
| Cut Points - Percentile rank on criterion measure | |||||||
| Cut Points - Performance score on criterion measure | |||||||
| Cut Points - Corresponding performance score (numeric) on screener measure | Text level A | Text level D | Text level I | Text level L | Text level A | Text level R | Text level V |
| Classification Data - True Positive (a) | 2695 | 9318 | 8059 | 3968 | 2360 | 1700 | 167 |
| Classification Data - False Positive (b) | 15018 | 6263 | 5311 | 2027 | 2001 | 1367 | 172 |
| Classification Data - False Negative (c) | 307 | 1160 | 926 | 1195 | 409 | 406 | 20 |
| Classification Data - True Negative (d) | 26475 | 30056 | 29601 | 12256 | 5519 | 4260 | 208 |
| Area Under the Curve (AUC) | 0.80 | 0.93 | 0.95 | 0.90 | 0.88 | 0.87 | 0.86 |
| AUC Estimate’s 95% Confidence Interval: Lower Bound | 0.79 | 0.93 | 0.94 | 0.89 | 0.88 | 0.86 | 0.82 |
| AUC Estimate’s 95% Confidence Interval: Upper Bound | 0.80 | 0.94 | 0.95 | 0.90 | 0.89 | 0.88 | 0.89 |
| Statistics | Kindergarten | Grade 1 | Grade 2 | Grade 3 | Grade 4 | Grade 5 | Grade 6 |
|---|---|---|---|---|---|---|---|
| Base Rate | 0.07 | 0.22 | 0.20 | 0.27 | 0.27 | 0.27 | 0.33 |
| Overall Classification Rate | 0.66 | 0.84 | 0.86 | 0.83 | 0.77 | 0.77 | 0.66 |
| Sensitivity | 0.90 | 0.89 | 0.90 | 0.77 | 0.85 | 0.81 | 0.89 |
| Specificity | 0.64 | 0.83 | 0.85 | 0.86 | 0.73 | 0.76 | 0.55 |
| False Positive Rate | 0.36 | 0.17 | 0.15 | 0.14 | 0.27 | 0.24 | 0.45 |
| False Negative Rate | 0.10 | 0.11 | 0.10 | 0.23 | 0.15 | 0.19 | 0.11 |
| Positive Predictive Power | 0.15 | 0.60 | 0.60 | 0.66 | 0.54 | 0.55 | 0.49 |
| Negative Predictive Power | 0.99 | 0.96 | 0.97 | 0.91 | 0.93 | 0.91 | 0.91 |
| Sample | Kindergarten | Grade 1 | Grade 2 | Grade 3 | Grade 4 | Grade 5 | Grade 6 |
|---|---|---|---|---|---|---|---|
| Date | |||||||
| Sample Size | 44495 | 46797 | 43897 | 19446 | 10289 | 7733 | 567 |
| Geographic Representation | |||||||
| Male | |||||||
| Female | |||||||
| Other | |||||||
| Gender Unknown | |||||||
| White, Non-Hispanic | |||||||
| Black, Non-Hispanic | |||||||
| Hispanic | |||||||
| Asian/Pacific Islander | |||||||
| American Indian/Alaska Native | |||||||
| Other | |||||||
| Race / Ethnicity Unknown | |||||||
| Low SES | |||||||
| IEP or diagnosed disability | |||||||
| English Language Learner |
Reliability
| Grade |
Kindergarten
|
Grade 1
|
Grade 2
|
Grade 3
|
Grade 4
|
Grade 5
|
Grade 6
|
|---|---|---|---|---|---|---|---|
| Rating |
|
|
|
|
|
|
|
Legend
Convincing evidencePartially convincing evidenceUnconvincing evidenceData unavailabledDisaggregated data available
- *Offer a justification for each type of reliability reported, given the type and purpose of the tool.
- Internal consistency and alternate form reliability: Internal consistency reliability refers to a person’s degree of confidence in the precision of scores from a single measurement. It’s used to indicate the variation of test scores which is attributable to measurement error. Alternate form reliability indicates the extent to which test results generalize to different forms. Alternate forms of the test with different items should give approximately the same scores. There are two to three books per text level in TRC. Books at the same text level are considered alternate forms. An individual student’s performance on the alternate books at the same text level should yield approximately the same scores on oral reading accuracy, comprehension, and/or retell/recall, as well as overall book performance. Inter-rater reliability evidence for grades K through 6: In observational assessments such as TRC, it is important that student performance be unrelated to or unaffected by a specific test administrator. Because there is a degree of subjectivity in scoring the accuracy of oral reading and comprehension, it is important to examine the degree to which TRC administrators can score student reading accuracy in a standardized and consistent manner. The sources of error associated with inter-rater reliability lie in the assessor.
- *Describe the sample(s), including size and characteristics, for each reliability analysis conducted.
- The internal consistency reliability was computed using TRC data at school year 2016-2017 in 9 regions and 19 states. The sample size is 2513 in total. The sample is comprised of 45% female and 46% male students; students were identified as 23% white, 25% Black or African-American, and 20% Hispanic-Latino. 39% of students were eligible for free or reduced lunch. Data from two samples are collected to provide evidence for alternate form reliability. The first sample contains 33 students from Kindergarten to Grade 5: 8 from kindergarten, 10 from Grade 1, four from Grade 2, four from Grade 3, two from Grade 4, and five from Grade 5. The sample was 39 percent female and 61 percent male; 9 percent white, 21 percent Hispanic, 67 percent black, and 3 percent represented other races. The 33 students are assessed from two schools in two Southern states during the 2013-2014 end of year benchmark administration period. The second sample includes 40 students in Grades 4-6 from two schools in two Southern states during the 2014-2015 middle of year benchmark administration period. The sample is composed of students in Grade 4 (n = 15), Grade 5 (n = 15), and Grade 6 (n = 10); 39 percent of the students were female and 61 percent male; 67 percent of students were black, 21 percent were Hispanic, 9 percent were white, and 3 percent were of other ethnicity. Inter-rater reliability evidence was obtained from two studies. STUDY 1: In one study, three raters assessed 33 students from two schools in two Southern states during the 2013–2014 end-of-year benchmark administration period. Among the students, representation was as follows: 8 from kindergarten, 10 from Grade 1, four from Grade 2, four from Grade 3, two from Grade 4, and five from Grade 5. The sample was 39 percent female and 61 percent male; 9 percent white, 21 percent Hispanic, 67 percent black, and 3 percent represented other races. STUDY 2: Another study was conducted between the 2014–2015 beginning-of-year and middle-of-year benchmark administration periods. In total, four raters assessed 40 students from two schools in two Southern states during the 2014–2015 MOY benchmark administration period. The sample is composed of students in Grade 4 (n = 15), Grade 5 (n = 15), and Grade 6 (n = 10); 39 percent of the students were female and 61 percent male; 67 percent of students were black, 21 percent were Hispanic, 9 percent were white, and 3 percent were of other ethnicity.
- *Describe the analysis procedures for each reported type of reliability.
- Cronbach’s alpha is used as the indicator for internal consistency, which quantifies the degree to which the items on an assessment all measure the same underlying construct. To avoid missing responses, the students in each grades who are reading at-grade proficient text level books are used to compute the Cronbach’s alpha. The 95% confidence interval of the Cronbach’s alpha is computed using the Bootstrap method, where 1000 samples with replacement are drawn from the data, calculating for each alpha and computing the 2.5% and 97.5% quantiles. For alternate form reliability, students were each assessed on two books at their instructional reading level and two books of the one level below their instructional reading level. Paired t-test comparisons are conducted to examine each component of TRC (accuracy, comprehension, retell/recall) as well as overall book performance. Raters’ scores are compared using intraclass correlations (ICC). ICC is one of the most commonly used statistics for assessing IRR on ordinal, interval, or ratio variables and is suitable for studies with two or more coders (Hallgren, 2012). Cicchetti (1994) provides cutoffs for ICC values, with IRR being poor for values less than 0.40, fair for values between 0.40 and 0.59, good for values between 0.60 and 0.74, and excellent for values between 0.75 and 1.00. Cohen’s kappa was also explored for overall book performance. Fleiss (1981) suggested kappa values greater than 0.75 to indicated excellent agreement, 0.40 to 0.75 as fair to good, and below 0.40 as poor. IRR estimates reported here are based on two or more independent assessors simultaneously scoring student performance during a single test administration (shadow-scoring).
*In the table(s) below, report the results of the reliability analyses described above (e.g., internal consistency or inter-rater reliability coefficients).
| Type of | Subgroup | Informant | Age / Grade | Test or Criterion | n | Median Coefficient | 95% Confidence Interval Lower Bound |
95% Confidence Interval Upper Bound |
|---|
- Results from other forms of reliability analysis not compatible with above table format:
- Additional data is available upon request from the Center.
- Manual cites other published reliability studies:
- Yes
- Provide citations for additional published studies.
- Reference: Amplify (2015). mCLASS Reading3D - Amplify Atlas Book Set Technical Manual, 2nd Edition. Brooklyn, NY: Author
- Do you have reliability data that are disaggregated by gender, race/ethnicity, or other subgroups (e.g., English language learners, students with disabilities)?
- No
If yes, fill in data for each subgroup with disaggregated reliability data.
| Type of | Subgroup | Informant | Age / Grade | Test or Criterion | n | Median Coefficient | 95% Confidence Interval Lower Bound |
95% Confidence Interval Upper Bound |
|---|
- Results from other forms of reliability analysis not compatible with above table format:
- Manual cites other published reliability studies:
- No
- Provide citations for additional published studies.
Validity
| Grade |
Kindergarten
|
Grade 1
|
Grade 2
|
Grade 3
|
Grade 4
|
Grade 5
|
Grade 6
|
|---|---|---|---|---|---|---|---|
| Rating |
d
|
d
|
d
|
d
|
d
|
d
|
d
|
Legend
Convincing evidencePartially convincing evidenceUnconvincing evidenceData unavailabledDisaggregated data available
- *Describe each criterion measure used and explain why each measure is appropriate, given the type and purpose of the tool.
- DIBELS Next measures are brief, powerful indicators of foundational early literacy skills that: are quick to administer and score; serve as universal screening (or benchmark assessment) and progress monitoring; identify students in need of intervention support; evaluate the effectiveness of interventions; and support the RtI/Multi-tiered model. DIBELS Next includes six measures: First Sound Fluency (FSF), Letter Naming Fluency (LNF), Phoneme Segmentation Fluency (PSF), Nonsense Word Fluency (NWF), DIBELS Oral Reading Fluency (DORF), and Daze. An overall composite score is calculated based on a student’s scores on grade-specific measures to provide an overall indication of literacy skills. DIBELS Next is considered an appropriate criterion measure given the strong reliability and validity evidence demonstrated by various studies (please refer to the DIBELS Next technical manual for details: Good et al., 2013). DIBELS Next was selected as the criterion measure as the Composite Score is a powerful indicator of overall reading skill. DIBELS Next serves a similar purpose to TRC: to provide an indicator of risk or proficiency with grade appropriate reading skills, and to measure growth in reading skills over time. While both DIBELS Next and TRC are available within the mCLASS platform, they are completely separate assessments.
- *Describe the sample(s), including size and characteristics, for each validity analysis conducted.
- The predictive validity was computed using TRC data at middle of year to predict DIBELS Next end of year at school year 2016-2017 in 9 regions and 18 states. The sample is comprised of 47% female and 45% male students; students were identified as 19% white, 22% Black or African-American, and 32% Hispanic-Latino. 32% of students were eligible for free or reduced lunch. The concurrent validity was computed using TRC data at end of year to predict DIBELS Next end of year at school year 2016-2017 in 9 regions and 18 states. The sample is comprised of 47% female and 45% male students; students were identified as 18% white, 22% Black or African- American, and 30% Hispanic-Latino. 29% of students were eligible for free or reduced lunch.
- *Describe the analysis procedures for each reported type of validity.
- Evidence of concurrent validity is often presented as a correlation between the assessment and an external criterion measure. Instructional reading levels determined from the administration of the Atlas edition of TRC should correlate highly with other accepted procedures and measures that determine overall reading achievement, including accuracy and comprehension. The degree of correlation between two conceptually related, concurrently administered tests suggests the tests measure the same underlying psychological constructs or processes. The correlation of final instructional reading level on TRC with the Composite score on DIBELS Next at the end of year is computed to provide predictive validity evidence. 95% confidence interval of the correlation is computed using the “stats” package in in R software package (R Development Core Team, 2017). Predictive validity provides an estimate of the extent to which student performance on TRC predicts scores on the criterion measure administered at a later point in time, defined as more than three months in this study. The correlation of final instructional reading level on TRC at the middle of year with the Composite score resulting from subsequent administration of DIBELS Next at the end of year is computed to provide predictive validity evidence. 95% confidence interval of the correlation is computed using the “stats” package in in R software package (R Development Core Team, 2017).
*In the table below, report the results of the validity analyses described above (e.g., concurrent or predictive validity, evidence based on response processes, evidence based on internal structure, evidence based on relations to other variables, and/or evidence based on consequences of testing), and the criterion measures.
| Type of | Subgroup | Informant | Age / Grade | Test or Criterion | n | Median Coefficient | 95% Confidence Interval Lower Bound |
95% Confidence Interval Upper Bound |
|---|
- Results from other forms of validity analysis not compatible with above table format:
- Manual cites other published reliability studies:
- Yes
- Provide citations for additional published studies.
- Reference: Amplify (2015). mCLASS Reading3D - Amplify Atlas Book Set Technical Manual, 2nd Edition. Brooklyn, NY: Author. (Contact vendor to inquire about technical manual.)
- Describe the degree to which the provided data support the validity of the tool.
- The table above summarizes the concurrent and predictive validity evidence for each grade. Across Grades K to 6, current validity coefficients range from 0.71 to 0.82, demonstrating strong correlations between final instructional reading level on TRC and DIBELS Next composite score at the end of year. The lower bounds of 95% confidence intervals are all above 0.70. Across Grades K to 6, predictive validity coefficients are in the range of 0.59 to 0.79. The lower bounds of 95% confident intervals are above 0.70 for Grades 1 to 6. The correlation with the DIBELS Next composite score is slighter lower in Kindergarten than the other grades, possibly because text levels at the lower grades are much less variable due to floor effect at Kindergarten.
- Do you have validity data that are disaggregated by gender, race/ethnicity, or other subgroups (e.g., English language learners, students with disabilities)?
- Yes
If yes, fill in data for each subgroup with disaggregated validity data.
| Type of | Subgroup | Informant | Age / Grade | Test or Criterion | n | Median Coefficient | 95% Confidence Interval Lower Bound |
95% Confidence Interval Upper Bound |
|---|
- Results from other forms of validity analysis not compatible with above table format:
- Manual cites other published reliability studies:
- No
- Provide citations for additional published studies.
Bias Analysis
| Grade |
Kindergarten
|
Grade 1
|
Grade 2
|
Grade 3
|
Grade 4
|
Grade 5
|
Grade 6
|
|---|---|---|---|---|---|---|---|
| Rating | Provided | Provided | Provided | Provided | Provided | Provided | Not Provided |
- Have you conducted additional analyses related to the extent to which your tool is or is not biased against subgroups (e.g., race/ethnicity, gender, socioeconomic status, students with disabilities, English language learners)? Examples might include Differential Item Functioning (DIF) or invariance testing in multiple-group confirmatory factor models.
- Yes
- If yes,
- a. Describe the method used to determine the presence or absence of bias:
- Classification analyses previously described were disaggregated by subgroups to determine whether the assessment is functioning similarly across subgroups.
- b. Describe the subgroups for which bias analyses were conducted:
- Data were disaggregated for the following groups: White, Black, Hispanic
- c. Describe the results of the bias analyses conducted, including data and interpretative statements. Include magnitude of effect (if available) if bias has been identified.
- AUC results are similar across subgroups.
Data Collection Practices
Most tools and programs evaluated by the NCII are branded products which have been submitted by the companies, organizations, or individuals that disseminate these products. These entities supply the textual information shown above, but not the ratings accompanying the text. NCII administrators and members of our Technical Review Committees have reviewed the content on this page, but NCII cannot guarantee that this information is free from error or reflective of recent changes to the product. Tools and programs have the opportunity to be updated annually or upon request.
