aimswebPlus Reading
Letter Word Sounds Fluency

Descriptive Information

Please provide a description of your tool:

aimswebPlus is a brief and valid assessment system for screening and monitoring reading and math skills for all students in Kindergarten through Grade 8. Normative data were collected in 2013-14 on a combination of fluency measures that are sensitive to growth and new standards-based assessments of classrooms skills. The resulting scores and reports inform instruction and help improve student performance. Letter Word Sounds Fluency is individually administered, with a teacher/examiner recording student data during the test session. Once testing is complete, summary and detailed reports for students, classrooms, and districts can be generated immediately.

Is your tool designed to measure progress towards an end-of-year goal (e.g., oral reading fluency) or progress towards a short-term skill (e.g., letter naming fluency)?

End-year goal
Short-term skill

The tool is intended for use with the following grade(s).

Preschool / Pre - kindergarten
Kindergarten
First grade
Second grade
Third grade
Fourth grade
Fifth grade
Sixth grade
Seventh grade
Eighth grade
Ninth grade
Tenth grade
Eleventh grade
Twelfth grade

The tool is intended for use with the following age(s).

0-4 years old
5 years old
6 years old
7 years old
8 years old
9 years old
10 years old
11 years old
12 years old
13 years old
14 years old
15 years old
16 years old
17 years old
18 years old

The tool is intended for use with the following student populations.

Students in general education
Students with disabilities
English language learners

ACADEMIC ONLY: What dimensions does the tool assess?

Reading

Global Indicator of Reading Competence
Listening Comprehension
Vocabulary
Phonemic Awareness
Decoding
Passage Reading
Word Identification
Comprehension

Spelling & Written Expression

Global Indicator of Spelling Competence
Global Indicator of Writting Expression Competence

Mathematics

Global Indicator of Mathematics Comprehension
Early Numeracy
Mathematics Concepts
Mathematics Computation
Mathematics Application
Fractions
Algebra

Other

Please describe specific domain, skills or subtests:

BEHAVIOR ONLY: Please identify which broad domain(s)/construct(s) are measured by your tool and define each sub-domain or sub-construct.

BEHAVIOR ONLY: Which category of behaviors does your tool target?

Acquisition and Cost Information

Administration

BEHAVIOR ONLY: What type of administrator is your tool designed for?

General education teacher
Special education teacher
Parent
Child
External observer
Other

If other, please specify:

BEHAVIOR ONLY: What is the administration format?

Direct observation
Rating scale
Checklist
Performance measure
Other

If other, please specify:

BEHAVIOR ONLY: What is the administration setting?

General education classroom
Special education classroom
School office
Recess
Lunchroom
Home
Other

If other, please specify:

Does the program require technology?

Yes

If yes, what technology is required to implement your program? (Select all that apply)

Computer or tablet
Internet connection
Other technology (please specify)

What is the administration context?

Individual

Small group   If small group, n=

Large group   If large group, n=

Computer-administered
Other

If other, please specify:

ACADEMIC ONLY: What are the discontinue rules?

No discontinue rules provided
Basals
Ceilings
Other

If other, please specify:

Discontinue if student incorrectly names 10 consecutive letters, without self-correcting.

Training & Scoring

Training

Is training for the administrator required?

Yes

Describe the time required for administrator training, if applicable:

Less than one hour of training.

Please describe the minimum qualifications an administrator must possess.

Paraprofessionals and professionals.

No minimum qualifications

Are training manuals and materials available?

Yes

Are training manuals/materials field-tested?

Yes

Are training manuals/materials included in cost of tools?

Yes

If No, please describe training costs:

Can users obtain ongoing professional and technical support?

Yes

If Yes, please describe how users can obtain support:

Pearson provides phone and email-based support, as well as a user group forum that facilitates the asking and answering of questions.

Scoring

BEHAVIOR ONLY: What types of scores result from the administration of the assessment?

Score

Observation	Behavior Rating
Frequency Duration Interval Latency	Raw score

Conversion

Observation	Behavior Rating
Rate Percent	Standard score Subscale/ Subtest Composite Stanine Percentile ranks Normal curve equivalents IRT based scores

Interpretation

Observation	Behavior Rating
Error analysis Peer comparison Rate of change	Dev. benchmarks Age-Grade equivalent

How are scores calculated?

Manually (by hand)
Automatically (computer-scored)
Other

If other, please specify:

What is the basis for calculating performance level and percentile scores?

Age norms
Grade norms
Classwide norms
Schoolwide norms
Stanines
Normal curve equivalents

What types of performance level scores are available?

Raw score
Standard score
Percentile score
Grade equivalents
IRT-based score
Age equivalents
Stanines
Normal curve equivalents
Developmental benchmarks
Developmental cut points
Equated
Probability
Lexile score
Error analysis
Composite scores
Subscale/subtest scores
Other

If other, please specify:

Please describe the scoring structure. Provide relevant details such as the scoring format, the number of items overall, the number of items per subscale, what the cluster/composite score comprises, and how raw scores are calculated.

Letter Word Sounds Fluency assesses a student's ability to make letter sounds, make the sounds of two-letter combinations, and read aloud consonant-vowel-consonant (CVC) words. This task is similar to the general developmental progression from letter-sound correspondence to oral word reading. Performance is reported on the raw number correct score. Raw scores are calculated by subtracting the number of errors from the number of items attempted, resulting in a total number correct raw score. Raw scores for LWSF include number of letter or word sounds made correctly in 60 seconds.

Do you provide basis for calculating slope (e.g., amount of improvement per unit in time)?

Yes

ACADEMIC ONLY: Do you provide benchmarks for the slopes?

Yes

ACADEMIC ONLY: Do you provide percentile ranks for the slopes?

Yes

What is the basis for calculating slope and percentile scores?

Age norms
Grade norms
Classwide norms
Schoolwide norms
Stanines
Normal curve equivalents

Describe the tool’s approach to progress monitoring, behavior samples, test format, and/or scoring practices, including steps taken to ensure that it is appropriate for use with culturally and linguistically diverse populations and students with disabilities.

Letter Word Sounds Fluency is an individually administered measure with printed content shown to students via a stimulus book. The examiner records student responses during the test session via digital record form accessed by a computer, tablet, or other mobile device. A sample item page and test blueprint information are available from the Center upon request. ● Overview: The student says the sounds of visually presented letters, syllables, and words for 1 minute. ● Test Format: individually-administered, timed ● Test Content: The student is presented with 45 letters and 10 three-letter words, each presented as an initial sound, a consonant-vowel syllable, and then the consonant- vowel-consonant word these word parts form. ● 13 unique forms, 3 benchmark and 10 progress monitoring; PM testing conducted at teacher-determined intervals ● Score: number of correct letter and word sounds made in 1 minute ● Time limit: 1 minute Student-facing content contains only numbers. Instructional text spoken by the examiner was written using simple, grade-appropriate language that keeps the students’ receptive language load to a minimum. LWSF scoring rules do not penalize for regional, dialectical, or articulation errors. Some letters that can be particularly difficult to pronounce in isolation (e.g., g, c, j, y, l) have been omitted from LWSF.

Rates of Improvement and End of Year Benchmarks

Is minimum acceptable growth (slope of improvement or average weekly increase in score by grade level) specified in your manual or published materials?

Yes

If yes, specify the growth standards:

aimswebPlus provides student growth percentiles (SGP) by grade and initial performance level (Fall and Winter) for establishing growth standards. An SGP indicates the percentage of students in the national sample whose seasonal (or annual) rate of improvement (ROI) fell at or below a specified ROI. Separate SGP distributions are computed for each of five levels of initial (Fall or Winter) performance. Goals are set in the system by selecting the measure and baseline score, the goal date, the monitoring frequency (default is weekly), and the goal score. When the user defines the goal score, the system automatically labels the ambitiousness of the goal. The rate of improvement needed to achieve the goal is computed and translated into an SGP. An SGP < 50 is considered Insufficient; an SGP between 50 and 85 is considered Closes the Gap; an SGP between 85 and 97 is considered Ambitious; and an SGP > 97 is considered Overly Ambitious. aimswebPlus recommends setting performance goals that represent rates of growth between the 85th and 97th SGP. However, the user ultimately determines what growth rate is appropriate on an individual basis.

Are benchmarks for minimum acceptable end-of-year performance specified in your manual or published materials?

Yes

If yes, specify the end-of-year performance standards:

aimswebPlus allows users to select a target from a range of end-of-year targets the one that is most appropriate for their instructional needs. aimswebPlus defines a meaningful target as one that is objective, quantifiable, and can be linked to a criterion that has inherent meaning for teachers. To establish a meaningful performance target using aimswebPlus tiers, the account manager (e.g., a school/district administrator) is advised to choose a target that: ● is linked to a criterion, ● is challenging and achievable, ● closes the achievement gap, and ● reflects historical performance results (when available). Customers are also advised to give consideration to the availability of resources to achieve the goal. The targets are based on spring reading or math composite score national percentiles. Twelve national percentile targets ranging from the 15th through the 70th percentile, in increments of 5 are provided. This range was chosen because it covers the breadth of passing rates on state assessments and the historical range of targets our customers typically use. The system provides a default spring performance target of the 30th national percentile. Targets can be set separately for Reading and Math. The aimswebPlus Tiers Guide provides more detail to help customers define a high quality performance target. It also provides a step-by-step method to align spring performance targets to performance levels on state accountability tests. Once a target is selected, the aimswebPlus system automatically identifies the fall (or winter) cut score that divides the score distribution into three instructional Tiers. Students above the highest cut score are in Tier 1 and have a high probability (80%–95%) of meeting the performance target; students between the upper and lower cut scores are in Tier 2 and have a moderate probability (40%–70%) of meeting the performance target; and students below the lower cut score are in Tier 3 and have a low probability (10%–40%) of meeting the performance target. The system recommends that a progress monitoring schedule be defined for any student below the 25th national percentile in a given season, or in Tiers 2 or 3.

What is the basis for specifying minimum acceptable growth and end of year benchmarks?

Norm-referenced
Criterion-referenced
Other

If other, please specify:

False

National representation (check all that apply):

Northeast:

New England

Middle Atlantic

Midwest:

East North Central

West North Central

South:

South Atlantic

East South Central

West South Central

West:

Mountain

Pacific

Local representation (please describe, including number of states)

11 states, across all 4 U.S. regions

Date

2013-14

Size

2000

Gender (Percent)

Male

50

Female

50

Unknown

SES indicators (Percent)

Eligible for free or reduced-price lunch

Other SES Indicators

Based on schoolwide eligibility for free or reduced lunch, students were sorted into Low (1-32% eligible), Moderate (33-66% eligible), and High (67-100% eligible) SES categories. Students were distributed fairly evenly among the three SES levels.

Race/Ethnicity (Percent)

White, Non-Hispanic

51

Black, Non-Hispanic

14

Hispanic

25

American Indian/Alaska Native

Asian/Pacific Islander

Other

10 (includes American-Indian/Alaska Native and Asian/Pacific Islander)

Unknown

Disability classification (Please describe)

The norm sample includes all students in the classroom with exceptions for moderate to severe intellectual disability; blind or deaf; or moderate to severe motor coordination disability.

First language (Please describe)

Language proficiency status (Please describe)

ELL (Percent): 10

Do you provide, in your user’s manual, norms which are disaggregated by race or ethnicity? If so, for which race/ethnicity?

White, Non-Hispanic
Black, Non-Hispanic
Hispanic
American Indian/Alaska Native
Asian/Pacific Islander
Other
Unknown

If criterion-referenced, describe procedure for specifying criterion for adequate growth and benchmarks for end-of-year performance levels.

Describe any other procedures for specifying adequate growth and minimum acceptable end of year performance.

To get the most value from progress monitoring, aimswebPlus recommends the following: (1) establish a time frame, (2) determine the level of performance expected, and (3) determine the criterion for success. Typical time frames include the duration of the intervention or the end of the school year. An annual time frame is typically used when IEP goals are written for student who are receiving special education services. For example, aimswebPlus goals can be written as follows: In 34 weeks, the student will compare numbers and answer computational problems to earn a score of 30 points on Grade 4 Number Sense Fluency forms. aimswebPlus provides several ways to define a level of expected performance. The goal can be based on: ● well-established performance benchmarks that can be linked to aimswebPlus measures via national percentiles (e.g., the link to state test performance levels) or total score (e.g., word read per minute in Grade 2); ● a national performance norm benchmark (e.g., the 50th national percentile is often used to indicate on-grade level performance); ● a local performance norm benchmark; ● or an expected or normative rate of improvement (ROI) such as the 85th national student growth percentile. When customers choose normative ROIs, aimswebPlus uses student growth percentiles to describe these normative rates of improvement. Within the aimswebPlus software, the user enters the goal date and moves a digital slider to the desired ROI. As the slider moves, it provides feedback about the strength of the ROI: Insufficient, Closes the Gap, Ambitious, or Overly Ambitious. Users are encouraged to use the Ambitious (85th–97th SGP) for students in need of intensive intervention.

Reliability

Grade	Kindergarten
Rating

Legend

Convincing evidence

Partially convincing evidence

Unconvincing evidence

Data unavailable

^dDisaggregated data available

*Offer a justification for each type of reliability reported, given the type and purpose of the tool.

Alternate-form reliability, where equivalent forms are administered close together in time, is highly appropriate for progress monitoring CBM measures because it shows the consistency of scores from independently timed administrations with different content. Internal consistency reliability is not appropriate for speeded CBM measures. The stability coefficient, where equivalent forms are administered with an interval of several months, reflects additional measurement error due to true change over time. As a result, these reliabilities are generally lower. The alternate-form stability coefficient is based on correlations between fall-winter and winter-spring benchmark scores.

*Describe the sample(s), including size and characteristics, for each reliability analysis conducted.

The concurrent alternate-form reliability sample is based on 10 schools from across the U.S. representing each of three SES levels (described above). Participating schools administered the alternate forms to all Kindergarten students in the school, with few exceptions for moderate to severe intellectual disabilities. Each student completed 2 of 3 alternate forms with forms administered in pairs: 1,2; 1,3; and 2, 3. The number of students completing each pair ranged from 90–217. The table reports the median reliability coefficient.

*Describe the analysis procedures for each reported type of reliability.

Pearson correlation coefficients of the scores from alternate-forms.

*In the table(s) below, report the results of the reliability analyses described above (e.g., model-based evidence, internal consistency or inter-rater reliability coefficients). Include detail about the type of reliability data, statistic generated, and sample size and demographic information.

Results from other forms of reliability analysis not compatible with above table format:

Manual cites other published reliability studies:

No

Provide citations for additional published studies.

Do you have reliability data that are disaggregated by gender, race/ethnicity, or other subgroups (e.g., English language learners, students with disabilities)?

No

If yes, fill in data for each subgroup with disaggregated reliability data.

Results from other forms of reliability analysis not compatible with above table format:

Manual cites other published reliability studies:

No

Provide citations for additional published studies.

Validity

Grade	Kindergarten
Rating

Legend

Convincing evidence

Partially convincing evidence

Unconvincing evidence

Data unavailable

^dDisaggregated data available

*Describe each criterion measure used and explain why each measure is appropriate, given the type and purpose of the tool.

Two validity studies are reported: a concurrent study and a predictive study of Letter-Word Sounds Fluency (LWSF) with Word Reading Fluency (WRF). WRF, like LWSF is an aimswebPlus timed fluency measure. However, unlike LWSF which measures the students fluency and proficiency saying letter sounds and decoding CVC words, WRF measures fluency and proficiency with wide range of sight words. The content of WRF does not overlap with LWSF. Moreover, the validity coefficients reported are based on the initial administration of each measure in the aimswebPlus benchmark system. As such, there is no practice/exposure effect impacting the validity coefficients.

*Describe the sample(s), including size and characteristics, for each validity analysis conducted.

Criterion: WRF ORF N: 2000 2000 Female: 50% 50% Male: 50% 50% Black: 14% 14% Hispanic: 25% 25% White: 51% 51% Other: 10% 10% ELL: 10% 10% 68–100% FRPL:32% 32% 34–67% FRPL: 33% 33% 0–33% FRPL: 35% 35%

*Describe the analysis procedures for each reported type of validity.

Predictive validity is based on winter LWSF scores and spring WRF scores. LWSF is first used in aimswebPlus benchmarking in the winter of Kindergarten; whereas WRF is introduced in the spring. The concurrent correlation is based on spring LWSF and WRF scores.

*In the table below, report the results of the validity analyses described above (e.g., concurrent or predictive validity, evidence based on response processes, evidence based on internal structure, evidence based on relations to other variables, and/or evidence based on consequences of testing), and the criterion measures.

Results from other forms of validity analysis not compatible with above table format:

Manual cites other published reliability studies:

No

Provide citations for additional published studies.

Describe the degree to which the provided data support the validity of the tool.

Letter Word Sounds Fluency (LWSF) is designed to measure fluency making letter sounds, making the sounds of two-letter combinations, and reading aloud consonant-vowel-consonant (CVC) words, a foundational skill important in the developmental progression from letter-sound correspondence to oral word reading. These validity studies support the interpretation of LWSF scores as foundational for success in the more advanced early literacy skills. Furthermore, they demonstrate that performance on Letter Word Sounds Fluency has a strong relationship with initial winter letter sounds and word building skills.

Do you have validity data that are disaggregated by gender, race/ethnicity, or other subgroups (e.g., English language learners, students with disabilities)?

No

If yes, fill in data for each subgroup with disaggregated validity data.

Results from other forms of validity analysis not compatible with above table format:

Manual cites other published reliability studies:

No

Provide citations for additional published studies.

Bias Analysis

Have you conducted additional analyses related to the extent to which your tool is or is not biased against subgroups (e.g., race/ethnicity, gender, socioeconomic status, students with disabilities, English language learners)? Examples might include Differential Item Functioning (DIF) or invariance testing in multiple-group confirmatory factor models.

No

If yes,

a. Describe the method used to determine the presence or absence of bias:

Note. Letter Word Sounds Fluency measures recognition of the letters and simple (CVC) words and does not require the same kind of bias analyses as more vocabulary- and context-heavy assessments (e.g., math word problems, reading comprehension). Instructional text spoken by the examiner was written using simple, grade-appropriate language that keeps the students’ receptive language load to a minimum.

b. Describe the subgroups for which bias analyses were conducted:

c. Describe the results of the bias analyses conducted, including data and interpretative statements. Include magnitude of effect (if available) if bias has been identified.

Sensitivity: Reliability of Slope

Grade	Kindergarten
Rating

Legend

Convincing evidence

Partially convincing evidence

Unconvincing evidence

Data unavailable

^dDisaggregated data available

Describe the sample, including size and characteristics. Please provide documentation showing that the sample was composed of students in need of intensive intervention. A sample of students with intensive needs should satisfy one of the following criteria: (1) all students scored below the 30th percentile on a local or national norm, or the sample mean on a local or national test fell below the 25th percentile; (2) students had an IEP with goals consistent with the construct measured by the tool; or (3) students were non-responsive to Tier 2 instruction. Evidence based on an unknown sample, or a sample that does not meet these specifications, may not be considered.

The sample consisted of 1,441 Kindergarten students below the 25th national percentile on the fall Letter-Word Sounds Fluency (LWSF) benchmark and who were assigned a reading performance goal and receiving frequent progress monitoring with LWSF. All progress monitoring schedules were at least 20 weeks in duration during the 2016–17 school year, and students with fewer than 20 weeks of progress monitoring were eliminated.

Describe the frequency of measurement (for each student in the sample, report how often data were collected and over what span of time).

The interval between the first and last administration was a minimum of 10 weeks. Most administrations occurred weekly, with a small percentage conducted twice monthly. Students with fewer than 20 weeks of progress monitoring were eliminated.

Describe the analysis procedures.

Each student’s progress monitoring administrations were sequenced by date and divided into two groups: odd numbered administrations (e.g, 1,3,5, etc) and even numbered administrations (e.g., 2,4,6, etc). Linear regression was used to compute the slope for each student by group. The following model was used: Scorei = Intercept + Datei where Date is the amount of time since the start of progress monitoring and i ranges from 1 to the number of administrations. The correlation between odd-group and even-group slopes across all students was computed and converted to a split-half reliability coefficient using the Spearman-Brown Formula: 2r/(1+r).

In the table below, report reliability of the slope (e.g., ratio of true slope variance to total slope variance) by grade level (if relevant).

Results from other forms of reliability analysis not compatible with above table format:

Manual cites other published reliability studies:

No

Provide citations for additional published studies.

Do you have reliability of the slope data that is disaggregated by subgroups (e.g., race/ethnicity, gender, socioeconomic status, students with disabilities, English language learners)?

No

If yes, fill in data for each subgroup with disaggregated reliability of the slope data.

Results from other forms of reliability analysis not compatible with above table format:

Manual cites other published reliability studies:

No

Provide citations for additional published studies.

Sensitivity: Validity of Slope

Grade	Kindergarten
Rating

Legend

Convincing evidence

Partially convincing evidence

Unconvincing evidence

Data unavailable

^dDisaggregated data available

Describe each criterion measure used and explain why each measure is appropriate, given the type and purpose of the tool.

Word Reading Fluency (WRF) served as the criterion measure. WRF measures proficiency and fluency reading sight words. It is a timed 1 minute measure and is an appropriate criterion measure for LWSF because it builds directly on a student’s proficiency with using letter sounds and building simple CVC words. Word reading skills is also what most of kindergarten literacy instruction builds towards. WRF is first used in aimswebPlus in the spring Kindergarten benchmark. Thus, there is no overlapping practice or exposure effect of WRF with LWSF.

Describe the sample(s), including size and characteristics. Please provide documentation showing that the sample was composed of students in need of intensive intervention. A sample of students with intensive needs should satisfy one of the following criteria: (1) all students scored below the 30th percentile on a local or national norm, or the sample mean on a local or national test fell below the 25th percentile; (2) students had an IEP with goals consistent with the construct measured by the tool; or (3) students were non-responsive to Tier 2 instruction. Evidence based on an unknown sample, or a sample that does not meet these specifications, may not be considered.

The sample is the same as that used to compute the reliability of the slope.

Describe the frequency of measurement (for each student in the sample, report how often data were collected and over what span of time).

The interval between the first and last administration was a minimum of 11 weeks. Most administrations occurred weekly, with a small percentage conducted twice monthly.

Describe the analysis procedures for each reported type of validity.

Spring benchmark WRF scores were regressed onto the Winter to Spring PM slope for LWSF and the Winter LWSF benchmark scores. Including Winter LWSF scores controls for differences in initial performance, thus removing its effect on the relationship between slope and outcome. Standardized regression coefficients and associated standard errors are reported in the table below. Model 1a: WRF Spring Score = Intercept + LWSF slope + LWSF Winter Score.

In the table below, report predictive validity of the slope (correlation between the slope and achievement outcome) by grade level (if relevant).
NOTE: The TRC suggests controlling for initial level when the correlation for slope without such control is not adequate.

Results from other forms of reliability analysis not compatible with above table format:

Manual cites other published validity studies:

Provide citations for additional published studies.

Describe the degree to which the provided data support the validity of the tool.

Letter Word Sounds Fluency (LWSF) is designed to measure fluency making letter sounds, making the sounds of two-letter combinations, and reading aloud consonant-vowel-consonant (CVC) words, a foundational skill important in the developmental progression from letter-sound correspondence to oral word reading. These validity studies support the interpretation of LWSF scores as foundational for success in the more advance early literacy skills. Furthermore, they demonstrate that performance on Letter Word Sounds Fluency has a strong relationship with initial winter letter sounds and word building skills.

Do you have validity of the slope data that is disaggregated by subgroups (e.g., race/ethnicity, gender, socioeconomic status, students with disabilities, English language learners)?

No

If yes, fill in data for each subgroup with disaggregated validity of the slope data.

Results from other forms of reliability analysis not compatible with above table format:

Manual cites other published validity studies:

No

Provide citations for additional published studies.

Alternate Forms

Grade	Kindergarten
Rating

Legend

Convincing evidence

Partially convincing evidence

Unconvincing evidence

Data unavailable

^dDisaggregated data available

Describe the sample for these analyses, including size and characteristics:

The sample consisted of 6,732 students from 281 schools each with a reading performance goal and a progress monitoring schedule who scored at or below the 30th national percentile on the spring LWSF benchmarks form. Each student completed at least one of the alternate LWSF PM forms within a window from 5 to 35 days after benchmark testing. Forms were randomly assigned to students. Because this test interval was fairly brief and students were randomly assigned to forms, comparability is based on a direct comparison of the sample means.

What is the number of alternate forms of equal and controlled difficulty?

Ten alternate forms were evaluated. LWSF has 10 PM forms because it is designed to be used to measure progress over one-half of a school year. Our prior research showed that many Kindergarten students are not ready for a task that involves saying letter sounds and decoding CVC words in the fall of Kindergarten. The research also shows that most students have mastered the skill be the spring of Kindergarten. Thus, the optimal time frame to measure this skill is from the winter to spring of Kindergarten. The average performance on the forms, administered in a 30 day window is the basis of form comparability. To demonstrate comparability we provide the effect size as the mean difference between each form and the average difficulty across all forms in standard deviations units. (X_i-μ)/SD The means ES is 0.08, and all 10 effect sizes are below 0.30, which is considered small. Comparability of the entire set of 10 forms is also summarized using analysis of variance where Form is treated as a fixed factor. The results indicate that Form accounts for only 0.89% of the total score variance. This is a very small percent and will have a trivial effect on the growth slope over the 10 or so administrations that are common for progress monitoring.

If IRT based, provide evidence of item or ability invariance

If computer administered, how many items are in the item bank for each grade level?

If your tool is computer administered, please note how the test forms are derived instead of providing alternate forms:

Decision Rules: Setting & Revising Goals

Grade	Kindergarten
Rating

Legend

Convincing evidence

Partially convincing evidence

Unconvincing evidence

Data unavailable

^dDisaggregated data available

In your manual or published materials, do you specify validated decision rules for how to set and revise goals?

Yes

If yes, specify the decision rules:

To get the most value from progress monitoring, aimswebPlus recommends the following: (1) establish a time frame, (2) determine the level of performance expected, and (3) determine the criterion for success. Typical time frames include the duration of the intervention or the end of the school year. An annual time frame is typically used when IEP goals are written for students who are receiving special education services. For example, aimswebPlus goals can be written as follows: In 34 weeks, the student will compare numbers and answer computational problems to earn a score of 30 points on Grade 4 Number Sense Fluency forms. aimswebPlus provides several ways to define a level of expected performance. The goal can be based on: ● well-established performance benchmarks that can be linked to aimswebPlus measures via national percentiles (e.g., the link to state test performance levels) or total score (e.g., word read per minute in Grade 2); ● a national performance norm benchmark (e.g., the 50th national percentile is often used to indicate on-grade level performance); ● a local performance norm benchmark; ● or an expected or normative rate of improvement (ROI), such as the 85th national student growth percentile. To use this last method (student growth percentile), the user begins by selecting the measure and baseline score, the goal date, the monitoring frequency (default is weekly), and a tentative goal score. The system automatically labels the ambitiousness of the goal as Insufficient (SGP below 50), Closes the Gap (SGP between 50 and 85), Ambitious (86 to 97), or Overly Ambitious (above 97). The user can then adjust the goal (or the goal date) in light of this feedback. For students in need of intensive intervention, aimswebPlus recommends setting performance goals that represent rates of growth between the 86th and 97th SGP (Ambitious). An SGP of 86 represents a growth rate achieved by just 15% of the national sample, which is why it is considered ambitious. However, it is reasonable to expect significantly higher than average growth when implementing effective, intensive intervention. If the goal is set according to a benchmark based on raw scores or national or local norms, the aimswebPlus system still labels the ambitiousness of the goal in one of the four levels described above. If the goal corresponds to an Insufficient or Overly Ambitious rate of growth, users are advised to consider adjusting the goal. However, the user ultimately determines what growth rate is required on an individual basis. With respect to the decision to revise a goal, aimswebPlus provides empirically-based feedback about the student’s progress relative to the initial goal using the statistical tool described in our response to question B5 below. If the projected score at the goal date is fully Above Target (ie., the 75% confidence interval for the student’s projected score at the goal date is entirely above the goal score), we recommend that the user consider raising the goal if the goal date is at least 12 weeks out. Otherwise, we recommend not changing the goal. On the other hand, if the upper end of the confidence interval on the projected score lies Below Target, we recommend either changing the intervention, increasing its intensity, or lowering the goal if the initial goal was Overly Ambitious.

What is the evidentiary basis for these decision rules?
NOTE: The TRC expects evidence for this standard to include an empirical study that compares a treatment group to a control and evaluates whether student outcomes increase when decision rules are in place.

As described above, the users have flexibility in the method they use to set and revise goals in aimswebPlus. The SGP-based labeling of goals as Overly Ambitious, Ambitious, Closes the Gap, or Insufficient is intended to assist the user in choosing a goal, but is not an automatic goal-setting system. Likewise, the analytical system that generates generates a confidence interval for the student'[s predicted performance at the goal date helps the user manage progress monitoring but does no make a decision about revising the goal. Certainly, a decision to lower a goal would rely primarily on the educator's judgement, since the first consideration would be to change the intervention. No experiment has been conducted in which the aimswebPlus information related to setting and revision goals was provided for some students receiving intensive intervention but not others.

Decision Rules: Changing Instruction

Grade	Kindergarten
Rating

Legend

Convincing evidence

Partially convincing evidence

Unconvincing evidence

Data unavailable

^dDisaggregated data available

In your manual or published materials, do you specify validated decision rules for when changes to instruction need to be made?

Yes

If yes, specify the decision rules:

aimswebPlus applies a statistical procedure, based on linear regression, to the student’s progress monitoring scores in order to provide empirically-based guidance about whether the student is likely to meet, fall short of, or exceed his/her goal. The calculation procedure (presented below) is fully described in the aimswebPlus Progress Monitoring Guide (Pearson, 2017). aimswebPlus users will not have to do any calculations—the online system does this automatically. The decision rule is based on a 75% confidence interval for the student’s predicted score at the goal date. This confidence interval is student-specific and takes into account the number and variability of progress monitoring scores and the duration of monitoring. Starting at the sixth week of monitoring (when there are at least four monitoring scores), the aimswebPlus report following each progress monitoring administration includes one of the following statements: A. Below Target. Projected to not meet the goal. This statement appears if the confidence interval is completely below the goal score. B. Above Target. Projected to meet or exceed the goal. This statement appears if the confidence interval is completely above the goal score. C. Near Target. Projected score at goal date: Between (X) and (Y). This statement appears if the confidence interval includes the goal score, with X and Y indicating the bottom and top of the confidence interval, respectively. If Statement A appears, the user has a sound basis for deciding that the current intervention is not sufficient and a change to instruction should be made. If Statement B appears, there is an empirical basis for deciding that the goal is not sufficiently challenging and should be increased. If Statement C appears, the student’s progress is not clearly different from the aimline, so there is not a compelling reason to change the intervention or the goal; however, the presentation of the confidence-interval range enables the user to see whether the goal is near the upper limit or lower limit of the range, which would signal that the student’s progress is trending below or above the goal. A 75% confidence interval was chosen for this application because it balances the costs of the two types of decision errors. Incorrectly deciding that the goal will not be reached (when in truth it will be reached) has a moderate cost: an intervention that is working will be replaced by a different intervention. Incorrectly deciding that the goal may be reached (when in truth it will not be reached) also has a moderate cost: an ineffective intervention will be continued rather than being replaced. Because both kinds of decision errors have costs, it is appropriate to use a modest confidence level. Calculation of the 75% confidence interval for the score at the goal date Calculate the trend line. This is the ordinary least-squares regression line through the student’s monitoring scores. Calculate the projected score at the goal date. This is the value of the trend line at the goal date. Calculate the standard error of estimate (SEE) of the projected score at the goal date, using the following formula: 〖SEE〗_(predicted score)= √((∑_i^k 〖(y_i-〖ý〗_i)〗^2)/(k-2))×√(1+1/k+〖(GW-(∑_1^k w_i)/k)〗^2/(∑_i^k 〖(w_i-(∑_1^k w_i)/k)〗^2 )) where k = number of completed monitoring administrations, w = week number of a completed administration, GW = week number of the goal date, y = monitoring score, y’ = predicted monitoring score at that week (from the student’s trendline). The means and sums are calculated across all of the completed monitoring administrations up to that date. Add and subtract 1.25 times the SEE to the projected score, and round to the nearest whole numbers.

What is the evidentiary basis for these decision rules?
NOTE: The TRC expects evidence for this standard to include an empirical study that compares a treatment group to a control and evaluates whether student outcomes increase when decision rules are in place.

The decision rules are statistically rather than empirically based. The guidance statements that result from applying the 75% confidence interval to the projected score are correct and probabilistic statements, under the certain assumptions that: the student's progress to data can be described by a linear trend line. If the pattern of the students' monitoring scores is obviously curvilinear, then the projected score based on a linear trend will likely be misleading. We provide training in the aimswebPlus Progress Monitoring Guide about the need for users to take nonlinearity into account when interpreting progress-monitoring data. Another assumption is that the student will continue to progress at the same rate as they have be progressing to that time. This is an unavoidable assumption for a decision system based on extrapolating from past growth. No controlled experimental study has been conduced to support the decision rules, however, an empirical study of actual progress monitoring results was undertaken to evaluate the accuracy of the decision rules as various points during the progress monitoring schedule. aimswebPlus Number Sense Fluency (NSF) and Oral Reading Fluency (ORF) progress monitoring data collected during the 2016-17 school year was used to evaluate the accuracy of the decision feedback. All students on a PM schedule who scored below the 0th national percentile on the fall benchmark and who had at least 20 PM administrations were included. Grades 2 and 3 were chosen. More than 1000 student's scores were used in each grade. Most administrations we collected about weekly. Because we did not have the student's actual goal score we generated a goal score based on the ROI that corresponds to a student growth percentile of 55. This level was chosen because it represents an average rate of improvement and it resulted in about 50% of the students meeting the goal. The goal score was computed as follows: Fall Benchmark Score + ROI55*Weeks. Where ROI55 is the ROI associated with the SGP of 55 and Weeks is the number of weeks from the baseline score (Fall Benchmark) and the Spring Benchmark. For each student, beginning with the 8th score and going through the last score, we computed the score feedback based on the rules described in the previous section. If the student was projected to be below target an intervention change was deemed necessary and coded 1. Otherwise, the student was assigned a score of zero for that administration (no change is needed). We computed the accuracy of the decision to change interventions by comparing the decision to whether the student ultimately did not meet the goal score by the Spring Benchmark. Accuracy was computed as the percentage of the decisions to change intervention of all students who did not ultimately meet the goal. The results showed that decision accuracy improved with each successive administration with 70% - 75% accuracy by the 8th administration and 75% - 80% by the 15th administration and 90% by the 20th administration. This trend was replicated in each sample and it provides evidence that the decision rules validly indicate when a change in the intervention should be made because the student is unlikely to achieve the goal with the current rate of improvement.