Phonological Awareness Literacy Screening (PALS)
PALS for Grades 1-3

Descriptive Information

Please provide a description of your tool:

The Phonological Awareness Literacy Screening (PALS) is a criterion-referenced screening, diagnostic, and progress monitoring tool. PALS consists of two instruments (PALS-K and PALS 1-3) that measure young children’s knowledge of important literacy fundamentals: phonological awareness, alphabet awareness, letter sound knowledge, spelling, concept of word, word recognition in isolation, and oral passage reading. The major purpose of PALS is to identify students who may be at risk for reading difficulties and who need additional reading instruction beyond what is provided to typically developing readers. The second purpose of PALS is to provide teachers with explicit diagnostic information about what students know and need to know about the fundamental components of literacy that may be used to target instruction to meet students’ needs. The third purpose of PALS is to monitor students’ progress and determine the effectiveness of instruction or intervention. PALS 1-3 PALS 1-3 uses a gated three-tiered approach that differentiates the administration of the assessment based on students’ scores. The first tier (or Entry Level) includes a screening measure of word recognition and spelling. The Entry Level also assesses the accuracy, fluency, rate, and comprehension of a student’s oral reading in context. If students do not meet the Entry Level Summed Score Benchmark, which is based on the screening tasks, they are routed to Level B for further diagnosis. Level B (Alphabetics) assesses emergent and beginning reading essentials in alphabet knowledge and concept of word. If the Level B Summed Score Benchmark is not met, students are routed to Level C (Phonemic Awareness) for a more in-depth evaluation of phonemic awareness skills, including blending and segmenting speech sounds.

The tool is intended for use with the following grade(s).

Preschool / Pre - kindergarten
Kindergarten
First grade
Second grade
Third grade
Fourth grade
Fifth grade
Sixth grade
Seventh grade
Eighth grade
Ninth grade
Tenth grade
Eleventh grade
Twelfth grade

The tool is intended for use with the following age(s).

0-4 years old
5 years old
6 years old
7 years old
8 years old
9 years old
10 years old
11 years old
12 years old
13 years old
14 years old
15 years old
16 years old
17 years old
18 years old

The tool is intended for use with the following student populations.

Students in general education
Students with disabilities
English language learners

ACADEMIC ONLY: What skills does the tool screen?

Reading

Phonological processing:

RAN
Memory
Awareness
Letter sound correspondence
Phonics
Structural analysis

Word ID

Accuracy
Speed

Nonword

Accuracy
Speed

Spelling

Accuracy
Speed

Passage

Accuracy
Speed

Reading comprehension:

Multiple choice questions
Cloze
Constructed Response
Retell
Maze
Sentence verification
Other (please describe):

Listening comprehension:

Multiple choice questions
Cloze
Constructed Response
Retell
Maze
Sentence verification
Vocabulary
Expressive
Receptive

Mathematics

Global Indicator of Math Competence

Accuracy
Speed
Multiple Choice
Constructed Response

Early Numeracy

Accuracy
Speed
Multiple Choice
Constructed Response

Mathematics Concepts

Accuracy
Speed
Multiple Choice
Constructed Response

Mathematics Computation

Accuracy
Speed
Multiple Choice
Constructed Response

Mathematic Application

Accuracy
Speed
Multiple Choice
Constructed Response

Fractions/Decimals

Accuracy
Speed
Multiple Choice
Constructed Response

Algebra

Accuracy
Speed
Multiple Choice
Constructed Response

Geometry

Accuracy
Speed
Multiple Choice
Constructed Response

Other (please describe):

Please describe specific domain, skills or subtests:

BEHAVIOR ONLY: Which category of behaviors does your tool target?

Internalizing
Externalizing
Internalizing and Externalizing

BEHAVIOR ONLY: Please identify which broad domain(s)/construct(s) are measured by your tool and define each sub-domain or sub-construct.

Acquisition and Cost Information

Where to obtain:

Email Address

pals@virginia.edu

Address

UVA/PALS PO Box 800785, Charlottesville, VA 22908-8785

Phone Number

888-UVA-PALS

Website

pals.virginia.edu

Initial cost for implementing program:

Cost

$5.50

Unit of cost

student

Replacement cost per unit for subsequent use:

Cost

Unit of cost

Duration of license

1 year

Additional cost information:

Describe basic pricing plan and structure of the tool. Provide information on what is included in the published tool, as well as what is not included but required for implementation.

*It is free for all teachers/students in public schools in VA K-3 and all children in PreK with state funding. Pricing below reflects IO Education/Illuminate pricing for all schools outside of VA or private schools in VA Materials include an Introduction and Overview, an Administration and Scoring Guide, teacher materials, student materials, information regarding interpreting PALS-K results, and a PALS-K Technical Reference.

Provide information about special accommodations for students with disabilities.

Published Tools Materials include an Introduction and Overview, an Administration and Scoring Guide, teacher materials, student materials, information regarding interpreting PALS-K results, and a PALS-K Technical Reference. Special Accommodations Large Print is available, as is, information on using the PALS-K assessment with students who have disabilities that require a non-standard approach to administering the assessment.

Administration

BEHAVIOR ONLY: What type of administrator is your tool designed for?

General education teacher
Special education teacher
Parent
Child
External observer
Other

If other, please specify:

What is the administration setting?

Direct observation
Rating scale
Checklist
Performance measure
Questionnaire
Direct: Computerized
One-to-one
Other

If other, please specify:

Does the tool require technology?

No

If yes, what technology is required to implement your tool? (Select all that apply)

Computer or tablet
Internet connection
Other technology (please specify)

If your program requires additional technology not listed above, please describe the required technology and the extent to which it is combined with teacher small-group instruction/intervention:

What is the administration context?

Individual
Small group   If small group, n=
Large group   If large group, n=
Computer-administered
Other

If other, please specify:

What is the administration time?

Time in minutes

30

per (student/group/other unit)

student

Additional scoring time:

Time in minutes

0

per (student/group/other unit)

student

ACADEMIC ONLY: What are the discontinue rules?

No discontinue rules provided
Basals
Ceilings
Other

If other, please specify:

Are norms available?

No

Are benchmarks available?

Yes

If yes, how many benchmarks per year?

2

If yes, for which months are benchmarks available?

Fall and Spring

BEHAVIOR ONLY: Can students be rated concurrently by one administrator?

If yes, how many students can be rated concurrently?

Training & Scoring

Training

Is training for the administrator required?

Yes

Describe the time required for administrator training, if applicable:

Less than 1 hr of training

Please describe the minimum qualifications an administrator must possess.

No minimum qualifications

Are training manuals and materials available?

Yes

Are training manuals/materials field-tested?

Yes

Are training manuals/materials included in cost of tools?

Yes

If No, please describe training costs:

Can users obtain ongoing professional and technical support?

Yes

If Yes, please describe how users can obtain support:

Options include email & phone technical & literacy support, data analytics, online self-paced literacy courses, monthly topical webinars, personalized PD, online PD modules & online PD

Scoring

How are scores calculated?

Manually (by hand)
Automatically (computer-scored)
Other

If other, please specify:

Do you provide basis for calculating performance level scores?

Yes

What is the basis for calculating performance level and percentile scores?

Age norms
Grade norms
Classwide norms
Schoolwide norms
Stanines
Normal curve equivalents

What types of performance level scores are available?

Raw score
Standard score
Percentile score
Grade equivalents
IRT-based score
Age equivalents
Stanines
Normal curve equivalents
Developmental benchmarks
Developmental cut points
Equated
Probability
Lexile score
Error analysis
Composite scores
Subscale/subtest scores
Other

If other, please specify:

item level responses

Does your tool include decision rules?

Yes

If yes, please describe.

Decision rules are built into the administration of tool to help guide the testing needed to establish reading level. The only decision rule is related to the number correct on the word list. All teachers begin with the grade-level word list. If 15 or more are correct, you continue to move up in level of word list until you find where the child no longer can achieve 15 or more correct. The highest list for which the child got 15 or more correct will guide the passages given and collectively the passage data and word list data determines a child’s reading level.

Can you provide evidence in support of multiple decision rules?

No

If yes, please describe.

There are not multiple decision rules. Evidence of the ability for users to adhere to the decision-rule in place, as well as the consistency in application of the rule can be seen in the technical manual, with evidence of test-retest reliability and interrater reliability (pp. 35-27 in the Technical manual

Please describe the scoring structure. Provide relevant details such as the scoring format, the number of items overall, the number of items per subscale, what the cluster/composite score comprises, and how raw scores are calculated.

Online Assessment Wizard (OAW) available for automatic scoring or student score sheet to guide manual scoring Number of correct responses in each task; sum of task scores = Summed Scores

Describe the tool’s approach to screening, samples (if applicable), and/or test format, including steps taken to ensure that it is appropriate for use with culturally and linguistically diverse populations and students with disabilities.

The nature and use of PALS 1-3 is described extensively in the attached PALS 1-3 Technical Reference. Briefly, a substantial research base has suggested key variables that help identify children most likely to experience subsequent difficulties with reading achievement. This research indicates that measures of phonological awareness, alphabet knowledge, letter-sound knowledge, and other elements of early literacy (e.g., phonetic spelling, word recognition) serve as robust predictors of children’s later literacy achievement. PALS 1-3 uses a three-tiered approach in which the first tier (or Entry Level) in the Screening Level and contains a routing appraisal that estimates a child’s general level of skill in reading and spelling. The Entry Level tier also indicates the first required passage to be read in Level A. Level A assesses the accuracy, fluency, rate, and comprehension of a child’s oral reading in context. Students can be identified as needing intensive literacy intervention if they do not meet the Entry Level benchmark. Level B assesses emergent and beginning reading essentials in alphabetic knowledge and concept of word, and is taken only by students who do not have a measurable reading at the Preprimer or higher level. If Level B benchmarks are not met, children are routed to Level C for a more in-depth evaluation of phonemic awareness skills including blending and segmenting speech sounds. Students demonstrate their skills in each domain to their classroom teacher, who administers PALS in the classroom (after reading the PALS Administration and Scoring Guide and watching online training modules). The performance-based tasks do not have a time limit; they are administered one-on-one, except for the Spelling task, which can be administered in small groups or in the class as a whole. Each task contains a criterion score or benchmark for a minimal level of competency. The benchmarks change from fall to spring. Cultural diversity was addressed in the development of PALS tasks and items, during which process external reviewers were asked to examine all PALS materials and items with an eye toward potential bias. Our reliability analyses, including differential item functioning (DIFF) have been disaggregated by gender, race/ethnicity, SES and even geographical regions for years, as described in the attached Technical Reference and in this application. All or part of PALS 1-3 has been used as the statewide literacy screener for the Commonwealth of Virginia since 1997-1998.

Classification Accuracy & Cross-Validation Summary

Grade	Grade 1	Grade 2	Grade 3
Classification Accuracy Fall
Classification Accuracy Winter
Classification Accuracy Spring

MAP

Classification Accuracy

Select time of year

Fall

Winter

Spring

Describe the criterion (outcome) measure(s) including the degree to which it/they is/are independent from the screening measure.

The MAP is a widely used computer-adaptive test of reading, language arts, and math. Analyses for this application used the reading MAP scores as a criterion/outcome measure and this measure has appeared as a screening tool in reading (like PALS) on the tools chart from the Center on Response to Intervention and also appears on the new chart within the National Center for Intensive Intervention. The reading test in grades 1-3 is designed to align to Common Core state standards and many state learning standards and provides an overall scaled score (RIT score), a percentile rank, and a lexile range. We use the scaled score in our analyses. The MAP test focuses broadly on reading, measuring individual foundational skills. It is designed to measure a child’s general ability in reading text, print concepts, vocabulary, phonemic awareness, listening, and writing. Subtest scores are not available. We focused on the relation of the percentiles of children’s performance based on PALS sum scores and MAP scaled scores. Each analysis contains the Spring PALS scores and the following semester’s MAP reading score (e.g. 1st grade Spring PALS to 2nd grade Fall MAP).

Do the classification accuracy analyses examine concurrent and/or predictive classification?

Concurrent
Predictive

Describe when screening and criterion measures were administered and provide a justification for why the method(s) you chose (concurrent and/or predictive) is/are appropriate for your tool.

Describe how the classification analyses were performed and cut-points determined. Describe how the cut points align with students at-risk. Please indicate which groups were contrasted in your analyses (e.g., low risk students versus high risk students, low risk students versus moderate risk students).

We used the established cutoffs based on the 20th percentile for outcome measures. The PALS cutoff is the benchmark score: Grade 1 = 35, Grade 2 = 54, Grade 3 = 65. Classifications were based on simple 2 x 2 analyses of risk status on both measures. All classification analyses were performed in R version 3.5.1 “Feather Spray” using the ‘pROC’, ‘caret’, and ‘EvaluationMeasures’ packages.

Were the children in the study/studies involved in an intervention in addition to typical classroom instruction between the screening measure and outcome assessment?

Yes

If yes, please describe the intervention, what children received the intervention, and how they were chosen.

Children scoring below the PALS benchmarks are designated to receive an additional 2.5 hours of intensive literacy intervention per week until the next screening window, as per an early reading policy initiative originating in the Commonwealth of Virginia. Intervention is to be based on strengths and needs identified through the PALS screening. At subsequent screening windows, children’s performance relative to PALS benchmarks determines whether they continue to receive this additional dose of literacy instruction.

Cross-Validation

Has a cross-validation study been conducted?

Yes

If yes,

Select time of year.

Fall

Winter

Spring

Describe the criterion (outcome) measure(s) including the degree to which it/they is/are independent from the screening measure.

The MAP is a widely used computer-adaptive test of reading, language arts, and math. Analyses for this application used the reading MAP scores as a criterion/outcome measure and this measure has appeared as a screening tool in reading (like PALS) on the tools chart from the Center on Response to Intervention and also appears on the new chart within the National Center for Intensive Intervention. The reading test in grades 1-3 is designed to align to Common Core state standards and many state learning standards and provides an overall scaled score (RIT score), a percentile rank, and a lexile range. We use the scaled score in our analyses. The MAP test focuses broadly on reading, measuring individual foundational skills. It is designed to measure a child’s general ability in reading text, print concepts, vocabulary, phonemic awareness, listening, and writing. Subtest scores are not available. We focused on the relation of the percentiles of children’s performance based on PALS sum scores and MAP scaled scores. Each analysis contains the Spring PALS scores and the following semester’s MAP reading score (e.g. 1st grade Spring PALS to 2nd grade Fall MAP).

Do the cross-validation analyses examine concurrent and/or predictive classification?

Concurrent
Predictive

Describe when screening and criterion measures were administered and provide a justification for why the method(s) you chose (concurrent and/or predictive) is/are appropriate for your tool.

Describe how the cross-validation analyses were performed and cut-points determined. Describe how the cut points align with students at-risk. Please indicate which groups were contrasted in your analyses (e.g., low risk students versus high risk students, low risk students versus moderate risk students).

We used the established cutoffs based on the 20th percentile for outcome measures. The PALS cutoff is the benchmark score: Grade 1 = 35, Grade 2 = 54, Grade 3 = 65. Classifications were based on simple 2 x 2 analyses of risk status on both measures. All classification analyses were performed in R version 3.5.1 “Feather Spray” using the ‘pROC’, ‘caret’, and ‘EvaluationMeasures’ packages.

Were the children in the study/studies involved in an intervention in addition to typical classroom instruction between the screening measure and outcome assessment?

Yes

If yes, please describe the intervention, what children received the intervention, and how they were chosen.

Children scoring below the PALS benchmarks are designated to receive an additional 2.5 hours of intensive literacy intervention per week until the next screening window, as per an early reading policy initiative originating in the Commonwealth of Virginia. Intervention is to be based on strengths and needs identified through the PALS screening. At subsequent screening windows, children’s performance relative to PALS benchmarks determines whether they continue to receive this additional dose of literacy instruction.

---

STAR Reading

Classification Accuracy

Select time of year

Fall

Winter

Spring

Describe the criterion (outcome) measure(s) including the degree to which it/they is/are independent from the screening measure.

The STAR Renaissance reading test is a computer adaptive measure designed to provide an indicator of how well children are mastering reading skills as described within standards of learning and provides a scaled score that is seen as highly predictive of proficiency levels on high-stakes 3rd grade tests. The STAR Reading test is applicable for independent readers.

Do the classification accuracy analyses examine concurrent and/or predictive classification?

Concurrent
Predictive

Describe when screening and criterion measures were administered and provide a justification for why the method(s) you chose (concurrent and/or predictive) is/are appropriate for your tool.

Describe how the classification analyses were performed and cut-points determined. Describe how the cut points align with students at-risk. Please indicate which groups were contrasted in your analyses (e.g., low risk students versus high risk students, low risk students versus moderate risk students).

We used the established cutoffs based on the 20th percentile for outcome measures. The PALS cutoff is the benchmark score: Grade 1 = 35, Grade 2 = 54, Grade 3 = 65. Classifications were based on simple 2 x 2 analyses of risk status on both measures. All classification analyses were performed in R version 3.5.1 “Feather Spray” using the ‘pROC’, ‘caret’, and ‘EvaluationMeasures’ packages.

Were the children in the study/studies involved in an intervention in addition to typical classroom instruction between the screening measure and outcome assessment?

Yes

If yes, please describe the intervention, what children received the intervention, and how they were chosen.

Children scoring below the PALS benchmarks are designated to receive an additional 2.5 hours of intensive literacy intervention per week until the next screening window, as per an early reading policy initiative originating in the Commonwealth of Virginia. Intervention is to be based on strengths and needs identified through the PALS screening. At subsequent screening windows, children’s performance relative to PALS benchmarks determines whether they continue to receive this additional dose of literacy instruction.

Cross-Validation

Has a cross-validation study been conducted?

Yes

If yes,

Select time of year.

Fall

Winter

Spring

Describe the criterion (outcome) measure(s) including the degree to which it/they is/are independent from the screening measure.

The STAR Renaissance reading test is a computer adaptive measure designed to provide an indicator of how well children are mastering reading skills as described within standards of learning and provides a scaled score that is seen as highly predictive of proficiency levels on high-stakes 3rd grade tests. The STAR Reading test is applicable for independent readers.

Do the cross-validation analyses examine concurrent and/or predictive classification?

Concurrent
Predictive

Describe when screening and criterion measures were administered and provide a justification for why the method(s) you chose (concurrent and/or predictive) is/are appropriate for your tool.

Describe how the cross-validation analyses were performed and cut-points determined. Describe how the cut points align with students at-risk. Please indicate which groups were contrasted in your analyses (e.g., low risk students versus high risk students, low risk students versus moderate risk students).

We used the established cutoffs based on the 20th percentile for outcome measures. The PALS cutoff is the benchmark score: Grade 1 = 35, Grade 2 = 54, Grade 3 = 65. Classifications were based on simple 2 x 2 analyses of risk status on both measures. All classification analyses were performed in R version 3.5.1 “Feather Spray” using the ‘pROC’, ‘caret’, and ‘EvaluationMeasures’ packages.

Were the children in the study/studies involved in an intervention in addition to typical classroom instruction between the screening measure and outcome assessment?

Yes

If yes, please describe the intervention, what children received the intervention, and how they were chosen.

Children scoring below the PALS benchmarks are designated to receive an additional 2.5 hours of intensive literacy intervention per week until the next screening window, as per an early reading policy initiative originating in the Commonwealth of Virginia. Intervention is to be based on strengths and needs identified through the PALS screening. At subsequent screening windows, children’s performance relative to PALS benchmarks determines whether they continue to receive this additional dose of literacy instruction.

---

Reading Standards of Learning (SOL)

Classification Accuracy

Select time of year

Fall

Winter

Spring

Describe the criterion (outcome) measure(s) including the degree to which it/they is/are independent from the screening measure.

The Reading Standards of Learning (SOL) test is a computer adaptive test that assesses students’ reading comprehension, word analysis strategies, and vocabulary knowledge. The scores are scaled to a range of 0 to 600 where below 400 indicates a failing score.

Do the classification accuracy analyses examine concurrent and/or predictive classification?

Concurrent
Predictive

Describe when screening and criterion measures were administered and provide a justification for why the method(s) you chose (concurrent and/or predictive) is/are appropriate for your tool.

Describe how the classification analyses were performed and cut-points determined. Describe how the cut points align with students at-risk. Please indicate which groups were contrasted in your analyses (e.g., low risk students versus high risk students, low risk students versus moderate risk students).

We used the established cutoffs based on the 20th percentile for outcome measures. The PALS cutoff is the benchmark score: Grade 1 = 35, Grade 2 = 54, Grade 3 = 65. Classifications were based on simple 2 x 2 analyses of risk status on both measures. All classification analyses were performed in R version 3.5.1 “Feather Spray” using the ‘pROC’, ‘caret’, and ‘EvaluationMeasures’ packages.

Were the children in the study/studies involved in an intervention in addition to typical classroom instruction between the screening measure and outcome assessment?

Yes

If yes, please describe the intervention, what children received the intervention, and how they were chosen.

Children scoring below the PALS benchmarks are designated to receive an additional 2.5 hours of intensive literacy intervention per week until the next screening window, as per an early reading policy initiative originating in the Commonwealth of Virginia. Intervention is to be based on strengths and needs identified through the PALS screening. At subsequent screening windows, children’s performance relative to PALS benchmarks determines whether they continue to receive this additional dose of literacy instruction.

Cross-Validation

Has a cross-validation study been conducted?

Yes

If yes,

Select time of year.

Fall

Winter

Spring

Describe the criterion (outcome) measure(s) including the degree to which it/they is/are independent from the screening measure.

The Reading Standards of Learning (SOL) test is a computer adaptive test that assesses students’ reading comprehension, word analysis strategies, and vocabulary knowledge. The scores are scaled to a range of 0 to 600 where below 400 indicates a failing score.

Do the cross-validation analyses examine concurrent and/or predictive classification?

Concurrent
Predictive

Describe when screening and criterion measures were administered and provide a justification for why the method(s) you chose (concurrent and/or predictive) is/are appropriate for your tool.

Describe how the cross-validation analyses were performed and cut-points determined. Describe how the cut points align with students at-risk. Please indicate which groups were contrasted in your analyses (e.g., low risk students versus high risk students, low risk students versus moderate risk students).

We used the established cutoffs based on the 20th percentile for outcome measures. The PALS cutoff is the benchmark score: Grade 1 = 35, Grade 2 = 54, Grade 3 = 65. Classifications were based on simple 2 x 2 analyses of risk status on both measures. All classification analyses were performed in R version 3.5.1 “Feather Spray” using the ‘pROC’, ‘caret’, and ‘EvaluationMeasures’ packages.

Were the children in the study/studies involved in an intervention in addition to typical classroom instruction between the screening measure and outcome assessment?

Yes

If yes, please describe the intervention, what children received the intervention, and how they were chosen.

Children scoring below the PALS benchmarks are designated to receive an additional 2.5 hours of intensive literacy intervention per week until the next screening window, as per an early reading policy initiative originating in the Commonwealth of Virginia. Intervention is to be based on strengths and needs identified through the PALS screening. At subsequent screening windows, children’s performance relative to PALS benchmarks determines whether they continue to receive this additional dose of literacy instruction.

---

Classification Accuracy - Spring

Evidence	Grade 1	Grade 2	Grade 3
Criterion measure	MAP	MAP	MAP
Cut Points - Percentile rank on criterion measure	20	20	20
Cut Points - Performance score on criterion measure	161	175	186
Cut Points - Corresponding performance score (numeric) on screener measure	35	56	65
Classification Data - True Positive (a)
Classification Data - False Positive (b)
Classification Data - False Negative (c)
Classification Data - True Negative (d)
Area Under the Curve (AUC)	0.76	0.80	0.80
AUC Estimate’s 95% Confidence Interval: Lower Bound	0.74	0.77	0.77
AUC Estimate’s 95% Confidence Interval: Upper Bound	0.78	0.82	0.82

Statistics	Grade 1	Grade 2	Grade 3
Base Rate
Overall Classification Rate
Sensitivity
Specificity
False Positive Rate
False Negative Rate
Positive Predictive Power
Negative Predictive Power

Sample	Grade 1	Grade 2	Grade 3
Date	Fall	Fall	Fall
Sample Size
Geographic Representation	Mountain (CO) South Atlantic (VA)	Mountain (CO) South Atlantic (VA)	Mountain (CO) South Atlantic (VA)
Male
Female
Other
Gender Unknown
White, Non-Hispanic
Black, Non-Hispanic
Hispanic
Asian/Pacific Islander
American Indian/Alaska Native
Other
Race / Ethnicity Unknown
Low SES
IEP or diagnosed disability
English Language Learner

Cross-Validation - Spring

Evidence	Grade 1	Grade 2	Grade 3
Criterion measure	MAP	MAP	MAP
Cut Points - Percentile rank on criterion measure	20	20	20
Cut Points - Performance score on criterion measure	161	175	186
Cut Points - Corresponding performance score (numeric) on screener measure	35	56	65
Classification Data - True Positive (a)
Classification Data - False Positive (b)
Classification Data - False Negative (c)
Classification Data - True Negative (d)
Area Under the Curve (AUC)	0.72	0.78	0.80
AUC Estimate’s 95% Confidence Interval: Lower Bound	0.70	0.76	0.77
AUC Estimate’s 95% Confidence Interval: Upper Bound	0.75	0.80	0.82

Statistics	Grade 1	Grade 2	Grade 3
Base Rate
Overall Classification Rate
Sensitivity
Specificity
False Positive Rate
False Negative Rate
Positive Predictive Power
Negative Predictive Power

Sample	Grade 1	Grade 2	Grade 3
Date	Fall	Fall	Fall
Sample Size
Geographic Representation	Mountain (CO) South Atlantic (VA)	Mountain (CO) South Atlantic (VA)	Mountain (CO) South Atlantic (VA)
Male
Female
Other
Gender Unknown
White, Non-Hispanic
Black, Non-Hispanic
Hispanic
Asian/Pacific Islander
American Indian/Alaska Native
Other
Race / Ethnicity Unknown
Low SES
IEP or diagnosed disability
English Language Learner

Reliability

Grade	Grade 1	Grade 2	Grade 3
Rating	d	d	d

*Offer a justification for each type of reliability reported, given the type and purpose of the tool.

The first reliability estimate is coefficient alpha. It is appropriate because of the use of task scores when making instructional decisions. The confidence interval for coefficient alpha is based on Felt’s derivation of the sampling distribution of coefficient alpha. The second type of reliability estimate is Pearson correlation coefficients between raters’ subtask scores. It is appropriate to determine that two scorers can reach nearly the same score while observing the same child. Test-retest reliability is the correlation between the same child’s scores on the test when administered at different points in time. Test-retest reliability is crucial in determining stability over time.

*Describe the sample(s), including size and characteristics, for each reliability analysis conducted.

Coefficient alphas are calculated from a sample of NNN students in the Commonwealth of Virginia. 60% of the students were male and most were non-white (64%). About 23% of the sample was ESL. A large percentage of the sample was in first grade (81%), but some second (15%) and third grade students were also involved (4%). Sample sizes vary by grade level as indicated in the table below. Inter-rater reliability scores for word recognition were calculated using a sample from Fall 2000 consisting of 478 students. Inter-rater reliability scores for spelling were calculated using a sample from Fall 2001 consisting of 2,137 students. Test-retest reliabilities were calculated from a sample from Fall 2002 consisting of 204 students.

*Describe the analysis procedures for each reported type of reliability.

Test-retest reliability was calculated as the Pearson coefficient between the two administrations (Table 22 in the PALS 1-3 Technical Manual). Inter-rater reliability was calculated as the correlation coefficient between the two raters’ scores (Table 23 in the PALS 1-3 Technical Manual). For the coefficient alphas, the analysis procedures involved subsetting the data for each subgroup and computing coefficient alpha and its confidence interval.

*In the table(s) below, report the results of the reliability analyses described above (e.g., internal consistency or inter-rater reliability coefficients).

Type of	Subgroup	Informant	Age / Grade	Test or Criterion	n	Median Coefficient	95% Confidence Interval Lower Bound	95% Confidence Interval Upper Bound

Results from other forms of reliability analysis not compatible with above table format:

Manual cites other published reliability studies:

No

Provide citations for additional published studies.

Do you have reliability data that are disaggregated by gender, race/ethnicity, or other subgroups (e.g., English language learners, students with disabilities)?

Yes

If yes, fill in data for each subgroup with disaggregated reliability data.

Type of	Subgroup	Informant	Age / Grade	Test or Criterion	n	Median Coefficient	95% Confidence Interval Lower Bound	95% Confidence Interval Upper Bound

Results from other forms of reliability analysis not compatible with above table format:

Additional disaggregated data is available from the Center upon request.

Manual cites other published reliability studies:

No

Provide citations for additional published studies.

Validity

Grade	Grade 1	Grade 2	Grade 3
Rating

*Describe each criterion measure used and explain why each measure is appropriate, given the type and purpose of the tool.

• Stanford Achievement Test and the California Achievement Test are standardized achievement tests used by school districts in the United States and in American schools abroad for assessing children from kindergarten through high school. • The Qualitative Reading Inventory (QRI) and the Developmental Reading Assessment (DRA) are both classroom-based informal reading inventories commonly used across the U.S. and in American schools around the world. • RIT description • iStation description

*Describe the sample(s), including size and characteristics, for each validity analysis conducted.

• The Stanford, the QRI, and the CAT were sub-samples from the larger state-wide sample in Grades 1-3 in Virginia. • The analyses relating Grade 2 PALS to Grade 2 MAP are based on a sample of 2,043 2nd Grade students from Colorado. 51.3% of students are male, 49.4% are nonwhite, and 22% are English Language Learners. • The analyses relating Grade 3 PALS to Grade 3 iStation are based on a sample of 23 3rd graders from Frederick County Virginia from 2016 and 2017. 70% of students are nonwhite, 52% are female, and 26% are English Language Learners. • The analyses relating Grade 3 PALS to Grade 3 MAP are based on a sample of 2,278 students from Colorado. 53% of students are nonwhite, 46% are female, and 23% are English Language Learners.

*Describe the analysis procedures for each reported type of validity.

Validity analyses were conducted by taking the Pearson’s correlation between PALS score and either concurrent outcome or score on a later test (predictive) or a test conducted in the same semester (concurrent). Grade 3 and Grade 2 analyses were performed in R.

*In the table below, report the results of the validity analyses described above (e.g., concurrent or predictive validity, evidence based on response processes, evidence based on internal structure, evidence based on relations to other variables, and/or evidence based on consequences of testing), and the criterion measures.

Type of	Subgroup	Informant	Age / Grade	Test or Criterion	n	Median Coefficient	95% Confidence Interval Lower Bound	95% Confidence Interval Upper Bound

Results from other forms of validity analysis not compatible with above table format:

Construct: As one piece of evidence in support of the construct validity of PALS 1-3, we conduct Principal Component Analyses on the tasks that make up the PALS 1-3 Entry Level Summed Score each year, based on statewide data. These analyses consistently yield a single factor (eigenvalue greater than 1.0) that accounts for more than three quarters of the variance in these scores. Content: The content validity is supported by the process of item development (described in the Technical Reference, Section III, pp. 11 – 19). The guiding principles underlying this process were that (a) tasks and items were a representative sample of tasks from other early literacy screening instruments, (b) items had a history of use in phonological awareness and early literacy research, and (c) items were aligned with Virginia’s Standards of Learning for English (Reading). Each of these principles was further supported by the process of review by an Advisory Panel of literacy professionals from across Virginia, and by an External Review Panel, which consisted of nationally recognized experts in the field of reading, communication sciences, or psychology.

Manual cites other published reliability studies:

Yes

Provide citations for additional published studies.

Huang, F. L. (2014). Using a bifactor model to assess the factor structure of the Phonological Awareness Literacy Screening for grades 1 through 3. Journal of Psychoeducational Assessment, 32, 638-650.

Describe the degree to which the provided data support the validity of the tool.

These data suggest that PALS has a strong relationship to the Stanford-9 and QRI-II in 1st grade, the Stanford-9 in 2nd grade, and MAP and iStation reading scores in 3rd grade

Do you have validity data that are disaggregated by gender, race/ethnicity, or other subgroups (e.g., English language learners, students with disabilities)?

No

If yes, fill in data for each subgroup with disaggregated validity data.

Type of	Subgroup	Informant	Age / Grade	Test or Criterion	n	Median Coefficient	95% Confidence Interval Lower Bound	95% Confidence Interval Upper Bound

Results from other forms of validity analysis not compatible with above table format:

Manual cites other published reliability studies:

No

Provide citations for additional published studies.

Bias Analysis

Grade	Grade 1	Grade 2	Grade 3
Rating	Yes	Yes	Yes

Have you conducted additional analyses related to the extent to which your tool is or is not biased against subgroups (e.g., race/ethnicity, gender, socioeconomic status, students with disabilities, English language learners)? Examples might include Differential Item Functioning (DIF) or invariance testing in multiple-group confirmatory factor models.

Yes

If yes,

a. Describe the method used to determine the presence or absence of bias:

We used the Mantel-Haenszel procedure to test for differential item functioning (DIF) for the Entry Level Screening Tasks. We combined this hypothesis testing procedures with the common-odds ratio effect size to determine the practical significance of DIF. We followed the ETS classification criteria of A, B, C to indicate no, moderate, and large amount of DIF.

b. Describe the subgroups for which bias analyses were conducted:

Male/Female, White/Non-White, ESL/Non-ESL

c. Describe the results of the bias analyses conducted, including data and interpretative statements. Include magnitude of effect (if available) if bias has been identified.

The majority of the items for each sub-task of the summed score received an ETS classification of A, indicating no bias in spelling or word recognition for most groups. Three percent of the items in spelling showed moderate bias for White/Non-White. Fifteen percent of the items of the Word Recognition Primer list showed moderate bias for ESL/Non-ESL and 20% of the items on the Word Recognition First Grade list showed moderate bias for Male/Female. Regardless of group, however, these words, which represent frequently occurring words at these grade levels, must be recognized automatically in order to be a fluent reader. (see doc attached under validity section for further info).