Phonological Awareness Literacy Screening (PALS)
PALS for Kindergarten

Descriptive Information

Please provide a description of your tool:

The Phonological Awareness Literacy Screening (PALS) is a criterion-referenced screening, diagnostic, and progress monitoring tool. PALS consists of two instruments (PALS-K and PALS 1-3) that measure young children’s knowledge of important literacy fundamentals: phonological awareness, alphabet awareness, letter sound knowledge, spelling, concept of word, word recognition in isolation, and oral passage reading. The major purpose of PALS is to identify students who may be at risk for reading difficulties and who need intensive literacy intervention beyond what is provided to typically developing readers. The second purpose of PALS is to provide teachers with explicit diagnostic information about what students know and need to know about the fundamental components of literacy that may be used to target instruction to meet students’ needs. The third purpose of PALS is to monitor students’ progress and determine the effectiveness of instruction or intervention PALS-K PALS-K measures kindergarten students’ knowledge of phonological awareness and early literacy skills. The phonological awareness component of PALS–K instrument is a measure of a young child’s ability to identify rhyme units and isolate beginning sounds. The literacy skills component of PALS-K measures a young child’s knowledge of alphabet recognition, letter sounds, phoneme-grapheme correspondences, concept of word, and word recognition.

The tool is intended for use with the following grade(s).

Preschool / Pre - kindergarten
Kindergarten
First grade
Second grade
Third grade
Fourth grade
Fifth grade
Sixth grade
Seventh grade
Eighth grade
Ninth grade
Tenth grade
Eleventh grade
Twelfth grade

The tool is intended for use with the following age(s).

0-4 years old
5 years old
6 years old
7 years old
8 years old
9 years old
10 years old
11 years old
12 years old
13 years old
14 years old
15 years old
16 years old
17 years old
18 years old

The tool is intended for use with the following student populations.

Students in general education
Students with disabilities
English language learners

ACADEMIC ONLY: What skills does the tool screen?

Reading

Phonological processing:

RAN
Memory
Awareness
Letter sound correspondence
Phonics
Structural analysis

Word ID

Accuracy
Speed

Nonword

Accuracy
Speed

Spelling

Accuracy
Speed

Passage

Accuracy
Speed

Reading comprehension:

Multiple choice questions
Cloze
Constructed Response
Retell
Maze
Sentence verification
Other (please describe):

Listening comprehension:

Multiple choice questions
Cloze
Constructed Response
Retell
Maze
Sentence verification
Vocabulary
Expressive
Receptive

Mathematics

Global Indicator of Math Competence

Accuracy
Speed
Multiple Choice
Constructed Response

Early Numeracy

Accuracy
Speed
Multiple Choice
Constructed Response

Mathematics Concepts

Accuracy
Speed
Multiple Choice
Constructed Response

Mathematics Computation

Accuracy
Speed
Multiple Choice
Constructed Response

Mathematic Application

Accuracy
Speed
Multiple Choice
Constructed Response

Fractions/Decimals

Accuracy
Speed
Multiple Choice
Constructed Response

Algebra

Accuracy
Speed
Multiple Choice
Constructed Response

Geometry

Accuracy
Speed
Multiple Choice
Constructed Response

Other (please describe):

Please describe specific domain, skills or subtests:

BEHAVIOR ONLY: Which category of behaviors does your tool target?

Internalizing
Externalizing
Internalizing and Externalizing

BEHAVIOR ONLY: Please identify which broad domain(s)/construct(s) are measured by your tool and define each sub-domain or sub-construct.

Acquisition and Cost Information

Where to obtain:

Email Address

pals@virginia.edu

Address

UVA/PALS PO Box 800785, Charlottesville, VA 22908-8785

Phone Number

888-UVA-PALS

Website

pals.virginia.edu

Initial cost for implementing program:

Cost

$5.50

Unit of cost

student

Replacement cost per unit for subsequent use:

Cost

Unit of cost

Duration of license

Additional cost information:

Describe basic pricing plan and structure of the tool. Provide information on what is included in the published tool, as well as what is not included but required for implementation.

ADDITIONAL COST INFO: *It is free for all teachers/students in public schools in VA K-3 and all children in PreK with state funding. Pricing below reflects IO Education/Illuminate pricing for all schools outside of VA or private schools in VA 5.50/child for an annual basic license; minimal of 20 licenses or 7.25/student license for “Premier” which includes access to PD resources such as webinars etc.; minimal 20 licenses. Shipping costs for one-time student materials/binders Materials include an Introduction and Overview, an Administration and Scoring Guide, teacher materials, student materials, information regarding interpreting PALS-K results, and a PALS-K Technical Reference.

Provide information about special accommodations for students with disabilities.

Large Print is available, as is, information on using the PALS-K assessment with students who have disabilities that require a non-standard approach to administering the assessment.

Administration

BEHAVIOR ONLY: What type of administrator is your tool designed for?

General education teacher
Special education teacher
Parent
Child
External observer
Other

If other, please specify:

What is the administration setting?

Direct observation
Rating scale
Checklist
Performance measure
Questionnaire
Direct: Computerized
One-to-one
Other

If other, please specify:

Does the tool require technology?

No

If yes, what technology is required to implement your tool? (Select all that apply)

Computer or tablet
Internet connection
Other technology (please specify)

If your program requires additional technology not listed above, please describe the required technology and the extent to which it is combined with teacher small-group instruction/intervention:

What is the administration context?

Individual
Small group   If small group, n=
Large group   If large group, n=
Computer-administered
Other

If other, please specify:

What is the administration time?

Time in minutes

20

per (student/group/other unit)

student

Additional scoring time:

Time in minutes

0

per (student/group/other unit)

student

ACADEMIC ONLY: What are the discontinue rules?

No discontinue rules provided
Basals
Ceilings
Other

If other, please specify:

Are norms available?

No

Are benchmarks available?

Yes

If yes, how many benchmarks per year?

2

If yes, for which months are benchmarks available?

Fall & Spring

BEHAVIOR ONLY: Can students be rated concurrently by one administrator?

If yes, how many students can be rated concurrently?

Training & Scoring

Training

Is training for the administrator required?

Yes

Describe the time required for administrator training, if applicable:

Less than 1 hr of training

Please describe the minimum qualifications an administrator must possess.

No minimum qualifications

Are training manuals and materials available?

Yes

Are training manuals/materials field-tested?

Yes

Are training manuals/materials included in cost of tools?

Yes

If No, please describe training costs:

Can users obtain ongoing professional and technical support?

Yes

If Yes, please describe how users can obtain support:

Options include email & phone technical & literacy support, data analytics, online self-paced literacy courses, monthly topical webinars, personalized PD, online PD modules & online PD

Scoring

How are scores calculated?

Manually (by hand)
Automatically (computer-scored)
Other

If other, please specify:

Do you provide basis for calculating performance level scores?

Yes

What is the basis for calculating performance level and percentile scores?

Age norms
Grade norms
Classwide norms
Schoolwide norms
Stanines
Normal curve equivalents

What types of performance level scores are available?

Raw score
Standard score
Percentile score
Grade equivalents
IRT-based score
Age equivalents
Stanines
Normal curve equivalents
Developmental benchmarks
Developmental cut points
Equated
Probability
Lexile score
Error analysis
Composite scores
Subscale/subtest scores
Other

If other, please specify:

item level responses

Does your tool include decision rules?

Yes

If yes, please describe.

If a child is below the threshold on the Rhyme Awareness or Beginning Sounds subtasks the teacher administers these individually (standard administration is a group of 5 or less) and then takes that score for the final score

Can you provide evidence in support of multiple decision rules?

No

If yes, please describe.

This decision rule has been in place since the creation of the test and the technical manual provides evidence of strong test-retest reliability (p. 26, Table 11 of the technical manual) as well as strong inter-rater reliability (Table 14 of the technical manual).

Please describe the scoring structure. Provide relevant details such as the scoring format, the number of items overall, the number of items per subscale, what the cluster/composite score comprises, and how raw scores are calculated.

There are criterion reference scores, not standard scores. Age and grade-based performance as well as expert review panels set the criterion reference scores. The cluster/composite score is called the sum score. In Kindergarten this is the raw total of the subtests (Rhyme Awareness, Beginning Sounds, Alphabet Recognition, Spelling, Letter Sounds, Concept of Word: Word List)

Describe the tool’s approach to screening, samples (if applicable), and/or test format, including steps taken to ensure that it is appropriate for use with culturally and linguistically diverse populations and students with disabilities.

The PALS K approach to screening is grounded in literacy developmental theory. Briefly, a substantial research base has suggested key foundational literacy skills that are most important to later reading success and that also help identify children most likely to experience subsequent difficulties with reading achievement. Robust predictors are related to print and alphabet, phonological awareness, as well as tasks that require their integration, such as invented spelling and concept of word in text. The PALS- K examines each of these subdomains in depth through individual tasks, as well as examines overall performance through a summed score. Students demonstrate their skills in each domain to their classroom teacher, who administers PALS in the classroom (after reading the PALS Administration and Scoring Guide and watching online training modules). The performance-based tasks do not have a time limit; they are administered in a combination of 1:1 and small group. For the small group tasks (rhyme and beginning sounds) teachers provide children an opportunity to take that test 1:1 should they fall below expectations in a small group setting. Each task contains a criterion score or benchmark for a minimal level of competency. The benchmarks change from fall to spring. Cultural diversity was addressed in the development of PALS tasks and items, during which process external reviewers were asked to examine all PALS materials and items with an eye toward potential bias. Our reliability analyses, including differential item functioning (DIFF) have been disaggregated by gender, race/ethnicity, SES and even geographical regions for years, as described in the attached Technical Reference and in this application. All of PALS K has been used as the statewide literacy screener for the Commonwealth of Virginia since 1997-1998.

Classification Accuracy & Cross-Validation Summary

Grade	Kindergarten
Classification Accuracy Fall
Classification Accuracy Winter
Classification Accuracy Spring

MAP

Classification Accuracy

Select time of year

Fall

Winter

Spring

Describe the criterion (outcome) measure(s) including the degree to which it/they is/are independent from the screening measure.

MAP DESCRIPTION: The MAP is a widely used computer-adaptive test of reading, language arts, and math. Analyses for this application used the reading MAP scores as a criterion/outcome measure and this measure has appeared as a screening tool in reading (like PALS) on the tools chart from the Center on Response to Intervention and also appears on the new chart within the National Center for Intensive Intervention. The reading test in K is designed to align to Common Core state standards and many state learning standards and provides an overall scaled score (RIT score), a percentile rank, and a lexile range. We use the scaled score in our analyses. The MAP test focuses broadly on reading, measuring individual foundational skills. It is designed to measure a child’s general ability in reading text, print concepts, vocabulary, phonemic awareness, listening, and writing. Subtest scores are not available. We focused on the relation of the percentiles of children’s performance based on PALS sum scores and MAP scaled scores.

Do the classification accuracy analyses examine concurrent and/or predictive classification?

Concurrent
Predictive

Describe when screening and criterion measures were administered and provide a justification for why the method(s) you chose (concurrent and/or predictive) is/are appropriate for your tool.

Describe how the classification analyses were performed and cut-points determined. Describe how the cut points align with students at-risk. Please indicate which groups were contrasted in your analyses (e.g., low risk students versus high risk students, low risk students versus moderate risk students).

We used the established cutoffs based on the 20th percentile for outcome measures as well as PALS sum scores. Classifications were based on simple 2 x 2 analyses of risk status on both measures. All classification analyses were performed in R version 3.5.1 “Feather Spray” using the ‘pROC’, ‘caret’, and ‘EvaluationMeasures’ packages.

Were the children in the study/studies involved in an intervention in addition to typical classroom instruction between the screening measure and outcome assessment?

Yes

If yes, please describe the intervention, what children received the intervention, and how they were chosen.

Children scoring below the PALS benchmarks are designated to receive an additional 2.5 hours of intensive literacy intervention per week until the next screening window, as per an early reading policy initiative originating in the Commonwealth of Virginia. Intervention is to be based on strengths and needs identified through the PALS screening. At subsequent screening windows, children’s performance relative to PALS benchmarks determines whether they continue to receive this additional dose of literacy instruction.

Cross-Validation

Has a cross-validation study been conducted?

Yes

If yes,

Select time of year.

Fall

Winter

Spring

Describe the criterion (outcome) measure(s) including the degree to which it/they is/are independent from the screening measure.

MAP DESCRIPTION: The MAP is a widely used computer-adaptive test of reading, language arts, and math. Analyses for this application used the reading MAP scores as a criterion/outcome measure and this measure has appeared as a screening tool in reading (like PALS) on the tools chart from the Center on Response to Intervention and also appears on the new chart within the National Center for Intensive Intervention. The reading test in K is designed to align to Common Core state standards and many state learning standards and provides an overall scaled score (RIT score), a percentile rank, and a lexile range. We use the scaled score in our analyses. The MAP test focuses broadly on reading, measuring individual foundational skills. It is designed to measure a child’s general ability in reading text, print concepts, vocabulary, phonemic awareness, listening, and writing. Subtest scores are not available. We focused on the relation of the percentiles of children’s performance based on PALS sum scores and MAP scaled scores.

Do the cross-validation analyses examine concurrent and/or predictive classification?

Concurrent
Predictive

Describe when screening and criterion measures were administered and provide a justification for why the method(s) you chose (concurrent and/or predictive) is/are appropriate for your tool.

Describe how the cross-validation analyses were performed and cut-points determined. Describe how the cut points align with students at-risk. Please indicate which groups were contrasted in your analyses (e.g., low risk students versus high risk students, low risk students versus moderate risk students).

We used the established cutoffs based on the 20th percentile for outcome measures as well as PALS sum scores. Classifications were based on simple 2 x 2 analyses of risk status on both measures. All classification analyses were performed in R version 3.5.1 “Feather Spray” using the ‘pROC’, ‘caret’, and ‘EvaluationMeasures’ packages.

Were the children in the study/studies involved in an intervention in addition to typical classroom instruction between the screening measure and outcome assessment?

Yes

If yes, please describe the intervention, what children received the intervention, and how they were chosen.

Children scoring below the PALS benchmarks are designated to receive an additional 2.5 hours of intensive literacy intervention per week until the next screening window, as per an early reading policy initiative originating in the Commonwealth of Virginia. Intervention is to be based on strengths and needs identified through the PALS screening. At subsequent screening windows, children’s performance relative to PALS benchmarks determines whether they continue to receive this additional dose of literacy instruction.

---

Istation Indicators of Progress (ISIP) Reading

Classification Accuracy

Select time of year

Fall

Winter

Spring

Describe the criterion (outcome) measure(s) including the degree to which it/they is/are independent from the screening measure.

iSTATION DESCRIPTION: Istation Indicators of Progress (the formative assessment series) for Reading in kindergarten are individually given computer adaptive tests with national norms that examine foundational skills in early literacy related to Listening Comprehension, letter knowledge, vocabulary, phonemic awareness. The total score allows children to have a Lexile equivalent score, percentile rank, grade equivalency, and also places children in a 1-3 categorical ‘priority’ group. This measure is entirely separate from the PALS but is based on a similar theoretical basis of early foundational skills in reading. The PALS test has specific overlap with iStation in its assessment of letter knowledge and phonemic awareness. The PALS additionally tests for spelling, concept of word, letter sound knowledge, and phonological awareness. It does not directly assess vocabulary or listening comprehension. However, given the strong correlation among oral and written language skills in the early grades (K), both iStation and PALS are seen to be screeners of early reading foundational skills and are expected to function similarly in the field. We focused on the relation of the percentiles of children’s performance based on PALS sum scores and iStation Lexile scores.

Do the classification accuracy analyses examine concurrent and/or predictive classification?

Concurrent
Predictive

Describe when screening and criterion measures were administered and provide a justification for why the method(s) you chose (concurrent and/or predictive) is/are appropriate for your tool.

Describe how the classification analyses were performed and cut-points determined. Describe how the cut points align with students at-risk. Please indicate which groups were contrasted in your analyses (e.g., low risk students versus high risk students, low risk students versus moderate risk students).

We used the established cutoffs based on the 20th percentile for outcome measures as well as PALS sum scores. Classifications were based on simple 2 x 2 analyses of risk status on both measures. All classification analyses were performed in R version 3.5.1 “Feather Spray” using the ‘pROC’, ‘caret’, and ‘EvaluationMeasures’ packages.

Were the children in the study/studies involved in an intervention in addition to typical classroom instruction between the screening measure and outcome assessment?

Yes

If yes, please describe the intervention, what children received the intervention, and how they were chosen.

Children scoring below the PALS benchmarks are designated to receive an additional 2.5 hours of intensive literacy intervention per week until the next screening window, as per an early reading policy initiative originating in the Commonwealth of Virginia. Intervention is to be based on strengths and needs identified through the PALS screening. At subsequent screening windows, children’s performance relative to PALS benchmarks determines whether they continue to receive this additional dose of literacy instruction.

Cross-Validation

Has a cross-validation study been conducted?

Yes

If yes,

Select time of year.

Fall

Winter

Spring

Describe the criterion (outcome) measure(s) including the degree to which it/they is/are independent from the screening measure.

iSTATION DESCRIPTION: Istation Indicators of Progress (the formative assessment series) for Reading in kindergarten are individually given computer adaptive tests with national norms that examine foundational skills in early literacy related to Listening Comprehension, letter knowledge, vocabulary, phonemic awareness. The total score allows children to have a Lexile equivalent score, percentile rank, grade equivalency, and also places children in a 1-3 categorical ‘priority’ group. This measure is entirely separate from the PALS but is based on a similar theoretical basis of early foundational skills in reading. The PALS test has specific overlap with iStation in its assessment of letter knowledge and phonemic awareness. The PALS additionally tests for spelling, concept of word, letter sound knowledge, and phonological awareness. It does not directly assess vocabulary or listening comprehension. However, given the strong correlation among oral and written language skills in the early grades (K), both iStation and PALS are seen to be screeners of early reading foundational skills and are expected to function similarly in the field. We focused on the relation of the percentiles of children’s performance based on PALS sum scores and iStation Lexile scores.

Do the cross-validation analyses examine concurrent and/or predictive classification?

Concurrent
Predictive

Describe when screening and criterion measures were administered and provide a justification for why the method(s) you chose (concurrent and/or predictive) is/are appropriate for your tool.

Describe how the cross-validation analyses were performed and cut-points determined. Describe how the cut points align with students at-risk. Please indicate which groups were contrasted in your analyses (e.g., low risk students versus high risk students, low risk students versus moderate risk students).

We used the established cutoffs based on the 20th percentile for outcome measures as well as PALS sum scores. Classifications were based on simple 2 x 2 analyses of risk status on both measures. All classification analyses were performed in R version 3.5.1 “Feather Spray” using the ‘pROC’, ‘caret’, and ‘EvaluationMeasures’ packages.

Were the children in the study/studies involved in an intervention in addition to typical classroom instruction between the screening measure and outcome assessment?

Yes

If yes, please describe the intervention, what children received the intervention, and how they were chosen.

Children scoring below the PALS benchmarks are designated to receive an additional 2.5 hours of intensive literacy intervention per week until the next screening window, as per an early reading policy initiative originating in the Commonwealth of Virginia. Intervention is to be based on strengths and needs identified through the PALS screening. At subsequent screening windows, children’s performance relative to PALS benchmarks determines whether they continue to receive this additional dose of literacy instruction.

---

Classification Accuracy - Spring

Evidence	Kindergarten	Kindergarten
Criterion measure	MAP	Istation Indicators of Progress (ISIP) Reading
Cut Points - Percentile rank on criterion measure	20	20
Cut Points - Performance score on criterion measure	CO Students: 149 / WI Students: 150	194.35
Cut Points - Corresponding performance score (numeric) on screener measure	90	90
Classification Data - True Positive (a)
Classification Data - False Positive (b)
Classification Data - False Negative (c)
Classification Data - True Negative (d)
Area Under the Curve (AUC)	0.74	0.74
AUC Estimate’s 95% Confidence Interval: Lower Bound	0.72	0.72
AUC Estimate’s 95% Confidence Interval: Upper Bound	0.76	0.76

Statistics	Kindergarten	Kindergarten
Base Rate
Overall Classification Rate
Sensitivity
Specificity
False Positive Rate
False Negative Rate
Positive Predictive Power
Negative Predictive Power

Sample	Kindergarten	Kindergarten
Date	Spring	Spring
Sample Size
Geographic Representation	East North Central (WI) Mountain (CO)	South Atlantic (VA)
Male
Female
Other
Gender Unknown
White, Non-Hispanic
Black, Non-Hispanic
Hispanic
Asian/Pacific Islander
American Indian/Alaska Native
Other
Race / Ethnicity Unknown
Low SES
IEP or diagnosed disability
English Language Learner

Cross-Validation - Spring

Evidence	Kindergarten	Kindergarten
Criterion measure	MAP	Istation Indicators of Progress (ISIP) Reading
Cut Points - Percentile rank on criterion measure	20	20
Cut Points - Performance score on criterion measure	CO Students: 149 / WI Students: 150	194.35
Cut Points - Corresponding performance score (numeric) on screener measure	90	90
Classification Data - True Positive (a)
Classification Data - False Positive (b)
Classification Data - False Negative (c)
Classification Data - True Negative (d)
Area Under the Curve (AUC)	0.71	0.71
AUC Estimate’s 95% Confidence Interval: Lower Bound	0.69	0.69
AUC Estimate’s 95% Confidence Interval: Upper Bound	0.72	0.72

Statistics	Kindergarten	Kindergarten
Base Rate
Overall Classification Rate
Sensitivity
Specificity
False Positive Rate
False Negative Rate
Positive Predictive Power
Negative Predictive Power

Sample	Kindergarten	Kindergarten
Date	Spring	Spring
Sample Size
Geographic Representation	East North Central (WI) Mountain (CO)	South Atlantic (VA)
Male
Female
Other
Gender Unknown
White, Non-Hispanic
Black, Non-Hispanic
Hispanic
Asian/Pacific Islander
American Indian/Alaska Native
Other
Race / Ethnicity Unknown
Low SES
IEP or diagnosed disability
English Language Learner

Reliability

Grade	Kindergarten
Rating	d

*Offer a justification for each type of reliability reported, given the type and purpose of the tool.

The first reliability estimate is coefficient alpha. It is appropriate because of the use of task scores when making instructional decisions. The confidence interval for coefficient alpha is based on Felt’s derivation of the sampling distribution of coefficient alpha. The second type of reliability estimate is Pearson correlation coefficients between raters’ subtask scores. It is appropriate to determine that two scorers can reach nearly the same score while observing the same child. Test-retest reliability is the correlation between the same child’s scores on the test when administered at different points in time. Test-retest reliability is crucial in determining stability over time.

*Describe the sample(s), including size and characteristics, for each reliability analysis conducted.

For PALS-K, the coefficient alphas are based on a sample of 114,686 Kindergarten students in the Commonwealth of Virginia. The percentage of males (52%) and females (48%) was similar. The percentage of white (51% and non-white students (49%) was also similar. About 11% were ESL students. Inter-rater reliabilities were calculated from studies conducted in Fall 1997 and Spring 1999. One person administered the PALS-K subtasks while a second person observed and scored at the same time. Test-retest reliability were calculated from the Fall of 2002 sample of 473 students. Students were tested a second time no more than two weeks from the first administration.

*Describe the analysis procedures for each reported type of reliability.

For the coefficient alphas, the analysis procedures involved subsetting the data for each subgroup and computing coefficient alpha and its confidence interval. For inter-rater reliabilities, the scores from the two raters were correlated (Table 14 in the PALS K Technical Manual). Test-retest reliabilities were calculated using the Pearson correlation between the two administrations (Table 11 in the PALS K Technical Manual).

*In the table(s) below, report the results of the reliability analyses described above (e.g., internal consistency or inter-rater reliability coefficients).

Type of	Subgroup	Informant	Age / Grade	Test or Criterion	n	Median Coefficient	95% Confidence Interval Lower Bound	95% Confidence Interval Upper Bound

Results from other forms of reliability analysis not compatible with above table format:

Manual cites other published reliability studies:

No

Provide citations for additional published studies.

Do you have reliability data that are disaggregated by gender, race/ethnicity, or other subgroups (e.g., English language learners, students with disabilities)?

Yes

If yes, fill in data for each subgroup with disaggregated reliability data.

Type of	Subgroup	Informant	Age / Grade	Test or Criterion	n	Median Coefficient	95% Confidence Interval Lower Bound	95% Confidence Interval Upper Bound

Results from other forms of reliability analysis not compatible with above table format:

Additional disaggregated data is available from the Center upon request.

Manual cites other published reliability studies:

No

Provide citations for additional published studies.

Validity

Grade	Kindergarten
Rating

*Describe each criterion measure used and explain why each measure is appropriate, given the type and purpose of the tool.

Stanford Achievement Test, developed by Pearson, is a set of standardized achievement tests used by school districts in the United States and in American schools abroad for assessing children from kindergarten through high school.

*Describe the sample(s), including size and characteristics, for each validity analysis conducted.

The analysis relating PALS-K to the Stanford Achievement Test is based on 78 males and 69 female kindergartners (11.6% Black or African American; 82.3% White; 4.8% Hispanic; 1.4% Other) from two schools in Virginia (1 rural, 1 suburban). 4.1% were ESL students and 6.8% received special education services (Developmental Delay, 2%; Speech & Language, 4.8%)

*Describe the analysis procedures for each reported type of validity.

Correlation between Fall PALS-K Summed Score and Stanford-9 Total reading scaled score.

*In the table below, report the results of the validity analyses described above (e.g., concurrent or predictive validity, evidence based on response processes, evidence based on internal structure, evidence based on relations to other variables, and/or evidence based on consequences of testing), and the criterion measures.

Type of	Subgroup	Informant	Age / Grade	Test or Criterion	n	Median Coefficient	95% Confidence Interval Lower Bound	95% Confidence Interval Upper Bound

Results from other forms of validity analysis not compatible with above table format:

1) “Exploratory factor analysis revealed a theoretically defensible measurement structure that was found to replicate in a randomly selected hold-out sample when examined through the lens of confirmatory factor analytic methods. Multigroup latent variable comparisons between Spanish-speaking English-language learners (ELLS) and non-ELL students largely demonstrated the PALS-K to yield configural and metric invariance with respect to associations between subtests and latent dimensions.” (Huang & Konold, 2014) 2) Construct Validity - As one piece of evidence in support of the construct validity of PALS-K, we conduct Principal Component Analyses on the tasks that make up the PALS-K Summed Score each year, based on statewide data. These analyses consistently yield a single factor (eigenvalue greater than 1.0) that accounts for approximately two-thirds of the variance (.65 in Spring 2008) in these data. 3) Content Validity - The content validity is supported by the process of item development (described in the Technical Reference, Section III, pp. 11 – 19). The guiding principles underlying this process were that (a) tasks and items were a representative sample of tasks from other early literacy screening instruments, (b) items had a history of use in phonological awareness and early literacy research, and (c) items were aligned with Virginia’s Standards of Learning for English (Reading). Each of these principles was further supported by the process of review by an Advisory Panel of literacy professionals from across Virginia, and by an External Review Panel, which consisted of nationally recognized experts in the field of reading, communication sciences, or psychology.

Manual cites other published reliability studies:

Yes

Provide citations for additional published studies.

1) Huang, F. L., & Konold, T. R. (2014). A latent variable investigation of the Phonological Awareness Literacy Screening-Kindergarten assessment: Construct identification and multigroup comparisons between Spanish-speaking English-language learners (ELLs) and non-ELL students. Language Testing, 31(2), 205-221. 2) Invernizzi, M., Robey, R., & Moon, T. (1999). Phonological Awareness Literacy Screening (PALS) 1997-1998: Description of sample, first-year results, task analyses, and revisions. Technical manual and report prepared for the Virginia Department of Education. Charlottesville, VA: University Printing Services.

Describe the degree to which the provided data support the validity of the tool.

Do you have validity data that are disaggregated by gender, race/ethnicity, or other subgroups (e.g., English language learners, students with disabilities)?

No

If yes, fill in data for each subgroup with disaggregated validity data.

Type of	Subgroup	Informant	Age / Grade	Test or Criterion	n	Median Coefficient	95% Confidence Interval Lower Bound	95% Confidence Interval Upper Bound

Results from other forms of validity analysis not compatible with above table format:

Manual cites other published reliability studies:

No

Provide citations for additional published studies.

Bias Analysis

Grade	Kindergarten
Rating	Yes

Have you conducted additional analyses related to the extent to which your tool is or is not biased against subgroups (e.g., race/ethnicity, gender, socioeconomic status, students with disabilities, English language learners)? Examples might include Differential Item Functioning (DIF) or invariance testing in multiple-group confirmatory factor models.

Yes

If yes,

a. Describe the method used to determine the presence or absence of bias:

We used the Mantel-Haenszel procedure to test for differential item functioning (DIF). We combined this hypothesis testing procedures with the common-odds ratio effect size to determine the practical significance of DIF. We followed the ETS classification criteria of A, B, C to indicate no, moderate, and large amount of DIF

b. Describe the subgroups for which bias analyses were conducted:

Identified/Not Identified (for intensive intervention) (see Table 16, p. 33 of Technical Reference); White/Non-White (see table below; ESL/Non-ESL (see table below); Male/Female (see table below)

c. Describe the results of the bias analyses conducted, including data and interpretative statements. Include magnitude of effect (if available) if bias has been identified.

see doc attached in validity section