Phonological Awareness Literacy Screening (PALS)

PALS for Grades 1-3

Cost

Technology, Human Resources, and Accommodations for Special Needs

Service and Support

Purpose and Other Implementation Information

Usage and Reporting

Initial Cost:

Free for public school classrooms (preschool through grade 3) in Virginia.

 

Basic: 5.50 per student*

Premier: 7.25 per student*

*Minimum of 20 students

 

Replacement Cost:

Annual license renewal fee subject to change.

 

Included in Cost:

Materials include an Introduction and Overview, an Administration and Scoring Guide, teacher materials, student materials, information regarding interpreting PALS 1-3 results, and a PALS 1-3 Technical Reference. Not included in the quoted cost are one-time shipping fees for student materials/binders.

 

Technology Requirements:

  • No technology is required*

*Access to the online scoring system and online reports would require computer and internet but it is not required to provide the assessment or score it as this can be done manually.

 

Training Requirements:

  • Less than 1 hour of training

 

Qualified Administrators:

  • No minimum qualifications specified

 

Accommodations:

Large Print is available, as is information on using the PALS 1-3 assessment on students with disabilities that require a non-standard approach to administering the assessment.

 

Where to Obtain:

Contact Information for Public Schools in Virginia:

Website: pals.virginia.edu

Address: UVA/PALS PO Box 800785, Charlottesville, VA 22908-8785

Phone number: 888-UVA-PALS

Email: pals@virginia.edu

 

Contact Information for Other Schools:

Website: www.ioeducation.com/pals

Address: IO Education/ Illuminate, ATTN: PALS, 1380 Peachtree Industrial Boulevard, Suite 200, Suwanee, GA 30024

Phone number: 866-817-0726 ext. 6

Email: support@palshelp.com


Access to Technical Support:

Options include email and phone technical and literacy support, data analytics, online self-paced literacy courses, monthly topical webinars, personalized PD, online PD modules, and online PD.

 

The Phonological Awareness Literacy Screening (PALS) is a criterion-referenced screening, diagnostic, and progress monitoring tool.  PALS consists of two instruments (PALS-K and PALS 1-3) that measure young children’s knowledge of important literacy fundamentals: phonological awareness, alphabet awareness, letter sound knowledge, spelling, concept of word, word recognition in isolation, and oral passage reading.

 

The major purpose of PALS is to identify students who may be at risk for reading difficulties and who need intensive literacy intervention beyond what is provided to typically developing readers.  The second purpose of PALS is to provide teachers with explicit diagnostic information about what students know and need to know about the fundamental components of literacy that may be used to target instruction to meet students’ needs. The third purpose of PALS is to monitor students’ progress and determine the effectiveness of instruction or intervention.

 

PALS 1-3 uses a gated three-tiered approach that differentiates the administration of the assessment based on students’ scores. The first tier (or Entry Level) includes a screening measure of word recognition and spelling.  The Entry Level also assesses the accuracy, fluency, rate, and comprehension of a student’s oral reading in context.  If students do not meet the Entry Level Summed Score Benchmark, which is based on the screening tasks, they are routed to Level B for further diagnosis. Level B (Alphabetics) assesses emergent and beginning reading essentials in alphabet knowledge and concept of word.  If the Level B Summed Score Benchmark is not met, students are routed to Level C (Phonemic Awareness) for a more in-depth evaluation of phonemic awareness skills, including blending and segmenting speech sounds.

Assessment Format:

  • Direct observation
  • Direct: Computerized
  • One-to-one

 

Administration Time:

  • 20-40 minutes per student
  • 10 minutes per group

 

Scoring Time:

  • 5 minutes per student*
  • 10-20 minutes per group*

*If using the Online Assessment Wizard, scoring is automatic.

 

Scoring Method:

  • Calculated manually*

*If using the Online Assessment Wizard, scoring is automatic.

 

Scores Generated:

  • Developmental benchmarks
  • Developmental cut points
  • Composite scores
  • Subscale/subtest scores
  • Item level responses

 

Classification Accuracy

Grade123
Criterion 1 Falldashdashdash
Criterion 1 Winterdashdashdash
Criterion 1 SpringHalf-filled bubbledHalf-filled bubbledHalf-filled bubbled
Criterion 2 Falldashdashdash
Criterion 2 Winterdashdashdash
Criterion 2 SpringHalf-filled bubbledHalf-filled bubbledHalf-filled bubbled

Primary Sample

 

Criterion 1, Spring

Grade

1

2

3

Criterion

MAP

MAP

MAP

Cut points: Percentile rank on criterion measure

Not Provided

Not Provided

Not Provided

Cut points: Performance score (numeric) on criterion measure

161

175

186

Cut points: Corresponding performance score (numeric) on screener measure

35

56

65

Base rate in the sample for children requiring intensive intervention

0.23

0.18

0.22

False Positive Rate

0.03

0.12

0.07

False Negative Rate

0.45

0.29

0.33

Sensitivity

0.55

0.71

0.67

Specificity

0.97

0.88

0.93

Positive Predictive Power

0.84

0.57

0.72

Negative Predictive Power

0.88

0.93

0.91

Overall Classification Rate

0.87

0.85

0.87

Area Under the Curve (AUC)

0.76

0.80

0.80

AUC 95% Confidence Interval Lower Bound

0.74

0.77

0.77

AUC 95% Confidence Interval Upper Bound

0.78

0.82

0.82

 

Criterion 2, Spring

Grade

1

2

3

Criterion

STAR

STAR

Reading SOL

Cut points: Percentile rank on criterion measure

Not Provided

Not Provided

Not Provided

Cut points: Performance score (numeric) on criterion measure

106

235

400

Cut points: Corresponding performance score (numeric) on screener measure

35

56

65

Base rate in the sample for children requiring intensive intervention

0.25

0.25

0.27

False Positive Rate

0.12

0.16

0.19

False Negative Rate

0.28

0.28

0.38

Sensitivity

0.72

0.72

0.62

Specificity

0.88

0.84

0.81

Positive Predictive Power

0.67

0.60

0.55

Negative Predictive Power

0.91

0.90

0.85

Overall Classification Rate

0.84

0.81

0.76

Area Under the Curve (AUC)

0.80

0.78

0.72

AUC 95% Confidence Interval Lower Bound

0.78

0.76

0.71

AUC 95% Confidence Interval Upper Bound

0.82

0.80

0.73

 

Additional Classification Accuracy

The following are provided for context and did not factor into the Classification Accuracy ratings.

 

Cross-Validation Sample

 

Criterion 1, Spring

Grade

1

2

3

Criterion

MAP

MAP

MAP

Cut points: Percentile rank on criterion measure

Not Provided

Not Provided

Not Provided

Cut points: Performance score (numeric) on criterion measure

161

175

186

Cut points: Corresponding performance score (numeric) on screener measure

35

56

65

Base rate in the sample for children requiring intensive intervention

0.33

0.20

0.21

False Positive Rate

0.04

0.10

0.07

False Negative Rate

0.52

0.35

0.34

Sensitivity

0.48

0.65

0.66

Specificity

0.96

0.90

0.93

Positive Predictive Power

0.86

0.62

0.72

Negative Predictive Power

0.79

0.91

0.91

Overall Classification Rate

0.80

0.85

0.87

Area Under the Curve (AUC)

0.72

0.78

0.80

AUC 95% Confidence Interval Lower Bound

0.70

0.76

0.77

AUC 95% Confidence Interval Upper Bound

0.75

0.80

0.82

 

Criterion 2, Spring

Grade

1

2

3

Criterion

STAR

STAR

Reading SOL

Cut points: Percentile rank on criterion measure

Not Provided

Not Provided

Not Provided

Cut points: Performance score (numeric) on criterion measure

106

235

400

Cut points: Corresponding performance score (numeric) on screener measure

35

56

65

Base rate in the sample for children requiring intensive intervention

0.406

0.406

0.485

False Positive Rate

0.12

0.16

0.19

False Negative Rate

0.28

0.26

0.38

Sensitivity

0.72

0.74

0.62

Specificity

0.89

0.85

0.81

Positive Predictive Power

0.82

0.77

0.76

Negative Predictive Power

0.82

0.83

0.69

Overall Classification Rate

0.82

0.80

0.72

Area Under the Curve (AUC)

0.80

0.79

0.72

AUC 95% Confidence Interval Lower Bound

0.79

0.78

0.71

AUC 95% Confidence Interval Upper Bound

0.82

0.81

0.72

 

Disaggregated Data

 

Criterion 1, Spring

Subgroup: Female

Grade

1

2

3

Criterion

MAP

MAP

MAP

Cut points: Percentile rank on criterion measure

Not Provided

Not Provided

Not Provided

Cut points: Performance score (numeric) on criterion measure

161

175

186

Cut points: Corresponding performance score (numeric) on screener measure

35

56

65

Base rate in the sample for children requiring intensive intervention

0.19

0.17

0.21

False Positive Rate

0.02

0.10

0.07

False Negative Rate

0.43

0.32

0.36

Sensitivity

0.57

0.68

0.64

Specificity

0.98

0.90

0.93

Positive Predictive Power

0.86

0.57

0.71

Negative Predictive Power

0.81

0.93

0.91

Overall Classification Rate

0.90

0.86

0.87

Area Under the Curve (AUC)

0.78

0.79

0.78

AUC 95% Confidence Interval Lower Bound

0.74

0.75

0.74

AUC 95% Confidence Interval Upper Bound

0.81

0.82

0.82

 

Criterion 1, Spring

Subgroup: Male

Grade

1

2

3

Criterion

MAP

MAP

MAP

Cut points: Percentile rank on criterion measure

Not Provided

Not Provided

Not Provided

Cut points: Performance score (numeric) on criterion measure

161

175

186

Cut points: Corresponding performance score (numeric) on screener measure

35

56

65

Base rate in the sample for children requiring intensive intervention

0.264

0.190

0.233

False Positive Rate

0.04

0.13

0.08

False Negative Rate

0.47

0.27

0.31

Sensitivity

0.54

0.73

0.69

Specificity

0.957

0.869

0.92

Positive Predictive Power

0.816

0.567

0.73

Negative Predictive Power

0.85

0.93

0.91

Overall Classification Rate

0.85

0.84

0.87

Area Under the Curve (AUC)

0.75

0.80

0.81

AUC 95% Confidence Interval Lower Bound

0.72

0.77

0.77

AUC 95% Confidence Interval Upper Bound

0.78

0.83

0.84

 

Criterion 2, Spring

Subgroup: Female

Grade

1

2

3

Criterion

STAR

STAR

Reading SOL

Cut points: Percentile rank on criterion measure

Not Provided

Not Provided

Not Provided

Cut points: Performance score (numeric) on criterion measure

106

235

400

Cut points: Corresponding performance score (numeric) on screener measure

35

56

65

Base rate in the sample for children requiring intensive intervention

0.22

0.21

0.24

False Positive Rate

0.11

0.15

0.17

False Negative Rate

0.32

0.26

0.39

Sensitivity

0.68

0.74

0.61

Specificity

0.89

0.85

0.83

Positive Predictive Power

0.64

0.57

0.54

Negative Predictive Power

0.91

0.92

0.87

Overall Classification Rate

0.85

0.82

0.78

Area Under the Curve (AUC)

0.79

0.79

0.72

AUC 95% Confidence Interval Lower Bound

0.76

0.76

0.71

AUC 95% Confidence Interval Upper Bound

0.82

0.83

0.73

 

Criterion 2, Spring

Subgroup: Male

Grade

1

2

3

Criterion

STAR

STAR

Reading SOL

Cut points: Percentile rank on criterion measure

Not Provided

Not Provided

Not Provided

Cut points: Performance score (numeric) on criterion measure

106

235

400

Cut points: Corresponding performance score (numeric) on screener measure

35

56

65

Base rate in the sample for children requiring intensive intervention

0.28

0.19

0.29

False Positive Rate

0.13

0.13

0.16

False Negative Rate

0.24

0.27

0.30

Sensitivity

0.76

0.73

0.70

Specificity

0.87

0.87

0.84

Positive Predictive Power

0.70

0.57

0.63

Negative Predictive Power

0.90

0.93

0.88

Overall Classification Rate

0.84

0.84

0.80

Area Under the Curve (AUC)

0.81

0.80

0.77

AUC 95% Confidence Interval Lower Bound

0.79

0.77

0.74

AUC 95% Confidence Interval Upper Bound

0.84

0.83

0.80

 

Criterion 1, Spring

Subgroup: ELL

Grade

1

2

3

Criterion

MAP

MAP

MAP

Cut points: Percentile rank on criterion measure

Not Provided

Not Provided

Not Provided

Cut points: Performance score (numeric) on criterion measure

161

175

186

Cut points: Corresponding performance score (numeric) on screener measure

35

56

65

Base rate in the sample for children requiring intensive intervention

0.24

0.23

0.42

False Positive Rate

0.04

0.13

0.13

False Negative Rate

0.43

0.28

0.32

Sensitivity

0.57

0.72

0.68

Specificity

0.96

0.87

0.87

Positive Predictive Power

0.82

0.62

0.79

Negative Predictive Power

0.88

0.91

0.79

Overall Classification Rate

0.87

0.83

0.79

Area Under the Curve (AUC)

0.77

0.79

0.78

AUC 95% Confidence Interval Lower Bound

0.73

0.75

0.73

AUC 95% Confidence Interval Upper Bound

0.80

0.84

0.82

 

Criterion 2, Spring

Subgroup: ELL

Grade

1

2

3

Criterion

STAR

STAR

Reading SOL

Cut points: Percentile rank on criterion measure

Not Provided

Not Provided

Not Provided

Cut points: Performance score (numeric) on criterion measure

106

235

400

Cut points: Corresponding performance score (numeric) on screener measure

35

56

65

Base rate in the sample for children requiring intensive intervention

0.16

0.23

0.31

False Positive Rate

0.07

0.13

0.18

False Negative Rate

0.33

0.28

0.38

Sensitivity

0.67

0.72

0.62

Specificity

0.93

0.87

0.82

Positive Predictive Power

0.65

0.62

0.60

Negative Predictive Power

0.94

0.91

0.83

Overall Classification Rate

0.89

0.83

0.76

Area Under the Curve (AUC)

0.80

0.79

0.72

AUC 95% Confidence Interval Lower Bound

0.73

0.75

0.62

AUC 95% Confidence Interval Upper Bound

0.87

0.84

0.81

 

Criterion 1, Spring

Subgroup: Non-ELL

Grade

1

2

3

Criterion

MAP

MAP

MAP

Cut points: Percentile rank on criterion measure

Not Provided

Not Provided

Not Provided

Cut points: Performance score (numeric) on criterion measure

161

175

186

Cut points: Corresponding performance score (numeric) on screener measure

35

56

65

Base rate in the sample for children requiring intensive intervention

0.23

0.17

0.17

False Positive Rate

0.03

0.11

0.06

False Negative Rate

0.47

0.30

0.34

Sensitivity

0.53

0.70

0.66

Specificity

0.97

0.89

0.94

Positive Predictive Power

0.85

0.55

0.68

Negative Predictive Power

0.88

0.94

0.93

Overall Classification Rate

0.87

0.86

0.89

Area Under the Curve (AUC)

0.75

0.79

0.80

AUC 95% Confidence Interval Lower Bound

0.72

0.77

0.76

AUC 95% Confidence Interval Upper Bound

0.78

0.82

0.83

 

Criterion 2, Spring

Subgroup: Non-ELL

Grade

1

2

3

Criterion

STAR

STAR

Reading SOL

Cut points: Percentile rank on criterion measure

Not Provided

Not Provided

Not Provided

Cut points: Performance score (numeric) on criterion measure

106

235

400

Cut points: Corresponding performance score (numeric) on screener measure

35

56

65

Base rate in the sample for children requiring intensive intervention

0.26

0.25

0.25

False Positive Rate

0.12

0.16

0.18

False Negative Rate

0.27

0.27

0.39

Sensitivity

0.73

0.73

0.61

Specificity

0.88

0.84

0.82

Positive Predictive Power

0.67

0.60

0.52

Negative Predictive Power

0.90

0.91

0.86

Overall Classification Rate

0.84

0.82

0.77

Area Under the Curve (AUC)

0.80

0.79

0.71

AUC 95% Confidence Interval Lower Bound

0.78

0.76

0.70

AUC 95% Confidence Interval Upper Bound

0.82

0.81

0.72

 

Criterion 1, Spring

Subgroup: Non-White

Grade

1

2

3

Criterion

MAP

MAP

MAP

Cut points: Percentile rank on criterion measure

Not Provided

Not Provided

Not Provided

Cut points: Performance score (numeric) on criterion measure

161

175

186

Cut points: Corresponding performance score (numeric) on screener measure

35

56

65

Base rate in the sample for children requiring intensive intervention

0.32

0.23

0.29

False Positive Rate

0.04

0.14

0.09

False Negative Rate

0.42

0.28

0.32

Sensitivity

0.58

0.72

0.68

Specificity

0.96

0.86

0.91

Positive Predictive Power

0.88

0.61

0.75

Negative Predictive Power

0.83

0.91

0.87

Overall Classification Rate

0.84

0.83

0.84

Area Under the Curve (AUC)

0.77

0.79

0.79

AUC 95% Confidence Interval Lower Bound

0.74

0.76

0.76

AUC 95% Confidence Interval Upper Bound

0.80

0.82

0.83

 

Criterion 2, Spring

Subgroup: Non-White

Grade

1

2

3

Criterion

STAR

STAR

Reading SOL

Cut points: Percentile rank on criterion measure

Not Provided

Not Provided

Not Provided

Cut points: Performance score (numeric) on criterion measure

106

235

400

Cut points: Corresponding performance score (numeric) on screener measure

35

56

65

Base rate in the sample for children requiring intensive intervention

0.27

0.27

0.35

False Positive Rate

0.12

0.17

0.20

False Negative Rate

0.28

0.28

0.37

Sensitivity

0.73

0.72

0.63

Specificity

0.88

0.83

0.80

Positive Predictive Power

0.69

0.61

0.63

Negative Predictive Power

0.90

0.89

0.80

Overall Classification Rate

0.84

0.80

0.74

Area Under the Curve (AUC)

0.80

0.78

0.72

AUC 95% Confidence Interval Lower Bound

0.78

0.75

0.71

AUC 95% Confidence Interval Upper Bound

0.82

0.80

0.73

 

Criterion 1, Spring

Subgroup: White

Grade

1

2

3

Criterion

MAP

MAP

MAP

Cut points: Percentile rank on criterion measure

Not Provided

Not Provided

Not Provided

Cut points: Performance score (numeric) on criterion measure

161

175

186

Cut points: Corresponding performance score (numeric) on screener measure

35

56

65

Base rate in the sample for children requiring intensive intervention

0.17

0.12

0.15

False Positive Rate

0.03

0.10

0.06

False Negative Rate

0.49

0.33

0.36

Sensitivity

0.51

0.68

0.64

Specificity

0.97

0.90

0.94

Positive Predictive Power

0.77

0.49

0.66

Negative Predictive Power

0.71

0.95

0.94

Overall Classification Rate

0.89

0.88

0.90

Area Under the Curve (AUC)

0.74

0.79

0.79

AUC 95% Confidence Interval Lower Bound

0.71

0.75

0.75

AUC 95% Confidence Interval Upper Bound

0.78

0.83

0.84

 

Criterion 2, Spring

Subgroup: White

Grade

1

2

3

Criterion

STAR

STAR

Reading SOL

Cut points: Percentile rank on criterion measure

Not Provided

Not Provided

Not Provided

Cut points: Performance score (numeric) on criterion measure

106

235

400

Cut points: Corresponding performance score (numeric) on screener measure

35

56

65

Base rate in the sample for children requiring intensive intervention

0.16

0.13

0.19

False Positive Rate

0.10

0.09

0.18

False Negative Rate

0.29

0.34

0.39

Sensitivity

0.71

0.66

0.61

Specificity

0.90

0.91

0.82

Positive Predictive Power

0.58

0.51

0.45

Negative Predictive Power

0.94

0.95

0.90

Overall Classification Rate

0.87

0.88

0.78

Area Under the Curve (AUC)

0.81

0.78

0.72

AUC 95% Confidence Interval Lower Bound

0.75

0.71

0.70

AUC 95% Confidence Interval Upper Bound

0.86

0.86

0.73

 

Reliability

Grade123
RatingFull bubbledFull bubbledFull bubbled
  1. Justification for each type of reliability reported, given the type and purpose of the tool: The first reliability estimate is coefficient alpha. It is appropriate because of the use of task scores when making instructional decisions. The confidence interval for coefficient alpha is based on Felt’s derivation of the sampling distribution of coefficient alpha. The second type of reliability estimate is Pearson correlation coefficients between raters’ subtask scores. It is appropriate to determine that two scorers can reach nearly the same score while observing the same child. Test-retest reliability is the correlation between the same child’s scores on the test when administered at different points in time. Test-retest reliability is crucial in determining stability over time.

 

  1. Description of the sample(s), including size and characteristics, for each reliability analysis conducted: Coefficient alphas are calculated from a sample of NNN students in the Commonwealth of Virginia. 60% of the students were male and most were non-white (64%). About 23% of the sample was ESL. A large percentage of the sample was in first grade (81%), but some second (15%) and third grade students were also involved (4%). Sample sizes vary by grade level as indicated in the table below. Inter-rater reliability scores for word recognition were calculated using a sample from Fall 2000 consisting of 478 students. Inter-rater reliability scores for spelling were calculated using a sample from Fall 2001 consisting of 2,137 students. Test-retest reliabilities were calculated from a sample from Fall 2002 consisting of 204 students.

 

  1. Description of the analysis procedures for each reported type of reliability: Test-retest reliability was calculated as the Pearson coefficient between the two administrations (Table 22 in the PALS 1-3 Technical Manual). Inter-rater reliability was calculated as the correlation coefficient between the two raters’ scores (Table 23 in the PALS 1-3 Technical Manual). For the coefficient alphas, the analysis procedures involved subsetting the data for each subgroup and computing coefficient alpha and its confidence interval.

 

  1. Reliability of performance level score (e.g., model-based, internal consistency, inter-rater reliability).

Type of Reliability

Age or Grade

n

Coefficient

95% Confidence Interval: Lower Bound

95% Confidence Interval: Upper Bound

 Test-retest

1 – Letter Sounds

77

0.90

0.85

0.94

 Test-retest

1 – Spelling

77

0.92

0.88

0.95

 Test-retest

1 – Pre-primer Word List

77

0.89

0.83

0.93

 Test-retest

1 – Sum Score

77

0.96

0.94

0.97

 Test-retest

2 – Spelling

59

0.89

0.82

0.93

 Test-retest

2 – 1st Grade Word List

59

0.88

0.81

0.93

 Test-retest

2 – Sum Score

59

0.92

0.87

0.95

 Test-retest

3 – Spelling

68

0.95

0.92

0.97

 Test-retest

3 – 2nd Grade Word List

68

0.93

0.89

0.96

 Test-retest

3 – Sum Score

68

0.97

0.95

0.98

 Inter-rater

1 – Word Recognition Pre-primer

51

0.99

0.98

0.99

 Inter-rater

1 - Word Recognition Primer

52

0.99

0.98

0.99

 Inter-rater

1 – Word Recognition

45

0.98

0.96

0.99

 Inter-rater

2 – Word Recognition

63

0.98

0.97

0.99

 Inter-rater

3 – Word Recognition

46

0.98

0.96

0.99

 Inter-rater

1 – Spelling

375

0.99

0.99

0.99

 Inter-rater

2 – Spelling

276

0.99

0.99

0.99

 Inter-rater

3 - Spelling

257

0.99

0.99

0.99

 Coefficient Alpha

1 – Spelling

135,058

0.93

0.93

0.93

 Coefficient Alpha

1 – Spelling

135,058

0.93

0.93

0.93

Coefficient Alpha

1 – Word Recognition Pre-primer

72,838

0.93

0.93

0.93

Coefficient Alpha

1 – Word Recognition Primer

68,999

0.76

0.76

0.76

 

Coefficient Alpha

1 – Word Recognition 1

95,315

0.92

0.92

0.921

Coefficient Alpha

2 – Spelling

124,589

0.95

0.95

0.95

 Coefficient Alpha

2 – Word Recognition Pre-primer

10,972

0.90

0.90

0.91

 Coefficient Alpha

2 – Word Recognition Primer

24,793

0.753

0.7486

0.76

Coefficient Alpha

2 – Word Recognition 1

62,573

0.93

0.93

0.93

Coefficient Alpha

3 – Spelling

53,599

0.94

0.94

0.94

Coefficient Alpha

3 – Word Recognition Pre-primer

2,367

0.92

0.91

0.92

Coefficient Alpha

3 – Word Recognition Primer

5,661

0.75

0.74

0.76

 

Disaggregated Reliability

The following disaggregated reliability data are provided for context and did not factor into the Reliability rating.

Type of Reliability

Subgroup

Age or Grade

n

Coefficient

95% Confidence Interval: Lower Bound

95% Confidence Interval: Upper Bound

Coefficient Alpha

Female

1 – Spelling

65,970

0.93

0.93

0.93

Coefficient Alpha

Female

1 – Word Recognition Preprimer

34,808

0.93

0.92

0.93

Coefficient Alpha

Female

1 – Word Recognition Primer

33,599

0.76

0.75

0.76

Coefficient Alpha

Female

1 – Word Recognition 1

46,737

0.91

0.91

0.91

Coefficient Alpha

Female

2 – Spelling

61,013

0.95

0.94

0.95

Coefficient Alpha

Female

2 – Word Recognition Preprimer

4,543

0.90

0.89

0.90

Coefficient Alpha

Female

2 – Word Recognition Primer

11,198

0.75

0.74

0.76

Coefficient Alpha

Female

2 – Word Recognition 1

29,968

0.92

0.92

0.92

Coefficient Alpha

Female

3 – Spelling

25,905

0.94

0.94

0.94

Coefficient Alpha

Female

3 – Word Recognition Preprimer

871

0.92

0.91

0.92

Coefficient Alpha

Female

3 – Word Recognition Primer

2,241

0.75

0.74

0.77

Coefficient Alpha

Female

3 – Word Recognition 1

3,061

0.76

0.86

0.87

Coefficient Alpha

Male

1 – Spelling

69,088

0.93

0.93

0.93

Coefficient Alpha

Male

1 – Word Recognition Preprimer

38,030

0.94

0.93

0.94

Coefficient Alpha

Male

1 – Word Recognition Primer

35,400

0.76

0.76

0.76

Coefficient Alpha

Male

1 – Word Recognition 1

48,578

0.93

0.93

0.93

Coefficient Alpha

Male

2 – Spelling

63,574

0.95

0.95

0.95

Coefficient Alpha

Male

2 – Word Recognition Preprimer

6,429

0.91

0.90

0.91

Coefficient Alpha

Male

2 – Word Recognition Primer

13,595

0.76

0.75

0.76

Coefficient Alpha

Male

2 – Word Recognition 1

32,605

0.94

0.94

0.94

Coefficient Alpha

Male

3 – Spelling

27,694

0.94

0.94

0.94

Coefficient Alpha

Male

3 – Word Recognition Preprimer

1,496

0.92

0.91

0.93

Coefficient Alpha

Male

3 – Word Recognition Primer

3,420

0.75

0.74

0.77

Coefficient Alpha

Male

3 – Word Recognition 1

4,405

0.89

0.88

0.89

Coefficient Alpha

White

1 – Spelling

67,689

0.94

0.93

0.94

Coefficient Alpha

White

1 – Word Recognition Preprimer

34,852

0.92

0.92

0.92

Coefficient Alpha

White

1 – Word Recognition Primer

33,332

0.76

0.75

0.76

Coefficient Alpha

White

1 – Word Recognition 1

48,147

0.91

0.91

0.91

Coefficient Alpha

White

2 – Spelling

61,859

0.95

0.95

0.95

Coefficient Alpha

White

2 – Word Recognition Preprimer

4,170

0.89

0.88

0.89

Coefficient Alpha

White

2 – Word Recognition Primer

10,189

0.75

0.74

0.76

Coefficient Alpha

White

2 – Word Recognition 1

29,945

0.93

0.92

0.93

Coefficient Alpha

White

3 – Spelling

25,898

0.94

0.94

0.94

Coefficient Alpha

White

3 – Word Recognition Preprimer

833

0.91

0.90

0.92

Coefficient Alpha

White

3 – Word Recognition Primer

2,083

0.75

0.74

0.77

Coefficient Alpha

White

3 – Word Recognition 1

2,799

0.88

0.87

0.88

Coefficient Alpha

Non-white

1 – Spelling

67,367

0.93

0.93

0.93

Coefficient Alpha

Non-white

1 – Word Recognition Preprimer

37,986

0.94

0.94

0.94

Coefficient Alpha

Non-white

1 – Word Recognition Primer

35,667

0.76

0.75

0.77

Coefficient Alpha

Non-white

1 – Word Recognition 1

47,168

0.93

0.93

0.93

Coefficient Alpha

Non-white

2 – Spelling

61,728

0.95

0.95

0.95

Coefficient Alpha

Non-white

2 – Word Recognition Preprimer

6,802

0.91

0.91

0.91

Coefficient Alpha

Non-white

2 – Word Recognition Primer

14,604

0.75

0.75

0.76

Coefficient Alpha

Non-white

2 – Word Recognition 1

32,628

0.93

0.93

0.93

Coefficient Alpha

Non-white

3 – Spelling

27,701

0.94

0.94

0.94

Coefficient Alpha

Non-white

3 – Word Recognition Preprimer

1,534

0.92

0.92

0.93

Coefficient Alpha

Non-white

3 – Word Recognition Primer

 

3,587

0.75

0.74

0.77

Coefficient Alpha

Non-white

3 – Word Recognition 1

4,667

0.88

0.87

0.88

Coefficient Alpha

ESL

1 – Spelling

15,876

0.92

0.92

0.92

Coefficient Alpha

ESL

1 – Word Recognition Preprimer

9,517

0.95

0.94

0.95

Coefficient Alpha

ESL

1 – Word Recognition Primer

8,476

0.76

0.75

0.77

Coefficient Alpha

ESL

1 – Word Recognition 1

10,611

0.94

0.93

0.94

Coefficient Alpha

ESL

2 – Spelling

15,878

0.94

0.94

0.94

Coefficient Alpha

ESL

2 – Word Recognition Preprimer

2,442

0.93

0.93

0.93

Coefficient Alpha

ESL

2 – Word Recognition Primer

4,617

0.76

0.75

0.77

Coefficient Alpha

ESL

2 – Word Recognition 1

8,636

0.93

0.93

0.93

Coefficient Alpha

ESL

3 – Spelling

7,357

0.94

0.93

0.94

Coefficient Alpha

ESL

3 – Word Recognition Preprimer

649

0.93

0.92

0.94

Coefficient Alpha

ESL

3 – Word Recognition Primer

1,307

0.76

0.74

0.77

Coefficient Alpha

ESL

3 – Word Recognition 1

1,675

0.89

0.88

0.90

Coefficient Alpha

Non-ESL

1 – Spelling

118,354

0.94

0.93

0.94

Coefficient Alpha

Non-ESL

1 – Word Recognition Preprimer

62,833

0.93

0.92

0.93

Coefficient Alpha

Non-ESL

1 – Word Recognition Primer

60,092

0.76

0.76

0.76

Coefficient Alpha

Non-ESL

1 – Word Recognition 1

84,132

0.92

0.915

0.92

Coefficient Alpha

Non-ESL

2 – Spelling

107,991

0.95

0.95

0.95

Coefficient Alpha

Non-ESL

2 – Word Recognition Preprimer

8,439

0.89

0.88

0.89

Coefficient Alpha

Non-ESL

2 – Word Recognition Primer

19,999

0.75

0.75

0.76

Coefficient Alpha

Non-ESL

2 – Word Recognition 1

53,558

0.93

0.93

0.93

Coefficient Alpha

Non-ESL

3 – Spelling

45,775

0.94

0.94

0.942

Coefficient Alpha

Non-ESL

3 – Word Recognition Preprimer

1,654

0.91

0.90

0.91

Coefficient Alpha

Non-ESL

3 – Word Recognition Primer

4,258

0.75

0.74

0.76

Coefficient Alpha

Non-ESL

3 – Word Recognition 1

5,667

0.87

0.87

0.88

 

Validity

Grade123
RatingFull bubbleHalf-filled bubbleFull bubble

1.Description of each criterion measure used and explanation as to why each measure is appropriate, given the type and purpose of the tool: Stanford Achievement Test and the California Achievement Test are standardized achievement tests used by school districts in the United States and in American schools abroad for assessing children from kindergarten through high school.

The Qualitative Reading Inventory (QRI) and the Developmental Reading Assessment (DRA) are both classroom-based informal reading inventories commonly used across the U.S. and in American schools around the world.

 

2.Description of the sample(s), including size and characteristics, for each validity analysis conducted:  The Stanford, the QRI, and the CAT were sub-samples from the larger state-wide sample in Grades 1-3 in Virginia.

The analyses relating Grade 2 PALS to Grade 2 MAP are based on a sample of 2,043 2nd Grade students from Colorado. 51.3% of students are male, 49.4% are nonwhite, and 22% are English Language Learners.

The analyses relating Grade 3 PALS to Grade 3 iStation are based on a sample of 23 3rd graders from Frederick County Virginia from 2016 and 2017. 70% of students are nonwhite, 52% are female, and 26% are English Language Learners.

The analyses relating Grade 3 PALS to Grade 3 MAP are based on a sample of 2,278 students from Colorado. 53% of students are nonwhite, 46% are female, and 23% are English Language Learners.

 

3.Description of the analysis procedures for each reported type of validity: Validity analyses were conducted by taking the Pearson’s correlation between PALS score and either concurrent outcome or score on a later test (predictive) or a test conducted in the same semester (concurrent). Grade 3 and Grade 2 analyses were performed in R.

 

4.Validity for the performance level score (e.g., concurrent, predictive, evidence based on response processes, evidence based on internal structure, evidence based on relations to other variables, and/or evidence based on consequences of testing), and the criterion measures.

Type of Validity

Age or Grade

Test or Criterion

n

Coefficient

95% Confidence Interval: Lower Bound

95% Confidence Interval: Upper Bound

Criterion-Related: predictive

1

Stanford-9 Total Reading

739

0.73

0.70

0.76

 

Concurrent

 2

 MAP

 2043

 0.80

 0.78

 0.81

 Criterion-Related: concurrent

 3

 iStation Grade 3 Spring: Reading

 23

 0.82

 0.62

 0.92

 Criterion-Related: concurrent

 3

 MAP Reading Grade 3 Fall

 2278

 0.78

 0.76

 0.79

 

5.Results for other forms of validity (e.g. factor analysis) not conducive to the table format: Construct: As one piece of evidence in support of the construct validity of PALS 1-3, we conduct Principal Component Analyses on the tasks that make up the PALS 1-3 Entry Level Summed Score each year, based on statewide data.  These analyses consistently yield a single factor (eigenvalue greater than 1.0) that accounts for more than three quarters of the variance in these scores.

Content: The content validity is supported by the process of item development (described in the Technical Reference, Section III, pp. 11 – 19).  The guiding principles underlying this process were that (a) tasks and items were a representative sample of tasks from other early literacy screening instruments, (b) items had a history of use in phonological awareness and early literacy research, and (c) items were aligned with Virginia’s Standards of Learning for English (Reading).  Each of these principles was further supported by the process of review by an Advisory Panel of literacy professionals from across Virginia, and by an External Review Panel, which consisted of nationally recognized experts in the field of reading, communication sciences, or psychology.

 

6.Describe the degree to which the provided data support the validity of the tool: These data suggest that PALS has a strong relationship to the Stanford-9 and QRI-II in 1st grade, the Stanford-9 in 2nd grade, and MAP and iStation reading scores in 3rd grade.

 

 

Disaggregated Validity

The following disaggregated validity data are provided for context and did not factor into the Validity rating.

Type of Validity

Subgroup

Age or Grade

Test or Criterion

n

Coefficient

95% Confidence Interval: Lower Bound

95% Confidence Interval: Upper Bound

None

 

 

 

 

 

 

 

 

Results for other forms of disaggregated validity (e.g. factor analysis) not conducive to the table format: Not Provided

 

 

If your manual cites other published validity studies, provide these citations: Huang, F. L. (2014). Using a bifactor model to assess the factor structure of the Phonological Awareness Literacy Screening for grades 1 through 3. Journal of Psychoeducational Assessment, 32, 638-650.

Sample Representativeness

Grade123
Data
  • Regional with Cross-Validation
  • Regional with Cross-Validation
  • Local with Cross-Validation
  • Primary Classification Accuracy Sample

    Criterion 1

    Grade

    1

    2

    3

    Criterion

    MAP

    MAP

    MAP

    National/Local Representation

    West Region (CO): 1375                 South Region (VA): 625

    South Region (VA): 1750                     West Region (CO): 500

    West Region (CO): 1500

     

    Date

    Fall

    Fall

    Fall

    Sample Size

    2000

    2250

    1500

    Male

    52%

    52%

    52%

    Female

    48%

    48%

    48%

    Gender Unknown

    Not Provided

    Not Provided

    Not Provided

    Free or Reduced-price Lunch Eligible

    Not Provided

    Not Provided

    Not Provided

    White, Non-Hispanic

    60%

    46%

    47%

    Black, Non-Hispanic

    Not Provided

    Not Provided

    Not Provided

    Hispanic

    Not Provided

    Not Provided

    Not Provided

    American Indian/Alaska Native

    Not Provided

    Not Provided

    Not Provided

    Other

    Not Provided

    Not Provided

    Not Provided

    Race/Ethnicity Unknown

    Not Provided

    Not Provided

    Not Provided

    Disability Classification

    Not Provided

    Not Provided

    Not Provided

    First Language

    54%

    79%

    77%

    Language Proficiency Status

    Not Provided

    Not Provided

    Not Provided

     

    Criterion 2

    Grade

    1

    2

    3

    Criterion

    STAR

    STAR

    Reading SOL

    National/Local Representation

    Midwest Region (WI): 2328

    South Region (VA): 169

    Midwest Region (WI): 2000

    South Region (VA): 17500

     

    Date

    Fall

    Fall

    Spring

    Sample Size

    2497

    2000

    17500

    Male

    50%

    49%

    52%

    Female

    50%

    51%

    49%

    Gender Unknown

    Not Provided

    Not Provided

    Not Provided

    Free or Reduced-price Lunch Eligible

    Not Provided

    Not Provided

    Not Provided

    White, Non-Hispanic

    19%

    17%

    51%

    Black, Non-Hispanic

    Not Provided

    Not Provided

    Not Provided

    Hispanic

    Not Provided

    Not Provided

    Not Provided

    American Indian/Alaska Native

    Not Provided

    Not Provided

    Not Provided

    Other

    Not Provided

    Not Provided

    Not Provided

    Race/Ethnicity Unknown

    Not Provided

    Not Provided

    Not Provided

    Disability Classification

    Not Provided

    Not Provided

    Not Provided

    First Language

    89%

    95%

    88%

    Language Proficiency Status

    Not Provided

    Not Provided

    Not Provided

     

    Cross Validation Sample

    Criterion 1, Spring

    Grade

    1

    2

    3

    Criterion

    MAP

    MAP

    MAP

    National/Local Representation

    West Region (CO): 716

    South Region (VA): 307

    South Region (VA): 1699

    West Region (CO): 1808

    West Region (CO): 1369

    Date

    Fall

    Fall

    Fall

    Sample Size

    1023

    3507

    1369

    Male

    51%

    52%

    53%

    Female

    49%

    48%

    47%

    Gender Unknown

    Not Provided

    Not Provided

    Not Provided

    Free or Reduced-price Lunch Eligible

    Not Provided

    Not Provided

    Not Provided

    White, Non-Hispanic

    60%

    48%

    49%

    Black, Non-Hispanic

    Not Provided

    Not Provided

    Not Provided

    Hispanic

    Not Provided

    Not Provided

    Not Provided

    American Indian/Alaska Native

    Not Provided

    Not Provided

    Not Provided

    Other

    Not Provided

    Not Provided

    Not Provided

    Race/Ethnicity Unknown

    Not Provided

    Not Provided

    Not Provided

    Disability Classification

    Not Provided

    Not Provided

    Not Provided

    First Language

    54%

    73%

    77%

    Language Proficiency Status

    Not Provided

    Not Provided

    Not Provided

     

    Criterion 2, Spring

    Grade

    1

    2

    3

    Criterion

    STAR

    STAR

    Reading SOL

    National/Local Representation

    Midwest Region (WI): 4063

     

    Midwest Region (WI): 4262

    South Region (VA): 16120

    Date

    Spring

    Spring

    Spring

    Sample Size

    4063

    4262

    16120

    Male

    51%

    51%

    52%

    Female

    49%

    49%

    48%

    Gender Unknown

    Not Provided

    Not Provided

    Not Provided

    Free or Reduced-price Lunch Eligible

    Not Provided

    Not Provided

    Not Provided

    White, Non-Hispanic

    16%

    15%

    46%

    Black, Non-Hispanic

    Not Provided

    Not Provided

    Not Provided

    Hispanic

    Not Provided

    Not Provided

    Not Provided

    American Indian/Alaska Native

    Not Provided

    Not Provided

    Not Provided

    Other

    Not Provided

    Not Provided

    Not Provided

    Race/Ethnicity Unknown

    Not Provided

    Not Provided

    Not Provided

    Disability Classification

    Not Provided

    Not Provided

    Not Provided

    First Language

    95%

    94%

    86%

    Language Proficiency Status

    Not Provided

    Not Provided

    Not Provided

    Primary Classification Accuracy Sample

    Criterion 1

    Grade

    1

    2

    3

    Criterion

    MAP

    MAP

    MAP

    National/Local Representation

    West Region (CO): 1375                 South Region (VA): 625

    South Region (VA): 1750                     West Region (CO): 500

    West Region (CO): 1500

     

    Date

    Fall

    Fall

    Fall

    Sample Size

    2000

    2250

    1500

    Male

    52%

    52%

    52%

    Female

    48%

    48%

    48%

    Gender Unknown

    Not Provided

    Not Provided

    Not Provided

    Free or Reduced-price Lunch Eligible

    Not Provided

    Not Provided

    Not Provided

    White, Non-Hispanic

    60%

    46%

    47%

    Black, Non-Hispanic

    Not Provided

    Not Provided

    Not Provided

    Hispanic

    Not Provided

    Not Provided

    Not Provided

    American Indian/Alaska Native

    Not Provided

    Not Provided

    Not Provided

    Other

    Not Provided

    Not Provided

    Not Provided

    Race/Ethnicity Unknown

    Not Provided

    Not Provided

    Not Provided

    Disability Classification

    Not Provided

    Not Provided

    Not Provided

    First Language

    54%

    79%

    77%

    Language Proficiency Status

    Not Provided

    Not Provided

    Not Provided

     

    Criterion 2

    Grade

    1

    2

    3

    Criterion

    STAR

    STAR

    Reading SOL

    National/Local Representation

    Midwest Region (WI): 2328

    South Region (VA): 169

    Midwest Region (WI): 2000

    South Region (VA): 17500

     

    Date

    Fall

    Fall

    Spring

    Sample Size

    2497

    2000

    17500

    Male

    50%

    49%

    52%

    Female

    50%

    51%

    49%

    Gender Unknown

    Not Provided

    Not Provided

    Not Provided

    Free or Reduced-price Lunch Eligible

    Not Provided

    Not Provided

    Not Provided

    White, Non-Hispanic

    19%

    17%

    51%

    Black, Non-Hispanic

    Not Provided

    Not Provided

    Not Provided

    Hispanic

    Not Provided

    Not Provided

    Not Provided

    American Indian/Alaska Native

    Not Provided

    Not Provided

    Not Provided

    Other

    Not Provided

    Not Provided

    Not Provided

    Race/Ethnicity Unknown

    Not Provided

    Not Provided

    Not Provided

    Disability Classification

    Not Provided

    Not Provided

    Not Provided

    First Language

    89%

    95%

    88%

    Language Proficiency Status

    Not Provided

    Not Provided

    Not Provided

     

    Cross Validation Sample

    Criterion 1, Spring

    Grade

    1

    2

    3

    Criterion

    MAP

    MAP

    MAP

    National/Local Representation

    West Region (CO): 716

    South Region (VA): 307

    South Region (VA): 1699

    West Region (CO): 1808

    West Region (CO): 1369

    Date

    Fall

    Fall

    Fall

    Sample Size

    1023

    3507

    1369

    Male

    51%

    52%

    53%

    Female

    49%

    48%

    47%

    Gender Unknown

    Not Provided

    Not Provided

    Not Provided

    Free or Reduced-price Lunch Eligible

    Not Provided

    Not Provided

    Not Provided

    White, Non-Hispanic

    60%

    48%

    49%

    Black, Non-Hispanic

    Not Provided

    Not Provided

    Not Provided

    Hispanic

    Not Provided

    Not Provided

    Not Provided

    American Indian/Alaska Native

    Not Provided

    Not Provided

    Not Provided

    Other

    Not Provided

    Not Provided

    Not Provided

    Race/Ethnicity Unknown

    Not Provided

    Not Provided

    Not Provided

    Disability Classification

    Not Provided

    Not Provided

    Not Provided

    First Language

    54%

    73%

    77%

    Language Proficiency Status

    Not Provided

    Not Provided

    Not Provided

     

    Criterion 2, Spring

    Grade

    1

    2

    3

    Criterion

    STAR

    STAR

    Reading SOL

    National/Local Representation

    Midwest Region (WI): 4063

     

    Midwest Region (WI): 4262

    South Region (VA): 16120

    Date

    Spring

    Spring

    Spring

    Sample Size

    4063

    4262

    16120

    Male

    51%

    51%

    52%

    Female

    49%

    49%

    48%

    Gender Unknown

    Not Provided

    Not Provided

    Not Provided

    Free or Reduced-price Lunch Eligible

    Not Provided

    Not Provided

    Not Provided

    White, Non-Hispanic

    16%

    15%

    46%

    Black, Non-Hispanic

    Not Provided

    Not Provided

    Not Provided

    Hispanic

    Not Provided

    Not Provided

    Not Provided

    American Indian/Alaska Native

    Not Provided

    Not Provided

    Not Provided

    Other

    Not Provided

    Not Provided

    Not Provided

    Race/Ethnicity Unknown

    Not Provided

    Not Provided

    Not Provided

    Disability Classification

    Not Provided

    Not Provided

    Not Provided

    First Language

    95%

    94%

    86%

    Language Proficiency Status

    Not Provided

    Not Provided

    Not Provided

     

    Bias Analysis Conducted

    Grade123
    RatingYesYesYes
    1. Description of the method used to determine the presence or absence of bias: We used the Mantel-Haenszel procedure to test for differential item functioning (DIF) for the Entry Level Screening Tasks. We combined this hypothesis testing procedures with the common-odds ratio effect size to determine the practical significance of DIF. We followed the ETS classification criteria of A, B, C to indicate no, moderate, and large amount of DIF.

     

    1. Description of the subgroups for which bias analyses were conducted: Male/Female, White/Non-White, ESL/Non-ESL

     

    1. Description of the results of the bias analyses conducted, including data and interpretative statements: The majority of the items for each sub-task of the summed score received an ETS classification of A, indicating no bias in spelling or word recognition for most groups. Three percent of the items in spelling showed moderate bias for White/Non-White. Fifteen percent of the items of the Word Recognition Primer list showed moderate bias for ESL/Non-ESL and 20% of the items on the Word Recognition First Grade list showed moderate bias for Male/Female. Regardless of group, however, these words, which represent frequently occurring words at these grade levels, must be recognized automatically in order to be a fluent reader.

     

     

     

    Percent Classified

    Task

    Groups

    A

    B

    C

    Spelling

    Male/Female

    100

    0

    0

     

    White/Nonwhite

    97

    3

    0

     

    NonESL/ESL

    89

    11

    0

    Word Recognition

    Male/Female

    75

    20

    5

     

    White/Nonwhite

    100

    0

    0

     

    NonESL/ESL

    100

    0

    0

    Word Recognition P

    Male/Female

    100

    0

    0

     

    White/Nonwhite

    100

    0

    0

     

    NonESL/ESL

    85

    15

    0

    Word Recognition PP

    Male/Female

    100

    0

    0

     

    White/Nonwhite

    100

    0

    0

     

    NonESL/ESL

    100

    0

    0

     

    Administration Format

    Grade123
    Data
  • Individual
  • Group
  • Individual
  • Group
  • Individual
  • Group
  • Administration & Scoring Time

    Grade123
    Data
  • 20-45 minutes
  • 25-45 minutes
  • 25-45 minutes
  • Scoring Format

    Grade123
    Data
  • Manual
  • Automatic
  • Manual
  • Automatic
  • Manual
  • Automatic
  • Types of Decision Rules

    Grade123
    Data
  • Administration Rules
  • Administration Rules
  • Administration Rules
  • Evidence Available for Multiple Decision Rules

    Grade123
    Data
  • No
  • No
  • No