Observation Survey of Early Literacy Achievement (OSELA)
Reading

Descriptive Information

Please provide a description of your tool:

An Observation Survey of Early Literacy Achievement comprises six systematic, standard observation tasks developed in research studies. The tasks yield a composite and comprehensive assessment of the literacy performance of young learners. All tasks have the qualities of sound assessment instruments with reliability and validity and discrimination indices established in research. Children are assessed individually by a specially trained teacher. The Observation Survey tasks are designed to allow children to work with the complexities of written language and allow the examiner to be confident of characteristics of good measurement instruments: a standard task; a standard way of administering the task; ways of knowing when we can rely on our observation and make valid comparisons; and a task that relates to real-world tasks (content validity). Because young children begin literacy learning and instruction in early grades with individually unique knowledge and confusions, assessment of beginning reading and writing should inform the examiner of literacy behaviors along several dimensions of learning. Therefore, early assessments must be wide-ranging, with tasks to observe • Concepts about print (how print encodes information) • The reading of continuous text • Letter knowledge • Reading vocabulary • Writing vocabulary • Phonemic awareness and sound-symbol relationships It is important to know how learning is proceeding in each of these areas and to identify problems early. No one task should be used in isolation because it may assess only one aspect of early literacy behavior. Therefore, all six tasks of the Observation Survey are considered as a composite when screening and selecting children for Reading Recovery (a literacy intervention for first graders who are struggling). Because of the comprehensive nature of the assessment of each individual, results are used for screening, to guide teaching, and to monitor progress

The tool is intended for use with the following grade(s).

Preschool / Pre - kindergarten
Kindergarten
First grade
Second grade
Third grade
Fourth grade
Fifth grade
Sixth grade
Seventh grade
Eighth grade
Ninth grade
Tenth grade
Eleventh grade
Twelfth grade

The tool is intended for use with the following age(s).

0-4 years old
5 years old
6 years old
7 years old
8 years old
9 years old
10 years old
11 years old
12 years old
13 years old
14 years old
15 years old
16 years old
17 years old
18 years old

The tool is intended for use with the following student populations.

Students in general education
Students with disabilities
English language learners

ACADEMIC ONLY: What skills does the tool screen?

Reading

Phonological processing:

RAN
Memory
Awareness
Letter sound correspondence
Phonics
Structural analysis

Word ID

Accuracy
Speed

Nonword

Accuracy
Speed

Spelling

Accuracy
Speed

Passage

Accuracy
Speed

Reading comprehension:

Multiple choice questions
Cloze
Constructed Response
Retell
Maze
Sentence verification
Other (please describe):

During the reading of connected text, teachers record all errors and self-corrections. Teachers then analyze each error and self-correction to determine if the responses reflected attention to meaning

Listening comprehension:

Multiple choice questions
Cloze
Constructed Response
Retell
Maze
Sentence verification
Vocabulary
Expressive
Receptive

Mathematics

Global Indicator of Math Competence

Accuracy
Speed
Multiple Choice
Constructed Response

Early Numeracy

Accuracy
Speed
Multiple Choice
Constructed Response

Mathematics Concepts

Accuracy
Speed
Multiple Choice
Constructed Response

Mathematics Computation

Accuracy
Speed
Multiple Choice
Constructed Response

Mathematic Application

Accuracy
Speed
Multiple Choice
Constructed Response

Fractions/Decimals

Accuracy
Speed
Multiple Choice
Constructed Response

Algebra

Accuracy
Speed
Multiple Choice
Constructed Response

Geometry

Accuracy
Speed
Multiple Choice
Constructed Response

Other (please describe):

Please describe specific domain, skills or subtests:

List specific skills or subtests: Concepts About Print — including book orientation, concept that print carries the message, directional rules, word-by-word pointing, concepts of first and last, inverted picture and inverted print, line sequence, left page before right, concept of word, concept of letter, word sequence, letter order, meaning of punctuation, capital and lowercase letters

BEHAVIOR ONLY: Which category of behaviors does your tool target?

Internalizing
Externalizing
Internalizing and Externalizing

BEHAVIOR ONLY: Please identify which broad domain(s)/construct(s) are measured by your tool and define each sub-domain or sub-construct.

Acquisition and Cost Information

Where to obtain:

Email Address

Address

Heinemann, P.O. Box 6926, Portsmouth, NH, 03802-6926

Phone Number

800-225-5800

Website

www.heinemann.com

Initial cost for implementing program:

Cost

Unit of cost

Replacement cost per unit for subsequent use:

Cost

Unit of cost

Duration of license

Additional cost information:

Describe basic pricing plan and structure of the tool. Provide information on what is included in the published tool, as well as what is not included but required for implementation.

Costs associated with An Observation Survey of Early Literacy Achievement are per teacher. No additional costs are required per student. Training costs are part of Reading Recovery implementation budgets. Reading Recovery sites employ a teacher leader who is trained in the administration, scoring, and interpretation of the Observation Survey. That teacher leader is responsible for training teachers to use the tool for screening, selecting children for intervention, for monitoring progress, and for making Reading Recovery exit decisions. Reading Recovery teachers are also trained to use outcomes of the Observation Survey to inform their teaching of each individual child. The following information is included within the book: 1. Theoretical information about observing and recording early literacy behaviors 2. Theoretical information about the processes of reading and writing 3. Information about early detection of literacy difficulties and early intervention 4. A chapter on each task of the Observation Survey that provides (a) rationales for the task; (b) procedures for administering, scoring, and interpreting the task; (c) student protocols; and (d) scoring sheets/ record sheets. The tasks within the Observation Survey book include • Letter Identification • Concepts About Print • Ohio Word Test • Writing Vocabulary • Hearing and Recording Sounds in Words • Running Record of Text Reading 5. A chapter on how to summarize, interpret, and use the results of all six tasks 6. Forms to show change over time (progress monitoring) 7. Appendices which include • New Zealand norms (including stanines and percentile ranks, inter-correlations, and validity and reliability reports) • Administration and Score Sheets • U. S. Norms (including stanines and percentile ranks) and correlations

Provide information about special accommodations for students with disabilities.

Because the Observation Survey measures authentic literacy knowledge and is administered individually, accommodations are made based on teacher observations of each child

Administration

BEHAVIOR ONLY: What type of administrator is your tool designed for?

General education teacher
Special education teacher
Parent
Child
External observer
Other

If other, please specify:

What is the administration setting?

Direct observation
Rating scale
Checklist
Performance measure
Questionnaire
Direct: Computerized
One-to-one
Other

If other, please specify:

Does the tool require technology?

If yes, what technology is required to implement your tool? (Select all that apply)

Computer or tablet
Internet connection
Other technology (please specify)

If your program requires additional technology not listed above, please describe the required technology and the extent to which it is combined with teacher small-group instruction/intervention:

What is the administration context?

Individual
Small group   If small group, n=
Large group   If large group, n=
Computer-administered
Other

If other, please specify:

What is the administration time?

Time in minutes

30

per (student/group/other unit)

student

Additional scoring time:

Time in minutes

15

per (student/group/other unit)

student

ACADEMIC ONLY: What are the discontinue rules?

No discontinue rules provided
Basals
Ceilings
Other

If other, please specify:

Writing Vocabulary has a 10-minute limit; Text Reading ends after 2 levels below 90% accuracy

Are norms available?

Yes

Are benchmarks available?

No

If yes, how many benchmarks per year?

If yes, for which months are benchmarks available?

BEHAVIOR ONLY: Can students be rated concurrently by one administrator?

If yes, how many students can be rated concurrently?

Training & Scoring

Training

Is training for the administrator required?

Yes

Describe the time required for administrator training, if applicable:

4-8 hours of training

Please describe the minimum qualifications an administrator must possess.

Professional

No minimum qualifications

Are training manuals and materials available?

Yes

Are training manuals/materials field-tested?

Yes

Are training manuals/materials included in cost of tools?

Yes

If No, please describe training costs:

Can users obtain ongoing professional and technical support?

Yes

If Yes, please describe how users can obtain support:

trained leaders

Scoring

How are scores calculated?

Manually (by hand)
Automatically (computer-scored)
Other

If other, please specify:

Do you provide basis for calculating performance level scores?

What is the basis for calculating performance level and percentile scores?

Age norms
Grade norms
Classwide norms
Schoolwide norms
Stanines
Normal curve equivalents

What types of performance level scores are available?

Raw score
Standard score
Percentile score
Grade equivalents
IRT-based score
Age equivalents
Stanines
Normal curve equivalents
Developmental benchmarks
Developmental cut points
Equated
Probability
Lexile score
Error analysis
Composite scores
Subscale/subtest scores
Other

If other, please specify:

Does your tool include decision rules?

If yes, please describe.

Can you provide evidence in support of multiple decision rules?

No

If yes, please describe.

Please describe the scoring structure. Provide relevant details such as the scoring format, the number of items overall, the number of items per subscale, what the cluster/composite score comprises, and how raw scores are calculated.

Raw scores are converted to stanines and all six tasks are considered as a composite to identify children who need a literacy intervention. For the composite measure submitted for approval as a screening tool, and for which all of the psychometric information submitted under Section III was based, the raw scores on the six tasks for each student are summed to compute a total raw score. Each student’s total raw score is converted to an IRT-based scale score that can range from 0 to 800 points. The IRT-based scale scores were derived by treating each task as a polythomously-scored item, and the six tasks were scaled based on a partial-credit Rasch model using the Winsteps computer program. Separate random samples of student task responses from either Fall 2009, mid-year (late December 2009 to early February 2010), or Spring 2010 were used as data to create the unidimensional scale that reflects literacy growth.

Describe the tool’s approach to screening, samples (if applicable), and/or test format, including steps taken to ensure that it is appropriate for use with culturally and linguistically diverse populations and students with disabilities.

The six tasks of the Observation Survey are used to screen first graders in general education classrooms and to identify those who are experiencing difficulty with literacy learning. Based on the scores on all six tasks, the lowest achievers are selected for the Reading Recovery intervention (a 12-20 week of intensive one-to-one teaching by a specially-trained teacher). [See information in the next box for a brief description and maximum scores for each task.] The Observation Survey is administered individually to children perceived by their first-grade classroom teachers as the lowest literacy achievers in their classrooms and serves as the screening process to identify the lowest achievers. The tester is a specially trained teacher. For each task, the tester records the raw score and the stanine. In consultation with the school team and the Reading Recovery teacher leader, the children with the lowest stanine scores across the Observation Survey are identified, without regard to cultural or linguistic diversity or disabilities. The intervention is intended to serve the lowest-performing first graders in regular classrooms to raise their literacy achievement so they can benefit from classroom instruction. For the few children who do not reach average performance as a result of the intervention, Reading Recovery serves as a prereferral intervention, providing information for future educational actions. For additional information about screening and selecting children for Reading Recovery intervention services, see Chapter 5 in Changing Futures: The Influence of Reading Recovery in the United States, or an article entitled “Selection of Children for Reading Recovery: Challenges and Responses” in the Fall 2005 issue of The Journal of Reading Recovery.

Classification Accuracy & Cross-Validation Summary

Grade	Grade 1
Classification Accuracy Fall
Classification Accuracy Winter
Classification Accuracy Spring

Slosson Oral Reading Test-Revised

Classification Accuracy

Select time of year

Fall

Winter

Spring

Describe the criterion (outcome) measure(s) including the degree to which it/they is/are independent from the screening measure.

The Slosson Oral Reading Test-Revised is a series of graded word lists. It is a standardized measure that is completely independent of the OS screening measure

Do the classification accuracy analyses examine concurrent and/or predictive classification?

Concurrent
Predictive

Describe when screening and criterion measures were administered and provide a justification for why the method(s) you chose (concurrent and/or predictive) is/are appropriate for your tool.

Describe how the classification analyses were performed and cut-points determined. Describe how the cut points align with students at-risk. Please indicate which groups were contrasted in your analyses (e.g., low risk students versus high risk students, low risk students versus moderate risk students).

We used the pROC package as described in Appendix A of the FAQ for the classification analyses. The Slosson Oral Reading Test-Revised manual does not provide end of first grade norms so we used a random sample of 8200 students to determine the cut point for the bottom 15%. We used this cut to identify children in need of intensive intervention at the end of first grade. None of these students had participated in the Reading Recovery intervention during first grade. The cut point on the Observation Survey Total Score in the fall was based on the 80% sensitivity level in the ROC analysis.

Were the children in the study/studies involved in an intervention in addition to typical classroom instruction between the screening measure and outcome assessment?

No

If yes, please describe the intervention, what children received the intervention, and how they were chosen.

Cross-Validation

Has a cross-validation study been conducted?

Yes

If yes,

Select time of year.

Fall

Winter

Spring

Describe the criterion (outcome) measure(s) including the degree to which it/they is/are independent from the screening measure.

The Slosson Oral Reading Test-Revised is a series of graded word lists. It is a standardized measure that is completely independent of the OS screening measure.

Do the cross-validation analyses examine concurrent and/or predictive classification?

Concurrent
Predictive

Describe when screening and criterion measures were administered and provide a justification for why the method(s) you chose (concurrent and/or predictive) is/are appropriate for your tool.

Describe how the cross-validation analyses were performed and cut-points determined. Describe how the cut points align with students at-risk. Please indicate which groups were contrasted in your analyses (e.g., low risk students versus high risk students, low risk students versus moderate risk students).

The cross-validation analysis used random sample students from the previous year, 2015-16. The cut point with 80% sensitivity from the 2016-17 sample was used to establish false positive and false negatives in the cross-validation sample.

Were the children in the study/studies involved in an intervention in addition to typical classroom instruction between the screening measure and outcome assessment?

No

If yes, please describe the intervention, what children received the intervention, and how they were chosen.

---

Observation Survey of Early Literacy Achievement: Text Reading Level

Classification Accuracy

Select time of year

Fall

Winter

Spring

Describe the criterion (outcome) measure(s) including the degree to which it/they is/are independent from the screening measure.

Observation Survey of Early Literacy Achievement, Text Reading Level is a standard set of passages that increase in difficult. The highest text level the child is able to read with 90% accuracy or higher is the child’s Text Reading Level (See attached article by D’Agostino et al., 2016.) Text Reading Level is one component of the fall screening measure, but no child identified for intervention in the fall would read passages above Text Level 6, so the Text Level 18 passage is completely independent of the screening measure.

Do the classification accuracy analyses examine concurrent and/or predictive classification?

Concurrent
Predictive

Describe when screening and criterion measures were administered and provide a justification for why the method(s) you chose (concurrent and/or predictive) is/are appropriate for your tool.

Describe how the classification analyses were performed and cut-points determined. Describe how the cut points align with students at-risk. Please indicate which groups were contrasted in your analyses (e.g., low risk students versus high risk students, low risk students versus moderate risk students).

The cross-validation analysis used random sample students from the previous year, 2015-16. The cut point with 80% sensitivity from the 2016-17 sample was used to establish false positive and false negatives in the cross-validation sample

Were the children in the study/studies involved in an intervention in addition to typical classroom instruction between the screening measure and outcome assessment?

No

If yes, please describe the intervention, what children received the intervention, and how they were chosen.

Cross-Validation

Has a cross-validation study been conducted?

Yes

If yes,

Select time of year.

Fall

Winter

Spring

Describe the criterion (outcome) measure(s) including the degree to which it/they is/are independent from the screening measure.

Observation Survey of Early Literacy Achievement, Text Reading Level is a standard set of passages that increase in difficult. The highest text level the child is able to read with 90% accuracy or higher is the child’s Text Reading Level (See attached article by D’Agostino et al., 2016.) Text Reading Level is one component of the fall screening measure, but no child identified for intervention in the fall would read passages above Text Level 6, so the Text Level 18 passage is completely independent of the screening measure.

Do the cross-validation analyses examine concurrent and/or predictive classification?

Concurrent
Predictive

Describe when screening and criterion measures were administered and provide a justification for why the method(s) you chose (concurrent and/or predictive) is/are appropriate for your tool.

Describe how the cross-validation analyses were performed and cut-points determined. Describe how the cut points align with students at-risk. Please indicate which groups were contrasted in your analyses (e.g., low risk students versus high risk students, low risk students versus moderate risk students).

The cross-validation analysis used random sample students from the previous year, 2015-16. The cut point with 80% sensitivity from the 2016-17 sample was used to establish false positive and false negatives in the cross-validation sample.

Were the children in the study/studies involved in an intervention in addition to typical classroom instruction between the screening measure and outcome assessment?

No

If yes, please describe the intervention, what children received the intervention, and how they were chosen.

---

Classification Accuracy - Fall

Evidence	Grade 1
Criterion measure	Slosson Oral Reading Test-Revised
Cut Points - Percentile rank on criterion measure
Cut Points - Performance score on criterion measure	<1.40
Cut Points - Corresponding performance score (numeric) on screener measure	421.00
Classification Data - True Positive (a)	236
Classification Data - False Positive (b)	452
Classification Data - False Negative (c)	56
Classification Data - True Negative (d)	1463
Area Under the Curve (AUC)	0.87
AUC Estimate’s 95% Confidence Interval: Lower Bound	0.84
AUC Estimate’s 95% Confidence Interval: Upper Bound	0.89

Statistics	Grade 1
Base Rate	0.13
Overall Classification Rate	0.77
Sensitivity	0.81
Specificity	0.76
False Positive Rate	0.24
False Negative Rate	0.19
Positive Predictive Power	0.34
Negative Predictive Power	0.96

Sample	Grade 1
Date	2016-17
Sample Size	2207
Geographic Representation
Male
Female
Other
Gender Unknown
White, Non-Hispanic
Black, Non-Hispanic
Hispanic
Asian/Pacific Islander
American Indian/Alaska Native
Other
Race / Ethnicity Unknown
Low SES
IEP or diagnosed disability
English Language Learner

Classification Accuracy - Winter

Evidence	Grade 1
Criterion measure	Slosson Oral Reading Test-Revised
Cut Points - Percentile rank on criterion measure
Cut Points - Performance score on criterion measure	<1.40
Cut Points - Corresponding performance score (numeric) on screener measure	498.00
Classification Data - True Positive (a)	236
Classification Data - False Positive (b)	355
Classification Data - False Negative (c)	48
Classification Data - True Negative (d)	1515
Area Under the Curve (AUC)	0.90
AUC Estimate’s 95% Confidence Interval: Lower Bound	0.88
AUC Estimate’s 95% Confidence Interval: Upper Bound	0.92

Statistics	Grade 1
Base Rate	0.13
Overall Classification Rate	0.81
Sensitivity	0.83
Specificity	0.81
False Positive Rate	0.19
False Negative Rate	0.17
Positive Predictive Power	0.40
Negative Predictive Power	0.97

Sample	Grade 1
Date	2016-2017
Sample Size	2154
Geographic Representation
Male
Female
Other
Gender Unknown
White, Non-Hispanic
Black, Non-Hispanic
Hispanic
Asian/Pacific Islander
American Indian/Alaska Native
Other
Race / Ethnicity Unknown
Low SES
IEP or diagnosed disability
English Language Learner

Cross-Validation - Fall

Evidence	Grade 1
Criterion measure	Slosson Oral Reading Test-Revised
Cut Points - Percentile rank on criterion measure
Cut Points - Performance score on criterion measure	<1.40
Cut Points - Corresponding performance score (numeric) on screener measure	421.00
Classification Data - True Positive (a)	252
Classification Data - False Positive (b)	515
Classification Data - False Negative (c)	73
Classification Data - True Negative (d)	1521
Area Under the Curve (AUC)	0.84
AUC Estimate’s 95% Confidence Interval: Lower Bound	0.82
AUC Estimate’s 95% Confidence Interval: Upper Bound	0.87

Statistics	Grade 1
Base Rate	0.14
Overall Classification Rate	0.75
Sensitivity	0.78
Specificity	0.75
False Positive Rate	0.25
False Negative Rate	0.22
Positive Predictive Power	0.33
Negative Predictive Power	0.95

Sample	Grade 1
Date	2015-2016
Sample Size	2361
Geographic Representation
Male
Female
Other
Gender Unknown
White, Non-Hispanic
Black, Non-Hispanic
Hispanic
Asian/Pacific Islander
American Indian/Alaska Native
Other
Race / Ethnicity Unknown
Low SES
IEP or diagnosed disability
English Language Learner

Cross-Validation - Winter

Evidence	Grade 1
Criterion measure	Slosson Oral Reading Test-Revised
Cut Points - Percentile rank on criterion measure
Cut Points - Performance score on criterion measure	<1.40
Cut Points - Corresponding performance score (numeric) on screener measure	498.00
Classification Data - True Positive (a)	255
Classification Data - False Positive (b)	376
Classification Data - False Negative (c)	62
Classification Data - True Negative (d)	1626
Area Under the Curve (AUC)	0.90
AUC Estimate’s 95% Confidence Interval: Lower Bound	0.88
AUC Estimate’s 95% Confidence Interval: Upper Bound	0.91

Statistics	Grade 1
Base Rate	0.14
Overall Classification Rate	0.81
Sensitivity	0.80
Specificity	0.81
False Positive Rate	0.19
False Negative Rate	0.20
Positive Predictive Power	0.40
Negative Predictive Power	0.96

Sample	Grade 1
Date	2015-16
Sample Size	2319
Geographic Representation
Male
Female
Other
Gender Unknown
White, Non-Hispanic
Black, Non-Hispanic
Hispanic
Asian/Pacific Islander
American Indian/Alaska Native
Other
Race / Ethnicity Unknown
Low SES
IEP or diagnosed disability
English Language Learner

Reliability

Grade	Grade 1
Rating	d

*Offer a justification for each type of reliability reported, given the type and purpose of the tool.

*Describe the sample(s), including size and characteristics, for each reliability analysis conducted.

*Describe the analysis procedures for each reported type of reliability.

*In the table(s) below, report the results of the reliability analyses described above (e.g., internal consistency or inter-rater reliability coefficients).

Type of	Subgroup	Informant	Age / Grade	Test or Criterion	n	Median Coefficient	95% Confidence Interval Lower Bound	95% Confidence Interval Upper Bound

Results from other forms of reliability analysis not compatible with above table format:

Manual cites other published reliability studies:

Provide citations for additional published studies.

Do you have reliability data that are disaggregated by gender, race/ethnicity, or other subgroups (e.g., English language learners, students with disabilities)?

Yes

If yes, fill in data for each subgroup with disaggregated reliability data.

Type of	Subgroup	Informant	Age / Grade	Test or Criterion	n	Median Coefficient	95% Confidence Interval Lower Bound	95% Confidence Interval Upper Bound

Results from other forms of reliability analysis not compatible with above table format:

Manual cites other published reliability studies:

Provide citations for additional published studies.

Validity

Grade	Grade 1
Rating

*Describe each criterion measure used and explain why each measure is appropriate, given the type and purpose of the tool.

*Describe the sample(s), including size and characteristics, for each validity analysis conducted.

*Describe the analysis procedures for each reported type of validity.

*In the table below, report the results of the validity analyses described above (e.g., concurrent or predictive validity, evidence based on response processes, evidence based on internal structure, evidence based on relations to other variables, and/or evidence based on consequences of testing), and the criterion measures.

Type of	Subgroup	Informant	Age / Grade	Test or Criterion	n	Median Coefficient	95% Confidence Interval Lower Bound	95% Confidence Interval Upper Bound

Results from other forms of validity analysis not compatible with above table format:

CONTENT VALIDITY The content of all tasks of the Observation Survey aligns with national reading standards and is well-documented in research. Developed for use in early-grade classrooms, Observation Survey tasks each “assess an area of literacy knowledge which provides an essential foundation for progress in reading and writing. The content of the tasks represents what is actually taught in the classroom” (Clay, 2002, 2006, p. 159). Denton et al. (2006) say that the wide implementation of the instrument suggests there is agreement about the content validity of the Observation Survey. All of the tasks in the Observation Survey represent real-world tasks and literacy instructional expectations in earlygrade classrooms. Support for content validity of each task is summarized below. Letter Identification Concept Assessed: Letter Knowledge Content Validity: It is common practice in the early grades for teachers to find out how many letters a child knows and if the child can visually distinguish letters one from another. In this task, all lowercase and uppercase letters (plus print forms of ‘a’ and ‘g’) are assessed. Children can respond with the letter name, a sound the letter makes, or a word beginning with the letter. Because this is a closed knowledge set, the task is most valid during the time a child is acquiring letter knowledge. The content represents what is actually taught in classrooms. Ohio Word Test Concept Assessed: Word Knowledge (Reading Vocabulary) Content Validity: Word knowledge correlates strongly with text reading performance. And teachers of early readers seek ways to document the number of words a child knows and how this developing knowledge changes across time. The content of this task represents typical classroom expectations. This task is structured to sample words that the children have had some opportunities to learn — words that occur frequently in their texts in school. Word lists (lists of 20 words) are drawn from the Dolch list of highfrequency words. Three equivalent lists are available. Because these lists include very high-frequency words, the task is most valid over the period when a child is acquiring an initial reading vocabulary. In addition to information about a child’s knowledge of words in isolation and how this knowledge develops over time, the teacher can learn how a child works with words through attempts and self-corrections. Concepts About Print Concept Addressed: Print Knowledge (what children know about the way spoken language is represented in print) Content Validity: Conventions used for printed language must be learned so the child can attend to the essential visual information on the page. The task assesses a child’s current knowledge of 24 print concepts. Content validity is supported through the use of a specially designed book that allows the child to demonstrate knowledge of print concepts in an authentic setting. Writing Vocabulary Concept Addressed: Writing Vocabulary (a child’s personal resource of known words) Content Validity: In this task, the child writes words he knows for a period of 10 minutes. The teacher may prompt the child in various ways to think of other known words. The score represents the number of words written independently and also serves as a screen on the child’s visual attention to print, sound sequence, motor control, and useful approximations. This task represents classroom instructional expectations. Hearing and Recording Sounds in Words Concept Addressed: Phonemic Awareness and Letter/Sound Relationships Content Validity: The importance of phonemic awareness and representing phonemes with letters or clusters of letters is well documented in the research literature. Teachers can use the score on this dictation task (with a sampling of 37 phonemes) as an indicator of a child’s developing knowledge in this area. This task measures classroom instructional practices. The teacher dictates a passage (five forms available) and asks the child to say the words slowly and write letters to represent the sounds. Each phoneme recorded in a way that is acceptable in English is counted. Because this is a closed knowledge set, the task is most valid over the period when a child is acquiring this knowledge and before the words in the passage become part of a child’s known writing vocabulary. Running Records of Text Reading Concept Addressed: Instructional Reading Level for Reading Continuous Text (also child’s behaviors while reading real books) Content Validity: The score on this task represents the highest text level (from texts representing a gradient of difficulty) that a child reads at 90% accuracy or higher. The task is an authentic assessment of the reading of continuous text. The testing packet used for Reading Recovery in the United States has been shown to be a stable measure of reading performance that represents escalating gradients of difficulty. Evidence from research and classroom practice confirms that text difficulty relates to a reader’s developing competencies. For learning to occur, the difficulty level of reading materials should present challenges from which the child can learn—texts that are not too hard or too easy. As the child reads the texts, the teacher uses established conventions to record behaviors in order to analyze the child’s reading behaviors.

Manual cites other published reliability studies:

Provide citations for additional published studies.

Describe the degree to which the provided data support the validity of the tool.

Do you have validity data that are disaggregated by gender, race/ethnicity, or other subgroups (e.g., English language learners, students with disabilities)?

If yes, fill in data for each subgroup with disaggregated validity data.

Type of	Subgroup	Informant	Age / Grade	Test or Criterion	n	Median Coefficient	95% Confidence Interval Lower Bound	95% Confidence Interval Upper Bound

Results from other forms of validity analysis not compatible with above table format:

Manual cites other published reliability studies:

Provide citations for additional published studies.

Bias Analysis

Grade	Grade 1
Rating	No

Have you conducted additional analyses related to the extent to which your tool is or is not biased against subgroups (e.g., race/ethnicity, gender, socioeconomic status, students with disabilities, English language learners)? Examples might include Differential Item Functioning (DIF) or invariance testing in multiple-group confirmatory factor models.

No

If yes,

a. Describe the method used to determine the presence or absence of bias:

b. Describe the subgroups for which bias analyses were conducted:

c. Describe the results of the bias analyses conducted, including data and interpretative statements. Include magnitude of effect (if available) if bias has been identified.