A simplified description of the methods used to create the SEDA 2024 achievement data shown in the Education Recovery Explorer is described on this page. For more detail, please refer to the SEDA 2024 Technical Documentation available on the Get the Data page.
What is SEDA 2024?
SEDA 2024 is a special release of the Stanford Education Data Archive designed to provide insight into how school district average achievement changed during and after the COVID-19 pandemic.
Source Data
The state proficiency data used to construct the SEDA 2024 test score estimates come from two sources. We use 2009-2019 data from the U.S. Department of Education’s EDFacts database, which is the same source data used to construct SEDA 5.0 (shown in the Educational Opportunity Explorer). While 2022 data are available in EDFacts, the data are not reported in sufficient detail to be used in our estimation. Instead, we use 2022 to 2024 data that was publicly released by states and reported in the zelma database. Not all state-released data is sufficiently complete for our use, therefore we only have district data for a subset of U.S. states in the current data.
We also draw on the National Assessment of Educational Progress (NAEP) 2009-2024 administrations in 4th and 8th grade to link the district estimates to a scale that is comparable among states and over grades and years.
Definition of a School District
In SEDA 2024, we report estimates for administrative school districts. Administrative school districts operate sets of public and charter schools. The schools operated by each school district are identified using the National Center for Education Statistics (NCES) school and district identifiers. Most commonly, administrative school districts operate local public schools within a given physical boundary; these are what we refer to as “traditional public school districts.” There are specialized administrative districts, like charter school and virtual school districts, that do not have a physical boundary. These districts will not appear on our maps; their data, however, are available in the download files.
Administrative districts differ from the geographic districts used in SEDA 5.0. The key difference is that for geographic school districts, we “reassign” charter schools to the district in which they are physically located (regardless of the entity that operates the schools). We do no reassignment of charter schools in producing the administrative district estimates; charter schools are attached to the traditional public or charter district that operates them.
We use administrative districts in SEDA 2024 for two reasons. First, one of the aims of SEDA 2024 is to help school districts understand their learning recovery needs. Administrative districts have authority to set policy for their schools, as such it is most useful for the estimates to reflect only the schools under their operation. Second, to construct geographic school districts, we need data for individual schools. While the state-reported source data sometimes includes school-level data, the data for many schools is suppressed due to the small numbers of students taking assessments. Because of this we cannot reliably construct geographic school district estimates from this data source.
Construction
The construction of the data occurs in a series of steps, the key steps are detailed below:
Estimating Cutscores
For the 2009 to 2019 data, we use a statistical technique called heteroskedastic ordered probit (HETOP) modeling to estimate the location of the thresholds that define the proficiency categories within each state, subject, grade, and year. This methodology is described on the Educational Opportunity Methods page.
For the 2022 to 2024 data, we assume the state-subject-grade-year test score distribution is normal and use the inverse cumulative standard normal distribution function to find the threshold associated with the proportions of students scoring in each category in the state, subject, grade, and year.
Estimating Means
The next step of our process is to estimate the mean test score in each unit for all students and by student subgroups (gender, race/ethnicity, and economic disadvantage). To do this, we estimate heteroskedastic ordered probit models using the proficiency count data (described under source data) and the thresholds from the prior step. In this step, we produce a mean standardized test score in each unit for every subgroup, subject, grade, and year.
For more information, see Reardon, Shear, et al. (2017)</a<>; and Shear and Reardon (2020).
Linking Means
The mean estimates are not yet on a comparable scale, so we use the National Assessment of Educational Progress (NAEP), a test taken in every state, to place the thresholds on the same scale. This step facilitates comparisons across states, grades, and years. For each subject, grade, and year, we multiply the mean by the state’s NAEP standard deviation and add the state’s NAEP average score.
For more information, see Reardon, Kalogrides & Ho (2019).
Constructing Annual Estimates
We average data across grades, within units, subjects, and years, to get a single annual estimate of average achievement. To ensure comparability across places, we report what this average would be halfway through fifth grade for all units.
Available Achievement Estimates
We provide the following state and district-level estimates by subgroup (where data are available):
- 2019-2022 change in average math scores
- 2019-2022 change in average reading scores
- 2022-2024 change in average math scores
- 2022-2024 change in average reading scores
- 2019-2024 change in average math scores
- 2019-2024 change in average reading scores
- 2019-2023 change in chronic absenteeism rates
Reporting Scales
In the Explorer, we report all test score changes in grade levels. Grade levels are defined using the 2019 national NAEP 4th and 8th grade data (“the 2019 norm group”). We compute one-fourth of the difference between national average 4th and 8th grade scores on NAEP (roughly 11 NAEP points); this describes the average number of NAEP points student test scores differ per grade in each subject using the 4th and 8th grade data. We then rescale the NAEP point scale estimates using those parameters. In this scale, each unit is interpretable as 1 grade level. For example, a 2019-2022 change in average math scores of -1 grade levels means that students in 2022 scored, on average, 1 grade level (roughly 11 NAEP points) below their 2019 counterparts.
Note that SEDA 2024 grade levels are not equivalent to SEDA 5.0 grade levels. In SEDA 5.0, the per-grade growth is defined by a 4-cohort norm group (rather than the 2019 norm group, described above). For more details on how we calculate SEDA 5.0 growth, we refer you to the 2009-2019 Opportunity Explorer Methods.
Interpretation and Data Accuracy
We think of changes in average scores as reflective of changes in the average educational opportunities available to students between two timepoints. For example, if the 2019-2022 change in average math or reading achievement is negative, it means that students in 2022 in that district scored lower, on average, than students in 2019 in that district. This suggests that students in 2022 had fewer educational opportunities (in their schools, homes, neighborhoods, and beyond) to learn to date than the student population in 2019.
Changes in test scores during and after the pandemic may be due to a variety of mechanisms. The test score data in SEDA 2024 may only enable understanding of some of these mechanisms. To provide context for interpreting the data, we include data flags and margins of error.
Population Change Flag
In many districts, the student population shifted from 2019 to 2024. Using enrollment data from the CCD, we flag any districts where the total number of students enrolled changed by more than 20%.
Margin of Error
In some cases, estimates are imprecise; in some cases, an estimated change in average scores is not statistically distinguishable from zero. We have constructed margins of error for each of the estimates to help users identify such cases. We also do not show any estimates on the website where the margin of error is large. For those downloading the data and using it in analysis, standard errors are included in the downloadable data files.