Analysis of the NCES 2006 Report Comparing Private Schools and Public Schools Using Hierarchical Linear Modeling
Public - Private School Comparisons of “Modest Utility”
What’s it all about?
On July 14, 2006, the National Center for Education Statistics (NCES) released a report, Comparing Private School and Public Schools Using Hierarchical Linear Modeling (HLM). The HLM study analyzed the 2003 grade 4 and grade 8 National Assessment of Educational Progress (NAEP) data.
This model attempts to examine the differences in mean scores when selected school and student characteristics are taken into account.
Actual scores
The 2003 NAEP reading and math tests were administered to 6,900 public school and 530 private school 4th graders and 5,500 public and 550 private school 8th graders.
The actual 4th grade private school NAEP mean scores in reading were 14.7 points higher that the public schools; 8th grade reading was 18.1 points higher. In math assessments the 4th grader private school scores were 7.8 points higher than public schools and for 8th grade the private school scores were 12.3 points higher than public schools. These are the actual scores of real students taking the tests under real conditions.
HLM Adjusted Scores
The HLM model takes the actual scores and subjects them to a statistical analysis of selected characteristics to determine whether the “private school” advantage disappears. The characteristics used in the comparison were school size and location, gender, race/ethnicity, family income, disability status and degree of Limited English Proficiency.
Discounting these “private school advantages”, the study found that there was no significant difference between public and private schools in 4th grade reading and 8th grade math results; there was a significantly higher difference (7.8) for private school 8th grade reading and a higher public school difference (4.5) for 4th grade math.
Caveats
The NCES report contains a section titled Cautions In Interpretations which advises that estimated effects should not be interpreted as casual relationships. In effect, it cautions that these statistical hypothetical models do not provide information about why stripping away the presumed “advantages” of private schools impacts results. The report itself states that the results are of “modest value."
This study looks at results on one test score at a given time – it does not measure progress over time. It takes raw test scores from a single year and applies statistical controls for demographic factors like race, income, and disabilities. Single-year snapshots of test scores provide limited information about student achievement and nothing about the relative quality of public and private schools
Test scores are just part of what parents consider when making school choices. Parents look at other factors as well: smaller school size, religious and character development, teacher quality, parental satisfaction and over-all “fit’ between the school culture and their family’s desires.
Comparisons relying on one measure of a complex situation cannot provide accurate information about the whole. The HLM report is an exercise in statistical probabilities that provides hypothetical answers, not an accurate picture of the real world of either the public or private schools.
The full report from NCES can be found at
here.
New- August 10, 2006:
NCES Commissioner's Comments on NCES Involvement in HLM Public vs. Private School Study
Mark S. Schneider, commissioner of the National Center for Educational Statistics, voiced reservations about the suitability of NCES becoming involved in studies that use subjective criteria to make comparisons as did this HLM study. The study was undertaken during the administration of his predecessor. Subsequent independent research using different variables produced different results.
"This is not what we should be doing," Mr. Schneider said, and announced that NCES would not conduct future studies where researchers would make subjective choices about selection of variables. He noted that it is the business of NCES to produce raw data --and that data would be made available to independent researchers for analyses in their own names that do not convey a sense of government endorsement of a subjective analysis.