The Education Trust-Midwest (ETM)
2016 Charter School Authorizer Scorecard Technical Appendix
Data
The Education Trust-Midwest’s (ETM) authorizer scorecard uses data from the following publically available sources:
- fall 2010 through fall 2015 Michigan Center for Educational Performance and Information’s (CEPI) educational entity masters (EEM);
- Michigan Department of Education (MDE) Top-to-Bottom Rankings for 2010-11 through 2013-14;
- CEPI non-resident student file for 2010-11 through 2013-14;
- CEPI list of charter school educational service providers (ESPs) for fall 2015; and
- MDE Public School Academy (PSA) Updates for fall 2011 through fall 2015.
Below is a brief description of the datasets in our analysis. Comprehensive business rules and descriptions for each of the datasets are available publically through MDE and CEPI.
CEPI Educational Entity Master (EEM)
The EEM is a comprehensive listing of every public and non-public educational entity in the state. The EEM includes information on school names, school types, school building addresses, charter authorizer names, grade levels served and other relevant information.[i]
The EEM provides a snapshot of all schools in Michigan at a given period of time. Upon request from CEPI, we are able to rely on previous iterations of the fall EEM to gather information that has changed over time. For our purposes, this includes the charter school authorizer and the school classification type (i.e. traditional public school or charter public school).
MDE Top-to-Bottom
Through 2013-14, the MDE has ranked all public schools in the state on a 0-99 scale, with 99 representing the top public schools in the state. This figure is based on an analysis of academic performance on the state assessment (MEAP and MME), academic improvement, graduation rates, and a gap analysis of academic performance between the top and bottom 30% of students in each school. Public schools are included in the ranking if they have two consecutive academic years of state assessment data for at least 30 full-academic year students in at least two tested subject areas. The MDE classifies the bottom 5% of schools on this list as “priority” schools.
In our analysis, we rely only on the academic improvement and the overall rankings found within the MDE Top-to-Bottom dataset. Improvement includes elementary/middle school MEAP math, elementary/middle school MEAP reading, high school MME math, and high school MME reading. According to the MDE Top-to-Bottom business rules, improvement is calculated in one of two ways: for specific grade and subject combinations that allow the use of Performance Level Change (PLC), a weighted PLC is applied. For those schools that do not, the improvement metric is based on either a four-year slope of mean z-scores, a three-year slope of mean z-scores, or a two-year change in mean z-scores, depending on the availability of data.
A detailed review of the Top-to-Bottom business rules are available through the MDE.[ii]
CEPI Non-Resident Student
Several local traditional school districts in Michigan participate in “schools of choice,” which allows students to enroll in schools outside of the local traditional school district they reside in. Not all traditional school districts in Michigan participate in this program and many that do limit their enrollment availability. On an annual basis, CEPI collects data on the number of students in each school that are “non-residents,” meaning the number of students who reside in a different traditional school district than the actual school they attend.
Unlike traditional public schools, charter schools do not have geographical district boundaries. Virtually any Michigan student may enroll in a charter school, given that there is sufficient enrollment and grade-level capacity. Since charter schools are not limited to geographic boundaries, CEPI lists all of the charter school’s students as “non-residents,” as the entire student population would otherwise attend the traditional school district they reside in.[iii]
CEPI Education Service Providers (ESP) Listings
CEPI also publishes a list of educational service providers (ESPs) or operators associated with each charter public school. ESPs provide a variety of services for schools ranging from payroll and human resources to curriculum development and staff professional development. Because no publically accessible data exists that classifies ESPs by the services they provide, we do not attempt to categorize ESPs beyond what is available publically. Although not every charter public school in Michigan is associated with operators, the vast majority of charter public schools are run by operators.
MDE Public School Academy (PSA) Updates
The MDE Public School Academy (Charter) office publishes an annual briefing outlining charter school openings, closings, authorizer changes, ESP changes, grade level modifications, and other relevant information.[iv] We use this briefing primarily to match the initial ESP assigned to each school at opening. In some cases, an operator may not have been listed in these annual briefings. If this occurs, we use the school’s most recent ESP (see “CEPI Education Service Providers (ESP) Listings”).
We use these annual briefings combined with the EEM databases to identify new school openings. To remain consistent with our 2015 report, Accountability for All: The Need for Real Charter School Authorizer Accountability, we carry over the same operator listings from the 2015 report for charter schools opened in fall 2015 or prior.
Methodology
In order to derive overall authorizer ratings, we first calculate three sets of scores from the following three categories:
- Authorizer decisions regarding the schools it currently authorizes: Is the authorizer overseeing a portfolio of schools that are as good as, or better than, other school options available to parents?
- Authorizer decisions regarding opening of new schools: Is the authorizer approving high-performing charter operator openings?
- Authorizer decisions regarding improvement of their worst performing schools: Is the authorizer improving its chronically failing schools?
Authorizer decisions regarding the schools it currently authorizes
For each charter school in the state, we apply a two-step “good or better” test for each academic year in 2011-12, 2012-13, and 2013-14:
1. Does the charter school perform in the top half of the state and the local district where most of its students reside for improvement (equal or better improvement in MEAP/MME reading and math as the state and the local district average on the MDE Top-to-Bottom ranking)?
a. The MDE Top-to-Bottom includes four variables that compare how each public school performs to the state for academic improvement in MEAP math, MEAP reading, MME math, and MME reading, as applicable. All four variables are in z-score units. The MME is only administered at the high school level, while MEAP is administered at the elementary and middle school grades. To find average improvement per district, we simply calculate the average z-score among schools within the district.
OR
2. Does the charter school perform in the top half of the state overall (50th percentile or above on MDE Top-to-Bottom ranking)?
A school is marked as having met the “good or better test” if it is either above or equal to the state and district average for school improvement or above or equal to the 50th percentile on the MDE Top-to-Bottom ranking.
For a school to be below the state or district improvement average, it must be below the state or district in math, reading, or both in the given academic year. For a school to be above the state or district improvement average, it must be above the state or district in both math and reading.
In the case of charter schools, we define the local district as the district where the majority of students reside for the given academic year. For example, in a hypothetical charter school located in the city of Grosse Pointe in which the majority of students live in bordering Detroit, the school’s improvement would be compared to that of Detroit Public Schools, not the Grosse Pointe Public School System.
This same two-step test is repeated for each academic year. From here, we categorize each school as “below the minimum quality standard” if it fails to meet our “good or better test” for three academic years in a row. Schools that meet the test all three academic years or have mixed results are categorized as “meeting the minimum quality standard.” Schools are excluded if they are currently closed, despite having consecutive years of data.[1] Michigan also has a number of schools that were formerly traditional public schools and later became charter schools. This includes schools formerly within the Highland Park, Muskegon Heights, and Detroit Public School systems. In order for a school to be included in our two-step test for a given academic year, the school must also be listed as a charter school in that same academic year.
To aggregate to the authorizer level, we derive the ratio between the total number of schools that failed to meet our standard for three consecutive academic years and the total number of schools with at least three consecutive academic years of data (non-missing data) per authorizer. Again, we also include only open-active charter schools in this calculation.
Authorizer decisions regarding the opening of new schools
In this category, we begin by asking the same questions as in the above section:
1. Does the charter school perform in the top half of the state and the local district where most of its students reside for improvement (equal or better improvement in MEAP/MME reading and math as the state and the local district average on the MDE Top-to-Bottom ranking)?
a. The MDE Top-to-Bottom includes four variables that compare how each public school performs next to the state for academic improvement in MEAP math, MEAP reading, MME math and MME reading, as applicable. All four variables are in z-score units. The MME is only administered at the high school level, while MEAP is administered at the elementary and middle school grades. To find average improvement per district, we simply calculate the average z-score among schools within the district.
OR
2. Does the charter school perform in the top half of the state overall (50th percentile or above on MDE Top-to-Bottom ranking)?
Again, a school is marked as having met the “good or better test” if it is either above or equal to the state and district average for school improvement or above or equal to the 50th percentile on the MDE Top-to-Bottom ranking.
We then categorize ESPs as “poor performing” if the majority of the ESP’s schools (greater than 50%) fail to meet our “minimum quality standard.” These are schools that fail our “good or better test” over three consecutive academic years. Schools without an ESP (also known as self-managed) do not receive an ESP rating. Newer ESPs to Michigan or those without past track-records are automatically counted as having met our “minimum quality standard.”
In addition to authorizing new charter schools, an authorizer may also choose to take on a charter that already exists, but wants to switch authorizers. A charter school may seek to switch authorizers for a variety of reasons, including when the former authorizer refuses to renew its charter contract on academic grounds. To get a sense of the kinds of schools authorizers were taking on, we identified schools that switched authorizers between fall 2011 and 2015, and then looked at their performance and improvement (based on the MDE Top-to-Bottom ranking) in the academic year immediately preceding the authorizer switch, on our “good or better test.” If the school failed the “good or better test” in that prior academic year, it was classified as “poor performing.”
For each authorizer, we then identified the total number of newly opened schools (schools opened between fall 2011 and 2015) that belonged to “poor performing” operators, and “poor performing” existing charter schools that an authorizer chose to take on during this time period. As mentioned, to find new school openings and schools that switched authorizers, we rely on EEM databases and the MDE PSA Status Updates. To note, we consider schools that converted from traditional public schools to charter schools as new school openings.
We divide this number by the total number of charters that an authorizer opened or took on between fall 2011 and 2015 to get the percentage of these new schools that were unlikely to succeed.
In order to maintain consistency with our 2015 report, Accountability for All: The Need for Real Charter School Authorizer Accountability, we carry over the operator “minimum quality standard” ratings from the 2015 scorecard instead of recalculating them for the 2016 report. This ensures that authorizers who have not opened or taken on any new schools since the last report receive the same sub-score in this category as the 2015 report.
Authorizer decisions regarding improvement of their worst performing schools
In this category, we aggregate the total number of schools-per-authorizer that appear on the MDE Top-to-Bottom priority list (bottom 5% of public schools statewide) for two consecutive academic years. However, schools are exempt if either of the following occur:
- in the second academic year that the school was classified as a priority school, the school had improvement that was greater than the state in both math and reading; or
- if the school is no longer open.
The first year a school can be in the bottom 5% is either 2010-11, 2011-12, or 2012-13. A school that appears on the bottom 5% list in 2013-14 for the first time couldn’t have two consecutive years until 2014-15 and is therefore excluded. Due to the state’s transition to the M-STEP assessment, MDE did not publish a new Top-to-Bottom ranking in 2014-15.
To derive a final ratio per authorizer, we find the total number of schools per authorizer that were classified as a priority school over two consecutive academic years (and did not meet any of our two exemptions above) by the total number of schools per authorizer that appeared on the MDE Top-to-Bottom priority list in any of the years 2010-11, 2011-12, or 2012-13.
Final Scores
To calculate final scores, we take the ratios found per authorizer in the three categories referenced above and find the difference from 100 percent to calculate the percentage of good decisions each authorizer made in each category. Next, we average across all three categories to find a final score.
For authorizers that did not open or take on a new school during the years of our analysis, we do no assign a score for the “authorizer decisions regarding opening of new schools” category. For authorizers that did not have any priority schools in our analysis, we assign full credit (100 points) for the “authorizer decisions regarding improvement of their worst performing schools” category. In the least, an authorizer must have a score in the “authorizer decisions regarding the schools it currently authorizes” category in order to receive a final score.
Final letter grades correspond to the following table:
| Average Score | Letter Grade |
| 90-100 | A |
| 80-89 | B |
| 70-79 | C |
| 50-69 | D |
| 0-49 | F |
Analyses were conducted in Stata 13.
[1] EEM Close date as of January 4, 2016.
[i] Center for Educational performance and Information,” Educational Entity Master (EEM)”, 2016. http://www.cepi.state.mi.us/eem/PublicDatasets.aspx
[ii] Michigan Department of Education, “Top-to-Bottom Rankings”, 2016. http://www.michigan.gov/mde/0,4615,7-140-22709_56562—,00.html
[iii] Center for Educational performance and Information, “Resident/Non-Resident”, 2016. http://www.michigan.gov/cepi/0,4546,7-113-21423—,00.html
[iv] Michigan Department of Education, “Charter Public Schools”, 2016. http://www.michigan.gov/mde/0,1607,7-140-6530_30334_40088—,00.html