Authorizer Scorecard Technical Appendix

Data

The Education Trust Midwest’s (ETM) authorizer scorecard uses data from the following publically available sources:

• Fall 2010 through Fall 2014 Michigan Center for Educational Performance and Information’s (CEPI) educational entity masters (EEM);
• Michigan Department of Education (MDE) Top-to-Bottom Rankings for 2010-11 through 2013-14;
• CEPI non-resident student file for 2010-11 through 2013-14;
• CEPI list of charter school educational service providers (ESPs) for Fall 2014; and
• MDE Public School Academy (PSA) Updates for Fall 2011 through Fall 2014.

Below is a brief description of the datasets in our analysis. Comprehensive business rules and descriptions for each of the datasets are available publically through the MDE and CEPI.

CEPI Educational Entity Master (EEM)

The EEM is a comprehensive listing of every public and non-public educational entity in the state. The EEM includes information on school names, school types, school building addresses, charter authorizer names, grade levels served, and other relevant information.[i]

The EEM provides a snapshot of all schools in Michigan at a given period of time. Upon request from CEPI, we are able to rely on previous iterations of the Fall EEM to gather information that has changed over time. For our purposes, this includes the charter school authorizer and the school classification type (i.e. traditional public school or charter public school).

MDE Top-to-Bottom

The MDE ranks all public schools in the state on a 0-99 scale, with 99 representing the top public schools in the state. This figure is based on an analysis of academic performance on the state assessment (Michigan Educational Assessment Program (MEAP) and Michigan Merit Examination (MME)), academic improvement, graduation rates, and a gap analysis of academic performance between the top and bottom 30% of students in each school. Public schools are included in the ranking if they have two academic years of state assessment data for at least 30 full-academic year students in at least two tested subject areas. The MDE classifies the bottom 5% of schools on this list as “priority” schools.

In our analysis, we rely only on the academic improvement and the overall rankings found within the MDE Top-to-Bottom dataset. Improvement includes elementary/middle school MEAP math, elementary/middle school MEAP reading, high school MME math, and high school MME reading. According to the MDE Top-to-Bottom business rules, improvement is calculated in one of two ways: for specific grade and subject combinations that allow the use of Performance Level Change (PLC), a weighted PLC is applied. For those schools that do not, the improvement metric is based on either a four-year slope of mean z-scores, a three-year slope of mean z-scores, or a two-year change in mean z-scores, depending on the availability of data.

A detailed review of the MDE Top-to-Bottom business rules are available through the MDE.[ii]

CEPI Non-Resident Student

Several local traditional school districts in Michigan participate in “schools of choice”, which allows students to enroll in schools outside of the local traditional school district they reside in. To note, not all traditional school districts in Michigan participate in this program and many that do limit their enrollment availability. On an annual basis, CEPI collects data on the number of students in each school that are “non-residents,” meaning the number of students who reside in a different traditional school district than the actual school they attend.

Unlike traditional public schools, charter schools do not have geographical district boundaries. Virtually any Michigan student may enroll in a charter, given that there is sufficient enrollment and grade-level capacity. Since charter schools are not limited to geographic boundaries, CEPI lists all of the charter’s students as “non-residents,” since the entire student population would otherwise attend the traditional school district they reside in.[iii]

CEPI Education Service Providers (ESP) Listings

CEPI also publishes a list of educational service providers (ESPs) or management companies associated with each charter public school. ESPs provide a variety of services for schools ranging from payroll and human resources to curriculum development and staff professional development. Because no publically accessible data exists that classifies ESPs by their service types, we do not attempt to categorize ESPs beyond what is available publically. Although not every charter public school in Michigan is associated with management companies, the vast majority of charter public schools are.

MDE Public School Academy (PSA) Updates

The MDE Public School Academy (Charter) office publishes an annual briefing outlining charter school openings, closings, authorizer changes, ESP changes, grade level modifications, and other relevant information.[iv] We use this database primarily to match the initial ESP assigned to each school at opening. In some cases, an ESP may not have been listed in these annual briefings. If this is occurs, we use the school’s most recent ESP (see “CEPI Education Service Providers (ESP) Listings”).

We use these annual briefings combined with the EEM databases (see “CEPI Educational Entity Master (EEM)”) to identify new school openings.

Methodology

In order to derive overall authorizer ratings, we first calculate three sets of scores from the following three categories:

• Authorizer decisions about the schools it currently authorizes: Is the authorizer overseeing a portfolio of schools that is as good as or better than other school options available to parents?
• Authorizer decisions about opening charter schools: Is the authorizer approving new Michigan charter schools that are likely to succeed?
• Authorizer decisions about improving struggling charter schools: Does the authorizer have consistently failing schools?

Authorizer decisions about the schools it currently authorizes

For each charter school in the state, we apply a two-step “good or better” test for each academic year in 2011-12, 2012-13, and 2013-14:

1. Does the charter school perform in the top half of the state overall? (50^th percentile or above on MDE Top-to-Bottom ranking)
2. If not, does the charter school perform in the top half of the state and the local district where most of its students reside for improvement? (Equal or better improvement in MEAP/MME reading and math as the state and the local district average on the MDE Top-to-Bottom ranking)

a. The MDE Top-to-Bottom includes four variables that compare how each public school performs to the state for academic improvement in MEAP math, MEAP reading, MME math, and MME reading, as applicable. All four variables are in z-score units. The MME is only administered at the high school level, while MEAP is administered at the elementary and middle school grades. To find average improvement per district, we simply calculate the average z-score among schools within the district.

A school is marked as having “met the good or better test” if it is either above or equal to the 50^th percentile on the MDE Top-to-Bottom ranking or if it is above or equal to the state and district average for school improvement.

For a school to be below the state or district improvement average, it must be below the state or district in math, reading, or both in that given academic year. For a school to be above the state or district improvement average, it must be above the state or district in both math and reading.

In the case of charter schools, we define the local district as the district where the majority of students reside for the given academic year. For example, consider a charter school located in the city of Grosse Pointe in which the majority of students live in bordering Detroit. This school’s improvement is compared to that of Detroit Public Schools, not the Grosse Pointe Public School System.

This same two-step test is repeated for each academic year. From here, we categorize each school as “below the minimum quality standard” if it fails to meet our “good or better test” for three academic years in a row. Schools that meet the test all three years or have mixed results are categorized as “meeting the minimum quality standard.” Schools are excluded if they are currently closed, despite having consecutive years of data.[1] Michigan also has a number of schools that were formerly traditional public schools that later became charter schools. This includes schools formerly within the Highland Park, Muskegon Heights, and Detroit Public School systems. In order for a school to be included in our two-step test for a given academic year, the school must also be listed as a charter school in that same academic year.

To aggregate to the authorizer level, we derive the ratio between the total number of schools that failed to meet our standard for three consecutive academic years and the total number of schools with at least three consecutive academic years of data (non-missing data) per authorizer. Again, we also are sure to include only open-active charter schools in this calculation.

Authorizer decisions about opening charter schools

In this category, we begin by asking the same questions as in the above section:

1. Does the charter school perform in the top half of the state overall? (50^th percentile or above on MDE Top-to-Bottom ranking)
2. If not, does the charter school perform in the top half of the state and the local district where most of its students reside for improvement? (Equal or better improvement in MEAP/MME reading and math as the state and the local district average on the MDE Top-to-Bottom ranking)

a. The MDE Top-to-Bottom includes four variables that compare how each public school performs next to the state for academic improvement in MEAP math, MEAP reading, MME math, and MME reading, as applicable. All four variables are in z-score units. The MME is only administered at the high school level, while MEAP is administered at the elementary and middle school grades. To find average improvement per district, we simply calculate the average z-score among schools within the district.

Again, a school is marked as having “met the good or better test” either if it is above or equal to the 50^th percentile on the MDE Top-to-Bottom ranking or if it is above or equal to the state and district average for school improvement (derived from MDE Top-to-Bottom).

We then categorize ESPs as “poor performing” if the majority of the ESP’s schools (greater than 50%) fail to meet our “minimum quality standard.” Again, these are schools that fail our “good or better test” over three consecutive academic years. To note, schools without an ESP (also known as self-managed) do not receive an ESP rating. Newer ESPs to Michigan or those without track-records are automatically counted as having met our “minimum quality standard.”

In addition to authorizing new charter schools, an authorizer may also choose to take on a charter that already exists, but that wants to switch authorizers. A charter school may seek to switch authorizers for a variety of reasons, including the fact that its former authorizer refuses to renew its charter contract. To get a sense of the kinds of schools authorizers were taking on, we identified schools that switched authorizers between Fall 2011 and 2014, and then looked at their performance and improvement (based on the MDE Top-to-Bottom ranking) in the academic year right before they made the switch, our “good or better test.” If the school failed the “good or better test” in that specific academic year, it was classified as “poor performing.”

For each authorizer, we then identified the total number of newly opened schools (schools opened between Fall 2011 and 2014) that belonged to “poor performing” operators, and “poor performing” existing charter schools that an authorizer chose to take on during this time period. As mentioned, to find new school openings and schools that switched authorizers, we rely on the EEM databases and the MDE PSA Status Updates. To note, we consider schools that converted from traditional public schools to charter schools as new school openings.

We divided this number by the total number of charters that an authorizer opened or took on between Fall 2011 and 2014 to get the percentage of these new schools that were unlikely to succeed.

Authorizer decisions about improving struggling charter schools

In this category, we aggregate the total number of schools per authorizer that appear on the MDE Top-to-Bottom priority list (bottom 5% of public schools statewide) for two consecutive academic years. However, schools are exempt if either of the following occur:

• in the second academic year that the school was classified as a priority school, the school had improvement that was greater than the state in both math and reading;
• or if the school is no longer open.

The first year a school can be in the bottom 5% is either 2010-11, 2011-12, or 2012-13. A school that appears on the bottom 5% list in 2013-14 for the first time couldn’t have two consecutive years until 2014-15 and is therefore excluded.

To derive a final ratio per authorizer, we find the total number of schools per authorizer that were classified as a priority school over two consecutive academic years (and did not meet any of our two exemptions) by the total number of schools per authorizer that appeared on the MDE Top-to-Bottom priority list in either 2010-11, 2011-12, or 2012-13.

Final Scores

To calculate final scores, we take the ratios found per authorizer in the three categories referenced above and find the difference from 100 percent to calculate the percentage of good decisions each authorizer made in each category. Next, we average across all three categories to find a final “overall” score.

For authorizers that did not open a new school during the years of our analysis, we do no assign a score for the “authorizer decisions about opening charter schools” category. For authorizers that did not have any priority schools in our analysis, we assign full credit (100 points) for the “authorizer decisions about improving struggling charter schools” category. In the least, an authorizer must have a score in the “authorizer decisions about the schools it currently authorizes” category in order to receive a final “overall” score.

Final letter grades correspond to the following table:

Analyses were conducted in Stata 13 and Microsoft Excel.

—————————————————

[1] EEM Close date as of October 15, 2014.

[i] Center for Educational performance and Information,” Educational Entity Master (EEM)”, 2014. http://www.cepi.state.mi.us/eem/PublicDatasets.aspx

[ii] Michigan Department of Education, “Top-to-Bottom Rankings”, 2014. http://www.michigan.gov/mde/0,4615,7-140-22709_56562—,00.html

[iii] Center for Educational performance and Information, “Resident/Non-Resident”, 2014. http://www.michigan.gov/cepi/0,4546,7-113-21423—,00.html

[iv] Michigan Department of Education, “Charter Public Schools”, 2014. http://www.michigan.gov/mde/0,1607,7-140-6530_30334_40088—,00.html