Friday, June 23, 2017

Research Proposal: Models of AP Exam Performance

Research Proposal: Models of AP Exam Performance


In the last two decades there has been a tremendous increase in the number of high school students taking Advanced Placement exams.   In the same time period there was also a substantial increase in the percent of exam takers who failed to receive a passing sentence.

Change in AP Exam Participation and Failure Rates
1996 to 2016
AP Exams Taken
% Getting a Grade of D or F
Number of Students Getting a Grade of D or F
% change

 The chart above reveals that in 2016 nearly 2 million test results on AP exams taken by high school students were a D or an F.  The percent of people taking an AP exam that got a grade of D or F rose from 36.3% in 1996 to 42.0% in 2016. 

These numbers indicate to me that policymakers and educators need to develop policies and procedures that better prepare students for AP courses.

Aggregate statistics may understate the extent to which many high school students struggle on their AP exams.   A recent blog examined the grades received on the 20 AP exams taken by the largest number of students.   The statistics obtained from the College Board reveal that the mode grade (the most commonly received grade) was an F on 4 tests and a D on 6 tests.    Also, the average grade on 14 of 20 of the most popular exams was less than a C.

Post on grades received on the most common AP Exams:

There is substantial literature indicating that many students benefit from participation in AP exams.  Academic research has found strong benefits to students in the form of higher graduation rates and higher grades in college from people who take AP exams.

Washington Post Article summarizing some academic research on the benefits of AP exams.

This type of research is a classic example of selection bias in statistics.    The students who sign up for AP courses and AP exams tend to be stronger students, more academically orientated and test better than students who take less challenging courses.    It is likely that the superior performance of AP students in college has much more to do with the characteristics of the AP students than any benefit incurred by taking the exam.

There are clear-cut benefits for students who pass AP exams.  These benefits often include the receipt of some college credit, a reduction in tuition expenditures, an increase in the probability of graduating on time and the possibility of taking more challenging courses in college.

Students who take and fail multiple AP exams do not obtain these benefits.   They have learned how to fail.  There is very little academic research on college performance of people who fail multiple AP exams.   Many educators appear to emphasize increases in AP exam participation over AP exam performance.  The academic research reflects this perspectival bias.  

To the best of my knowledge, there is no academic research on the factors determining success or failure on AP exams.   Neither the College Board nor the academic literature provides information on or guidance on the optimal number of AP exams that students should take.  There is no published empirical work that uses information on AP exam workload and student performance on previous work to forecast the outcomes of AP exams for different students.  There also does not appear to be academic work relating multiple AP exam failures to future GPA in college or on-time graduation rates.  

The proposed research described below attempts to fill this void in the literature.  I am proposing several statistical tests and models that could be used to better understand the impact of AP course load and different student attributes on AP exam outcome.   I am also proposing a method to rank high schools on the basis of both AP exam participation rate and AP exam outcomes.  The information obtained from these models can help students choose a more appropriate AP schedule, help teachers better prepare their students for their AP exams and provide parents, school systems and the public better information on the performance of AP programs at particular schools.

Issues Pertaining to AP Exam Workload

The first area studied involves the impact of AP exam workload on the on the pass rate on AP exams and the number of AP exams passed.   The analysis can be conducted on data from a single high school, multiple high schools separately, an entire school district, or high schools in a broader geographic area.

The issue of the impact of an additional test taken on the percent of total AP exams passed differs from the question of the impact of additional AP exams on total tests passed for a population.  Consider 400 people taking 3 tests each with a 50% pass rate.   The total number of tests taken is 1,200 and the total number of tests passed is 600.   Consider the same number 400 people taking 4 tests each.   The people take a total of 1,600 tests.    At the 50% pass rate the people will pass 800 AP exams.   The pass rate could fall to 37.5% and the 400 people would pass 600 tests.   A pass rate below 37.5% would in this example lead to an increase in the number of tests taken reducing the number of exams passed.

A third statistic of interest is the percent of people taking multiple exams that fail all exams or pass all exams compared to the expected percent of people failing or passing all exams.   This statistic would incorporate information on the pass rates on particular exams and would adjust for difference in pass rates among exams.

The proposed AP exam performance measures looks at the entire distribution of exams passed for each combination of exams taken.  The proposed statistical procedure compares actual exam outcomes for people taking multiple exams to the expected outcome under the assumption that each exam outcome was independent with the pass probability identical for all people in the pertinent population taking the test.

Consider a person taking three exams AP biology, AP calculus AB and AP History.   The probability of getting an A or B on each exam is 0.276 for biology, 0.42 for calculus AB and 0.298 for history.   (These probabilities are based on national proportions for the 2016 exam.)  The table below contains the probability of getting zero, one, two or three A or B grades based on the assumption that the test results are independent.  

AP Exam Probabilities Under the
Assumption of Independence
Probability of Getting an A or a B
Calculus AB
Probability of three A or B grades
Probability of exactly two A or B grades
Probability of exactly one A or B grade
Probability of no A or B grades

The actual percent of people getting exactly 0, 1, 2, or 3 A or B grades can be compared to the expected percent based on the assumption of independence.   If the percent of people who get no A or B grades is substantially above 0.295 then it is possible that some effort should be made to arrange a smaller course load for students in the sample.

It may also be the case that the proportion of people getting three A or B grades is greater than 3.5%. 

 I suspect that there will be a lot of people at both ends of the distribution and fewer than expected in the middle.

The Impact of Student History and Workload
on AP Exam Performance

The second set of issues involves the creation of statistical models, which attempt to determine whether a particular student is likely to succeed with a particular AP exam course load.  I am considering the creation of two types of regression models.

The first model in this section would predict the number of exams passed based on the number of tests taken and the attributes of the student including GPA, SAT scores, and SAT II participation rates and scores.

The second model in this section would predict passage rates on particular AP exams for a set of students taking the same schedule.   For example, all students in the sample would take AP Calculus, AP History, and AP Biology.   The dependent variable in separate logistical regression model would be the probability of passing each test.  Potential explanatory variable include SAT scores, High school GPA, quality of high school, and SAT II participation dummy and score.

I suspect that these regressions can provide invaluable information about what students need to do to increase the likelihood that they pass AP exams and what school students need to do to better prepare their students for the exam.  For example the models might reveal that students who do not take or do well on the SAT achievement tests in a particular field like biology or math do not do well on the respective AP exams in those subject areas.   This type of result suggests that resources and time should be spent preparing for subject area SAT II tests prior to enrolling in AP courses.

The models would be estimated for all available AP course combinations.    Each set of estimated logistical regression model coefficients would be used to obtain predicted pass probabilities.   The predicted pass probabilities from a model where a person takes AP calculus and AP biology could be compared to the pass probabilities obtained from a model where the same student takes AP history in addition to AP calculus and AP Biology.     A comparison of predicted pass rates from these two regression models would provide insight into the wisdom of taking the third exam.

The information obtained from logistical regression models of AP exam performance for different course loads would be incorporated into simple to use software that will provide insights about the wisdom of taking different AP schedules for each student based on the previous performance of the student.

The proposed statistical research attempts to increase our understanding of the factors that determine the outcome of AP exam performance.   These models might help educators better prepare students for some tests and prevent some students from failing multiple tests.

Rating High Schools Based on Student AP Exam
 Participation and Performance

My informal discussion of AP tests with educators and my reading of the literature suggest that many in the education profession place too high a priority on participation and too low a priority on performance.  

A case in point is the U.S, News and World Reports rating index for high school.    The rating index incorporates two pieces of information on AP exams – the percent of students who take an AP exam and the percent of test takers who pass at least one exam.

This index does not penalize a high school when a student takes a really large number of AP exams (let’s say 5) and only passes 1.    In fact, the student that goes 1 for 1 on AP exams gives the school the same amount of credit as a student who goes 1 for 5.  

My view is that an appropriate rating of high school AP exam performance should provide some penalty when students take and fail multiple AP exams.   This goal could be accomplished by adding a third component into the U.S. News and World Report rating system where the third component is C3 = 100 – percent of AP Test Takers who take more than 2 AP tests and fail to get a 3 or above on the majority of their AP exams.  

At the very least, high schools should be required to publish the number of students who take 3 or more AP exams and fail to get an A or a B on at least one of the three or more exams taken.

This formula would not discourage schools from enrolling their students in taking one or two exams but would encourage schools to examine difficult schedules more closely. 

Concluding Remarks:  Education costs are accelerating upwards.    Success on AP exams can save tuition, improve educational outcomes, and help people graduate on time.  Students who take and fail to pass multiple AP exams do not receive these benefits.   They do obtain a habit of failure, one that needs to be broken.

The academic literature and the educators that I have spoken to often appear more interested in expanding participation on AP exams than in improving student performance.    This emphasis appears misguided. 

Policies and programs that aim to improve AP exam performance should not be that difficult to create.   For example, it is possible the case that students who take SAT subject tests do much better on AP exams in the same subject than students who do not take AP subject tests.  Also, performance in some AP courses could be enhanced by additional in-school course time.  

In addition, resources should be allocated to on-line course like the ones offered by Johns Hopkins in order to provide choices to students in lower-performing schools and to expand participation in schools with few AP course offerings.

The research proposed here has the potential to provide the information needed for more students to become successful on AP exams.   The initial empirical research would be followed with policy proposals including changes the ways courses are taught and structured that might improve student performance on AP exams. I hope educators shift their emphasis from increasing AP exam participation towards improvements in student performance.   I also hope that I obtain funds and data to complete my empirical research modeling AP exam performance.

Monday, May 22, 2017

Competition Inside Schools

I am creating a list of education policy proposals for the 2020 campaign.   This proposal on competition between course providers inside public schools is the second one on the list.

Competition Inside Public Schools:

I very much hope to work on a 2020 campaign.

Proposal Two:   Provide competition inside schools between school faculty and private firms that teach specific subject areas.

The use of private firms would be encouraged and funded when the traditional public option was not achieving desired results.  For example, it might be useful to employ private firms to replace publicly hired math teachers at schools with low scores on standardized math exams.  Private firms might also be employed when there was a shortage of teachers in certain subjects or when the private firm could realize economies of scale and scope compared to school employees. Three examples where this might happen include instruction of some foreign languages, computer programming and coding courses and camps, and instruction of Advanced Placement courses.

Comment One:  Most of the political debate over education reform centers around issues like whether poorly performing public schools should be closed and/or whether charter schools should be created to compete with public schools. Policy makers have learned that is very difficult and expensive to close and replace schools.   In rural and suburban districts, there is often room for only one high school or middle school.  Charter schools have a role but charter schools often select the best students, leaving many other students behind.

Often the largest problem in a school district involves a single subject.   It is less disruptive and expensive to bring in private educators to teach one subject than to close or replace an entire school.    The economic argument for facilitating competition between course providers is laid out in the following essay.

Competition in the Education Industry:

Comment Two:   One reason for the emphasis on mathematics is that there is a shortage of people graduating from American schools who are capable of filling STEM positions.   Recent statistics indicate that schools are not placing a high priority on teaching computer schools.  Around 1.4 percent of students take AP computer science compared to around 40 percent who take AP English.

Schools aren’t teaching kids to code; Here is who is filling the gap?

One of the biggest impediments to teach computer science and coding in schools is that schools have trouble recruiting and keeping qualified computer teachers. A firm that specializes in teaching several computer courses and programming languages would be more effective at recruiting and keeping teachers.   Moreover, one firm with a staff of teachers and some on-line material could cost effectively service several schools and districts. 

Comment Three:   Students in the United States are significantly falling behind students in Europe in learning foreign languages.   Twenty European countries mandate that students learn at least one foreign language.  The United States does not have any nationwide foreign language standards.   Only 25 percent of Americans report that they speak a second language and only 7 percent of this group report they learned the language in school.

Many schools cannot find qualified language instructors in basic European languages.  These articles document the existence of a shortage of foreign language teachers in Maine and Connecticut.

Very few high schools can offer courses in important languages including Hindi, Mandarin, Japanese, Arabic, and Russian.   Knowledge of these languages is vital to national security and our economy and could for some students lead directly to a job.   It is economically impractical for small schools to offer courses in these languages even though these languages can provide a valuable skill to some students. However, one firm with several language teachers and on-line resources could teach language courses in several school districts.

Comment Four:   A successful Advanced Placement Program can help prepare students for college, provide college credit, and lead to earlier graduation.   This is large dispersion among high schools in AP exam participation and pass rates.  This is illustrated by the data below on exam pass rates at 22 high schools in the Denver Metropolitan area where more than 10 students took the AP English Exam

Performance on AP English Exam at
22 Denver High Schools with More Than 10 Test Takers

The quality of the AP teacher in a course may impact AP exam performance.   A school with low AP exam performance in some courses might be able to improve course performance by using on-line AP exam vendors instead of the in-school AP exam teacher.  (Several programs offer on-line AP exam courses including the Johns Hopkins gifted and talented program.)