Physics First Yields Dramatic and Sustained Increases in ACT Scores......

ACT^© Test Performance and

the Implementation of a

Physics First Course Sequence

2000-2016

EXECUTIVE SUMMARY

         A new high school science course sequence was evaluated using ACT section scores taken by 180-220 students each year over a seventeen year period.  The new sequence of high school science courses known as “Physics First” has all freshman take Physics, all sophomores take Chemistry, all juniors take Biology and in the senior year students can elect to take any of the sciences available and of interest to them.  This research looked at the performance of the 1100 student college preparatory urban private high school on the ACT test taken by students for college selection and scholarships.  The outcome of the new sequence was an immediate, rapid, statistically significant, large increases in every ACT section with scores rising during the transition and sustained for four years after the transition to "Physics First" was completed.  Surprisingly the largest ACT section increase were the English section scores. This unexpected large improvement may reflect the training of five teachers in "modeling physics" as preparation for the transition from the prior sequence to "Physics First".  Modeling Physics emphasizes verbal communication, productive discourse, and the written summary of the lab experience.  The ACT Composite scores displayed an increase almost 3 times larger than the average improvement in scores from year to year prior to the sequence change.  All ACT sections displayed rapidly increased rates of upward shifts in scores during the transition to "Physics First".  The higher ACT scores were maintained in all sections in the four years following the transition to the new “Physics First” science sequence.  Statistical analysis revealed highly significant results (p<.0001) and large effect sizes (>0.5 using Cohen guidelines) when comparing ACT scores before and after the implementation of "Physics First".

ACT^© Test Performance and

the Implementation of a

Physics First Course Sequence

2000-2016

INTRODUCTION

          An urban college preparatory high school (grades 9-12) of 1,100 students undertook a science sequence change to one that has all freshman taking physics that was called “Physics First”. The change began in the fall of 2008 and was fully completed in the spring of 2012. Performance of the students in the school on the ACT test were examined prior to the sequence change, during the change and for four years after the change was completed.

          The prior sequence had freshman taking an integrated science class with with units that sampled physics, chemistry and biology. The reasoning was that this would provide exposure and vocabulary that would increase comprehension of the later more advanced biology, chemistry and physics courses. In this prior sequence the sophomores took biology, the juniors took chemistry and the seniors could elect physics if they had the mathematical prerequisites. Seniors could also opt out of physics and take another science if they wished. The result of this prior sequence was 60-70% of the students graduating with exposure to a full year of physics. In the new “Physics First” sequence all students were required to take a full year physics course as freshman. In the sophomore year all students take a chemistry course. During the junior year students take a Biology and then in the senior year students take an elective science of their choosing. The result of the new sequence is 100% of the students take a full year physics course in their first year of high school.

An outcome measure utilized a standardized tests normally taken bey every student in the college preparatory high school studied. A widely applied external measure of student intellectual skills includes the externally administered ACT test. Sections of English, Math, Science and Reading are scored separately and are validated to predict college performance and are nationally normed and used to compare individual performance nationally and by state. This research compiled the ACT performance data for this school each year during the time when a graduating class ranged in size from 180-220 students from 2000 to 2016.

          Performance on the ACT by students at this school over a seventeen year period (2000-2016) was tracked ¹. Composite and subsection scores in Science, English, Math nd Reading were included in the analysis. Performance 8 years prior to the new sequence (2000-2007), during the 3 years of transition and then for 5 years after the transition to the new sequence are examined in this study. If the new “Physics First” sequence is successful then the scores of the students at this school relative to previous years would increase. If the new sequence impacts student intellectual skills in a negative manner (as measured by the ACT) then action should would be taken to mitigate the impact (or revert to the older more traditional sequence). Evaluating the data trends of the students was a prudent and responsible action in service to the community.

STUDY LIMITATIONS

          During the years of 2008-2016 there were a number of initiatives which could have influenced ACT scores by the students. During the 2008-2012 science department personnel changes included 4 departures and new hires in the science department out of nine full time permanent positions. The changes in personnel during this time provided opportunities matching new talents and training to current needs of the science program. Administrative initiatives repeatedly emphasized increasing student performance on the ACT over the years and from 2014-2016 a special “standardized testing day” was implemented to raise scores on all sections of the ACT with a school wide class day dedicated to the goal in every academic course and class. Three of the four full time freshman science teachers were trained in the application of the modeling program in the freshman physics classes. A "Science and Faith" initiative provided summer workshop and in-service training that helped teachers reflect on the philosophical roots of the sciences that they teach in 2007, 2009 and 2012. In 2009 science classes moved into a new science building with space, facilities and equipment not previously available. In 2008 a new "Aquinas Program" focused on encouraging students to engage a wide variety of challenging courses were implemented for the highest 10% of the students. Educational research and evaluation is never clear and definitive. The many initiatives acted to confound the possibility of definitive conclusion that only “Physics First” impacted the ACT performance but the evidence and the timing of shifts in scores can help determine the probable impact of the new science sequence.

EXPECTATIONS

          If the new science course sequence has a positive impact on intellectual skills as measured by the ACT then some patterns over using data over the twelve year period were expected to include:

     (1) The clearest increase in ACT Science section scores would be expected to start in 2011 (when the junior class of students in the new sequence would begin taking the ACT in large numbers).

     (2) The Math section would display an increase second only to science in 2011. Math skills and insights would be facilitated with the exposure to physics earlier in the curriculum and so a “Physics First” course sequence would contribute the greater development of quantitative skills

     (3) The Composite score would reflect the increase in Science and Math Scores and show a gentler increase that would still be detectable.

     (4) If there is an impact on the Reading and English scores would be less than the Science (or Math) increase. It is possible that the lab reports generated by students in modeling physics would contribute to language development. Five teachers were trained in modeling physics prior to the sequence change (and 2-3 year assignment to teaching freshman physics).

     (5) The observed increase is expected to start in 2011 and be sustained over the 2-5 years as a result of the sequence that emphasizes the quantitative and analytical aspects of all the sciences starting with Physics and as teachers gained experience in leveraging more process skills and content knowledge using the new sequence.

          The Science section (1) would directly test science related skills and so this would be the reason for expecting the largest gains. The Math scores in other schools that have implemented the new sequence have also shown standardized score increases. Previous studies would then explain the Math section (2) improvement in Math section ACT performance. The Composite score (3) would display an average increase of all sections and tend to reflect the increases in the Science and Math Scores. Finally English and Reading would be expected to show some increase as the freshman Physics emphasis is on extensive writing and interpreting of lab reports expected in 3 of the 4 freshman teachers who have "Modeling" training.

          All subject areas have historically shown correlations of greater than 0.80 between sections and so it is reasonable to assume that improvement in the science section would likely influence other section scores and the Composite score as well.

SMOOTHED DATA ANALYSIS

          Fluctuations from year to year in ACT scores can be influenced by many factors including loss of days to natural events (example: hurricanes), timing of test dates (example: ACT tests date is same as homecoming dance), emotional disruptions (example: September 11, 2001 attacks) or between class differences talents, capabilities and interests. External events and aggregate class differences can pose fluctuations in ACT scores from year to year that act to hide trends. In this study a 3 year rolling average of ACT scores over 12 years are used to establish general trends and mask typical annual variations in the scores. Three years was chosen because it matches the threshold transition period when Physics First cohorts would begin taking the ACT test.

          The rolling average change in ACT scores from 2001 which is 8 years prior to sequence change which started in 2009. The first and final years have only two years averaged. The first year includes 2001 and the year after. The last year includes 2016 and the year prior. All other years are averaged with both the year following and the year prior to the year averaged. This resulted in 17 consecutive years of rolling averaged ACT test scores at this college preparatory school. The larger this change during the transition years then the more likely that “Physics First” impacted the program scores. The timing the improvement is important because so many confounding initiatives were happening during these 17 years of study.

          The use of the rolling average was implemented only for the graphs yo assist in visual recognition of the patterns revealed in this study. All statistical and numerical analysis were performed on the raw data supplied by the ACT in aggregate form.

ACT SCIENCE SECTION SCORE ANALYSIS

          A gentle upward trend in ACT Science Scores is apparent from 2002 through 2007 are followed by a leveling off of Science Section Scores prior to the time when the "Physics First cohorts" began to take the ACT exam in their junior year (2011). In 2011 both seniors who were in the prior sequence and juniors who were in the new sequence are taking the ACT test. Data differentiating these two groups was not available.

          The ACT Science Section scores have a clear pattern. Prior to the transition the Science section scores rose (or remained unchanged) in 62.5% of the years with the average year to year score increase of 0.2933. Science section scores from the middle of the transition (2010-14) rose 80% of the time with an average year over year increase of 0.520. The year over year increase in Science section scores was almost twice (1.96) the average year over year positive change in scores prior to the sequence change. Starting in the middle of the transition until 2 years after the transition the ACT Science scores rose the most of any other section. The Science section scores rose faster than all other section scores during the transitional period. The sudden increase in the rate of rise in Science scores coincides precisely with the transition to the new “Physics First” course sequence.

          After 2013 the Science section scores appear to remain higher than prior to the sequence change. If we compare the Science section scores from prior to the transition (9 years 2000-8) to the Science section scores after the transition was complete (4 years 2013-2016) using a one-tailed t-test and assume unequal variance the probability is 0.0000126 that this happened by chance. There is less than half a ten thousandth of a chance that these results happened only by chance. The data indicates that the link between Science section scores and the change in science sequence are highly statistically significant.

          In the years after the transition to “Physics First” the Science section scores (2013-16) was 1.65 points higher on average than the average scores prior to the science course sequence change (2000-8). Assuming that the control data were the years prior to the sequence change (2000-8) the standard deviation of the ACT Science section scores is 0.260. The effect size ratio of mean differences and standard deviation is 6.35. Using the Cohen² guidelines (6.35>0.5) would classify this change in sequence as having a “large difference effect” on Science section ACT scores.

          The data analysis confirms that the change in science course sequence had a “large difference effect” and had a high statistically significant impact on ACT Science section scores.

ACT MATH  SECTION SCORE ANALYSIS

          In previous studies³ a positive impact on standardized ACT Math section scores were noted with the implementation of a “Physics First” science sequence. The use of mathematics skills earlier in high school when Physics is taken the freshman year was predicted to help improve ACT Math section scores. There is a gentle decline and recovery trend in ACT Math section scores from 2000-8 prior to the transition to the new “Physics First” science sequence. ACT Math section scores trend consistently upward during the transition period and for the year following full implementation in 2013. “Physics First cohorts" began to take the ACT exam in their junior year (2011) with seniors who were in the prior sequence. Data differentiating these two groups was not available.

          Prior to the transition ACT Math section scores rose (or remained unchanged) in 75% of the years with the average year to year score increase of 0.4167. Math Scores from the middle of the transition in 2010 to 2014 rose 60% of the time with an average year over year increase of 0.700. Math section scores rose almost 1.7 times faster from 2010-2014 than the average year over year change in scores prior to the science sequence change. The increase in the rate of rise in Math section scores coincides precisely with the transition to the new “Physics First” course sequence.

          If we compare the Math section scores from prior to the transition (9 years 2000-8) to the Math section scores after the transition was complete (4 years 2013-2016) using a one-tailed t-test and assume unequal variance the probability is 0.000005929 that this happened by chance. The data indicates that there is less that a hundred thousandth of a chance that this result was due to chance and is highly statistically significant.

          In the four years after the transition to “Physics First” the ACT Math section scores (2013-16) were 1.27 points higher on average than the average scores during the prior science course sequence (2000-8). Assuming that the control data were the years prior to the sequence change (2000-8) the standard deviation of the ACT Science section scores is 0.4147. The effect size ratio of mean differences and standard deviation is 3.05. Using the Cohen⁴ guidelines (3.05>0.5) would classify this change in science sequence as having a “large difference effect” on ACT Math section scores.

          The data analysis confirms that the change in science course sequence had a “large difference effect” and the evidence is that the science sequence change had a high statistically significant impact on ACT Math section scores.

ACT ENGLISH SECTION SCORE ANALYSIS

          A gentle incline in ACT English section scores from 2003 with an increased rate of rise during the transition to the new “Physics First” science sequence from 2008-2010 were noted in the graph. ACT English section scores trended consistently upward until 2013 which is the year following full implementation of the new science sequence in 2013. “Physics First cohorts" began to take the ACT exam in their junior year with seniors in 2011. Data differentiating these two groups was not available.

          The observed changes in ACT English Section scores have a clear pattern. Prior to the transition English section scores rose (or remained unchanged) in 75% of the years with the average year to year score increase of 0.2571. English scores from the middle of the transition in 2010 until after the transition in 2014 rose 60% of the time with an average year over year increase of 0.800. English section scores rose almost 3.11 times faster from 2010-2014 than the average year over year increase in scores prior to the science sequence change. The increase in the rate of rise in English section scores coincides precisely with the transition to the new “Physics First” course sequence.

          After 2013 the English section scores appear to remain higher after the science sequence change. If we compare the English section scores from prior to the transition (9 years 2000-8) to the ACT English section scores after the transition was complete (4 years 2013-2016) using a one-tailed t-test and assume unequal variance the probability is 0.0000210 that this happened by chance. The data indicates that there is less that a fourth of a ten thousandth of a chance that this result was due to chance and is highly statistically significant.

          In the four years after the transition to “Physics First” the ACT English section scores (2013-16) were 2.61 points higher on average than the average English section scores during the prior science course sequence (2000-8). Assuming that the control data were the years prior to the sequence change (2000-8) and the standard deviation of the ACT English section scores is 0.3742. The effect size ratio of mean differences and standard deviation is 6.971. Using Cohen⁵ guidelines (6.971>0.5) this would provide evidence that classifies the change in science sequence as having a “large difference effect” on ACT English section scores.

          The data analysis confirms that the change in science course sequence had a “large difference effect” and a highly statistically significant impact on ACT English section scores.

ACT READING SECTION SCORE ANALYSIS

          A gentle incline in ACT Reading section scores from 2003 until 2005 is observed in a rather stable period from 2000 until 2008. There is an increased rate of rise during the transition to the new “Physics First” science sequence from 2008-2010 were noted in the data. ACT Reading section scores trended consistently upward until 2013 which is the year following full implementation of the new science sequence in 2013. “Physics First cohorts" began to take the ACT exam in their junior year (2011). In 2011 both seniors who were in the prior sequence and juniors who were in the new sequence are taking the ACT test. Data differentiating these two groups was not available.

Prior to the science course sequence transition ACT Reading section scores rose (or remained unchanged) in 75% of the years with the average year to year score increase of 0.300. Reading Scores from the middle of the transition in 2010 until after the transition in 2014 rose 80% of the time with an average year over year increase of 0.440. ACT Reading section scores rose almost 1.47 times faster from 2010-2014 than the average year over year increase in scores prior to the science sequence change. The increase in the rate of the rise in Reading section scores coincides precisely with the transition to the new “Physics First” course sequence.

After 2013 the Reading section scores appear to remain higher than prior to the science sequence change. If we compare the Reading section scores from prior to the transition (9 years 2000-8) to the ACT Reading section scores after the transition was complete (4 years 2013-2016) using a one-tailed t-test and assuming unequal variance the probability is 0.0000125 that this happened by chance. The data indicates that there is less that a ten thousandth of a chance that this result was due to chance and is highly statistically significant.

          In the four years after the transition to “Physics First” the Reading section scores (2013-16) were 2.20 points higher on average than the average Reading scores during the prior science course sequence (2000-8). Assuming that the control data were the years prior to the sequence change (2000-8) the standard deviation of the ACT Reading section scores is 0.419. The effect size ratio of mean differences and standard deviation is 5.26. Using the Cohen⁶ guidelines (6.971>0.5) would classify the change in science sequence as having a “large difference effect” on ACT Reading section scores.

          The data analysis confirms that the change in science course sequence had a “large difference effect” and had a high statistically significant impact on ACT Reading section scores.

ACT COMPOSITE SCORE ANALYSIS

          The composite score is an equal weighted average of the all sections of the ACT that include English, Math, Reading and Science Section. There is a gently rising ACT Composite score form 2003 until 2008. ACT Composite scores trend consistently upward during the transition period and for the year following full implementation in 2013. “Physics First cohorts" began to take the ACT exam in their junior year (2011). In 2011 both seniors who were in the prior sequence and juniors who were in the new sequence are taking the ACT test. Data differentiating these two groups was not available.

          The observed changes in ACT Composite scores have a clear pattern. Prior to the transition ACT Composite scores rose (or remained unchanged) in 50% of the years with the average year to year score increase of 0.200. The Composite Scores rose 80% of the years from the middle of the transition in 2010 to 2014 with an average year over year increase of 0.575. Composite scores rose 2.87 times faster from 2010-2014 than the average year over year change in scores prior to the science sequence change. The increase in the rate of rise in Composite section scores coincides precisely with the implementation of the new “Physics First” course sequence.

          After 2013 the Composite scores appear to remain higher than prior to the science sequence change. If we compare the Composite scores from prior to the transition (9 years 2000-8) to the Composite section scores after the transition was complete (4 years 2013-2016) using a one-tailed t-test and assume unequal variance the probability is 0.0000408 that this happened by chance. The data indicates that there is less than half a ten thousandth of a chance that this observed result was due to chance and is highly statistically significant.

          In the four years after the transition to “Physics First” the ACT Composite scores (2013-16) were 2.00 points higher on average than the average scores during the prior science course sequence (2000-8). Assuming that the control data were the years prior to the sequence change (2000-8) the standard deviation of the ACT Composite scores is 0.315. The effect size ratio of mean difference and standard deviation is 6.36. Cohen guidelines⁷ (6.36>0.5) would classify this change in science sequence as having a “large difference effect” on Composite section scores.

The data analysis confirms that the change in science course sequence had a “large difference effect” and had a high statistically significant impact on Composite section scores.

Summary of ACT Score Analysis 2000-2016

and the Possible Impact of Physics First

          The data strongly supports the assertion that "Physics First" has had an immediate and positive impact on ACT scores particularly on Math and Science Section but may also in English and Reading sections of the ACT as well.

          The largest increase in ACT Science section scores were observed from 2010-2014 in all sections of the ACT. English showed that greatest rate of increase during the transition (3.11) while Math scores we the second fastest rising scores during the transition (2.87). All of the section scores showed increase in the rate of rise during the transition to the new Physics First science course sequence change.

          The largest increases in average section scores were in the English (2.61) and Reading (2.20) sections of the ACT. All of the sections showed increases the were highly statistically significant.

The effect size of the ACT English section score increases was slightly larger than the Science section (6.34). Math had the smallest improvement of the four section scores. All of the effect sizes were large and positive.

          The largest single effect size was observed on the standardized ACT English section scores concurrent with the implementation of a “Physics First” science sequence.  Four of five science teachers were trained in the use of “Modeling” in the physics classroom and this approach emphasizes productive classroom discourse, student articulation during discussions and written lab reports. The use of language skills to generate required formal lab reports in modeling based freshman physics classes is suspected to have influenced the outcomes in English of this study. Surprisingly the ACT English Section scores also displayed a large increase among all years studied at precisely when Physics First Cohorts first began to take the ACT test. Due to the general increase of the English Section Scores prior to the implementation of Physics First it is probable that this increase in not, alone, responsible for the increase in scores.

ACT SCIENCE SCORES COMPARISON WITH NATIONAL AVERAGES

          In this section we compare the ACT Science Section scores to the national average.  The scores clearly show that while national scores remained stable over the seventeen year period there school scores were slightly higher than the national averages until 2009 which was during the transition of our school from the prior traditional sequence to the new Physics First sequence. Following the transition to Physics First the upward shift compared to the national average increased.  This increase is is evaluated using the prior difference in scores compare to the scores after the transition to the new Physics First sequence is complete.

          The ACT Science Section scores have a clear pattern. Prior to the transition from 2000-2007 the Science section scores at the school was 0.53 points higher on average each year.  After the transition is complete (2012-16) the school was higher than the national average with an average difference of 2.52 points higher each year after the transition is complete.  The increase in difference between the school average and national average scores was 440% higher than it was prior to the change to the new Physics First sequence.

         When comparing the differences before and after we assume the control data were the years prior to the sequence change (2000-7) the standard deviation difference in ACT Science section scores was 0.838.  Using a one-tailed t-test and assume unequal variance the probability is 0.000002  that this happened by chance.  This result means that there is 2 in a million chance that this outcome occurred by chance. The effect size ratio of mean differences and standard deviation is 5.3. Using the Cohen² guidelines (5.3>0.5) would classify this change in sequence as having a “large difference effect” on Science section ACT scores.

          The data analysis confirms that the change in science course sequence had a “large difference effect” and had a high statistically significant impact on ACT Science section scores when compared to the national average.

PREDICTION AND OUTCOMES 

          All five of the expectations outlined at the start of this study, which were identified as indicating the positive impact of the "Physics First", were realized or exceeded by the results observed in the data. (1) Large and clear increases in ACT section scores including Science were observed at the start of 2010 and continued after the sequence was fully implemented. (2) The Math section scores displayed a statistically significant and large increase realized after the implementation of the Physics First course sequence. (3) The Composite score reflected even larger increases than the increases in Science and/or Math alone. (4) Surprisingly the rise in English and Reading scores were larger than expected possibly because the teachers were trained in the use of modeling approaches in the classroom that has an emphasis on verbal communication and lab report production. (5) The observed increases were maintained for four years following the full transition of the school to a Physics First sequence.

An archived pdf version of this research is available at:

https://drive.google.com/file/d/1TMcclxWg8ohAGUkwmqWIFz7DxMaMkzEO/view?usp=sharing

Raw Data Available upon request at DrTimCBurgess@gmail.com