Summary

Student achievement in mathematics declines drastically starting in the fifth grade in Puerto Rico according to the results of the Measurement and Evaluation for Educational Transformation (META-PR) standardized tests[1]. The Puerto Rico Department of Education (PRDE) lacks a uniform process for formative assessment of mathematics content to be administered after each unit in all grades. To provide a resource for both teachers and students, we developed standardized formative tests for grades 3-11. The Mathematics Impact Tests (MITs) measure conceptual mastery in the acquisition and transfer objectives for each unit/topic of the grade level[2]. The MIPs are administered using Moodle and results are available instantly. As part of the testing process, teachers are trained and receive a curriculum guide with specific recommendations on how to administer, assess, and re-teach students to optimize their pedagogical practice. During this presentation we will share how teachers and their coaches at two high schools in the San Juan Education Region optimized their pedagogical practice by increasing their students' academic achievement on the 2016 META-PR tests.

Introduction

During the last twenty years, questions have been raised in the educational field about how well traditional educational paradigms respond to the processes of how learning takes place and how knowledge is developed. This is why a series of changes in educational approaches, processes and practices have been proposed. As time goes by, technology has positively increased the learning process of students. With this, it is demonstrated that there are many benefits that it offers to education in our society. Even more so when the youth is so exposed to it.

As established by the Department of Education in Circular Letter # 07-2016-2017[3], mathematics as a school discipline, must promote that students form mental schemes in which mathematical contents and processes acquire meaning, with a reasonable degree of abstraction to understand mathematics with meaning and not simply memorize a set of rules and procedures. Although technology does not replace the teacher, new technologies open spaces in which the student can live experiences that are difficult to reproduce with traditional means such as pencil and paper. Aliaga & Bartolome (2005), in their research, state that it is necessary to develop in students competencies in relation to self-regulation of learning, and in general, to autonomous learning and to insist on the development of competencies (skills, knowledge and criteria) in the search, evaluation, selection, interpretation and application of information. The key element is which tools/strategies are the most appropriate to ensure student learning and in turn which are the most feasible for teachers to provide quality teaching.

Currently, Puerto Rico's public schools use the Measurement and Evaluation Tests for the Academic Transformation of Puerto Rico (META-PR) to evaluate the academic achievement of students in the subjects of English as a Second Language, mathematics, Spanish and science. These tests are aligned to the content excellence standards established in 2014 (PRCS 2014), by the Puerto Rico Department of Education and complies with state and federal requirements, and the agreement that the PRDE has with the U.S. Department of Education, including the new federal Every Student Succeeds Act (ESSA). The results of this test, administered during the 2015-16 academic year, show that as students advance in grade, the level of proficiency decreases (see Figure 1).

Percentage of students at proficient/advanced level according to degree (academic year 2015-16).

One of the "problems" with the META-PR tests is that they are administered only once a year and the results are received only after the academic year is over. So they only give the teacher a glimpse of student achievement at the end of the course and not necessarily during the process. In addition, it does not measure 9th and 10th grades, leaving them devoid of the evaluation of academic achievement through standardized tests. This is why Global Education Exchange Opportunities developed the Mathematics Impact Tests (MIP) from 3rd to 11th grade. Through the use of the PIM, teachers are empowered with the content they teach and provide students with a system of constant feedback on their academic achievement.

Development

The Mathematics Impact Tests were developed and validated during the 2014-15 academic year. These tests use as a theoretical framework the curriculum maps used by teachers to teach mathematics. The curriculum map[4] is an official document of the Department of Education, which contains in the desired outcomes; the unit summary, essential questions and enduring understanding, transfer and acquisition objectives, the Puerto Rico Standards (PRCS 2014), stage 1 (expected outcomes); unit alignment, content focus, mastery and skills, stage 2 (assessment evidence); performance tasks and other evidence and stage 3 (learning plan); suggested learning activities and examples for lesson plans. Each grade level contains several units (curriculum map) according to the content that the teacher will develop throughout the school year. The PIMs are designed to measure academic achievement before and after each of the grade level units (see Figure 2).

Figure 2. Structure of the PIMs of the seventh grade Pre-Algebra course.

The professionals participating in the PIMs were as follows: Dr. Odette Piñeiro - project supervision, Dr. Sylvia Hernandez - project supervision, Dr. Cristina Alvarado - content validation of all grades, test administration, advising of the whole study, Prof. Julmarie Alvarado - project coordination and testing in Moodle, Dr. Edgardo Reyes - eleventh grade test design and writing, reliability study and data analysis, Prof. Maria A. Hernández Alvarado - design and writing of eleventh grade, Prof. Juanita Alvarado - design and writing of third to sixth grade and Prof. Jenyfer Albelo - design and writing of seventh, eighth and ninth grade.

In the validation process of these tests, the content of each item and its alignment with the transfer and acquisition objectives and the Grade Level Standards and Expectations (PRCS 2014), according to the Curriculum Map for grades 3 to 11, were analyzed.

Elements that influenced the development of the MIPs (see Figure 3):

Figure 3. Process for the design of PIMs.

Once the MIPs were validated, a pilot study was conducted to determine their reliability (Fraenkel and Wallen, 2006). That is, the tests were administered to a small sample of students similar to the population of interest that is not a participant in the Project. The reliability or internal consistency coefficient was calculated with the data obtained in the pilot study. Cronbach's alpha was calculated with the data obtained in the pilot study, in order to determine the level of reliability of the tests (Fraenkel and Wallen, 2006). In this way, the internal consistency of the tests was evaluated.

The schools participating in the reliability study were SU Ana Dalila Burgos Ortiz and Alberto Meléndez School in the town of Orocovis P.R., Albert Einstein School in the town of San Juan P.R., Rosalina C. Martínez School and Mariano Abril School in the town of Guaynabo P.R.

According to Hernández, Fernández and Baptista (2006), the reliability coefficient or internal consistency coefficient Cronbach's Alpha can range from zero to one; where an internal consistency coefficient with a value of zero Cronbach's alpha value of zero means zero reliability; and an internal consistency coefficient with a value of one represents a maximum Cronbach's alpha value of one represents a maximum of reliability. For purposes of the Mathematics Impact Test the reliability coefficient should be approximately 0.70 (Fraenkel and Wallen, 2006). The computer program Statistical Package for Social Sciences (SPSS)was used to calculate the reliability coefficient Cronbach's alpha.

What is sought is that all values are positive. As well as the Cronbach's AlphaCronbach's alpha, the scores must be high. If the item - test correlation is negative or even if it is positive, it is very low, they are not strongly related to the total test score. In general, when this occurs, the logical thing to do is to remove the items that do not perform well from the analysis in order to increase the Cronbach's alpha.

During the 2015-2016 school year, these assessments were validated for grades three through eleventh, and are available with scoring guides, rubrics and anchoring for paper, pencil and/or digital administration (Moodle), which allows for easy administration and grading of results in less than 24 hours without the need for reliable Internet access. Using Cronbach's Alpha indicator, which is used to study correlations between test items, better known as internal consistency. The data are as follows (Figure 4):

Figure 4. Internal consistency of the PIM from third to eleventh grade.

After validation, during the 2015-16 academic year these tests were administered in two high schools in the San Juan educational region. As a result, students showed a significant increase in their academic achievement. This evidenced with the results of the 2016 META-PR Tests and grades. In the Rosalina Caraballo de Martínez school, an increase in the level of proficiency was reflected in the 8th and 11th grades of 14% and 16% respectively. On the other hand, the Mariano Abril Intermediate school reflected an increase in 7th and 8th grades of 11% and 1%, respectively.

The National Council of Teachers of Mathematics (2007) stresses the importance of using Formative Assessment so that teaching and learning can be altered based on the results. When formative assessment is used, it helps students clarify and understand what they need to know, creates effective communication or discussions, and students are a learning resource for each other. Other research such as Hanover Research (2014) mentions that students who receive formative assessment do better and improve learning outcomes than those who do not. It is also beneficial for students who do not achieve at the appropriate level (Black & William, 1998). This strategy has been taken for granted.

With impact testing teachers can assess and evaluate almost immediately and students can answer questions and improve the learning-teaching process. When delivered appropriately, teachers are required to (a) know their standards, (b) share clean learning objectives, (c) collect and document evidence of student learning, (d) interpret the data to provide effective feedback that promotes student learning, (Chapuis et al., 2012) cited in Chan et al., 2014. If feedback is targeted at the appropriate level, it can help students understand, engage, or develop effective strategies for processing the information intended to be learned (Hattie and Timperley, 2016).

Once the students' pre- and post-test results were administered and obtained by unit, they were grouped by indicator (transfer and acquisition objectives), tabulated, analyzed, interpreted, and prioritized by group and grade level. The math coach then discussed and established recommendations for how the established priorities were addressed with the regular classroom teacher and the resource room teacher from the special education program. These student priorities and needs were addressed during the school year by the math teacher and the special education resource room teacher during school time. In addition, they were attended to in the Extended School Time (TLE) at the Rosalina Caraballo de Martínez School and the Skills Recovery Center (CRD) at the Mariano Abril Middle School, after school hours. This strategy was instrumental in increasing the academic achievement of students according to grades and in increasing the level of proficiency in the results of the 2016 META-PR Tests.

According to the results of the PIMs, one can see the increase and decrease in academic achievement according to the units administered during the 2016-2017 school year (see Figure 5).

The distribution of grades in the comparison of the 10 and 30 weeks (see Figure 6), shows the increase or decrease in academic achievement. In the Mariano Abril Intermediate School, it is observed that there was an increase in academic achievement in the 30-week grades compared to the 10-week grades. On the other hand, the report shows an increase at the Rosalina Caraballo de Martínez School in the ninth and tenth grades, with the seventh and eighth grades as priorities.

Figure 6. Grade Distribution Report (comparative 10 weeks vs. 30 weeks, Mariano Abril Intermediate and Rosalina Caraballo de Martínez Schools (March 2017).

Training, coaching and follow-up were provided to participating teachers on: PIMs, use of data (PIM results), decision making (accountability), Math Instructional Guides and Math Skills Workbooks.

In order to facilitate the planning and learning process in the classroom and as complementary tools to the PIM, the Mathematics Instructional Guides and the Mathematics Skills Workbooks were designed for grades 6, 7, 8, 9 and 11. Taking into account that the teacher often does not have the resources such as: technological resources (internet), books, guides, exercise books, among others. Sometimes there are teachers who have doubts about how to teach an indicator, these tools are of great support, help and easy access for the teacher.

Math Instructional Guides:

  • Develop all concepts and skills to be taught according to the units of the Curriculum Maps of the Puerto Rico Department of Education.
  • The DEPR normative documents such as Curriculum Alignment, Content Standards and Grade Expectations (PRCS 2014) for Mathematics, Curriculum Maps, Unit Plan and Sequence Calendar were used as a frame of reference.
  • They are composed of different parts among them: description of the units, vocabulary, content: it includes examples one of these developed by steps, the others include the answers.
  • They include Performance Tasks: developed step-by-step and practice worksheets for students. The Performance Tasks are offered with the objective of providing alternatives for their use and application in the classroom.
  • As part of the Performance Tasks, some suggested recommendations are included that could be implemented for differentiated education in the classroom.

Math Skills Notebook:

  • They contain practice exercises with grade level skills.
  • Different sources, including META - PR 2015-2016, DEPR Pre and Post and NAEP exercises for 8th grade.
  • Descriptors of GOALS - PR 2016.
  • Answers to the exercises.
  • Addresses of different resources with additional exercises and games.

Conclusion

PIMs were implemented for the purpose of:

  • diagnose through the results of the pre-test, the prior knowledge of each student on the content of the unit.
  • quantify through post-test results, the percent increase acquired per unit for each student by comparing pre- and post-test results.
  • identify the most and least mastered concepts and skills in each unit on both the pre-test and post-test.
  • address the specific needs of each student for the teaching-learning and re-teaching process.

PIM is focused on monitoring and increasing student achievement in mathematics. It promotes teacher data-driven decision making, (see Figure 7).

Figure 7. Decision-making process based on the results of the PIM.

The PIM allows differentiation in education by providing the teacher with a diagnostic assessment (Pre-Test) of the student in each unit. It identifies the strengths and weaknesses of each student on the content of the units. It can be part of the summative evaluation of the student. Serves as a focus for formative evaluation. Favors the assessment process. Access to test results (pre and post) immediately, if administered through the Moodle platform.

Promotes the action research process in the classroom for decision making:

  • time investment since it can be used as part of the student's evaluation instruments and the results are available once the student finishes answering the pre- or post-test (if it is through the Moodle platform).
  • the results are used for lesson planning, the re-teaching process.
  • integration and alignment between standards and between curriculum map units.
  • Differentiated Education Strategies to be used with subgroups of students in the classroom.
  • the development and implementation of the teacher's Innovative Project.
  • the development and implementation of the MECPA Plan in the area of mathematics (learning communities).
  • its construction is multiple-choice and extended-response, thus promoting practice for the META-PR tests.

The PIMs provide the opportunity for students to increase their academic achievement according to the needs (skills not mastered) identified in the pre-test results and then compared to the post-test.

The student becomes a participant in their academic progress; beforehand they have a notion of their mastery of the content of the unit and the areas of need, so they focus on the construction of their learning individually and among peers. They can access their results to self-assess their academic achievement and make decisions. Develops leadership in student mentors in peer-to-peer teaching.

Projections for the 2017-2018 school year, are: develop progress sheets of Pre and Post PIM results by student to be discussed with parents at the 10, 20, 30 and 40 week grade submissions.

Bibliography

Frankel, J. and Wallen, N. (2006). How to Design and Evaluate Research in Education. 6th ed. Mc Graw Hill, New York.

Hernández, R., Fernández, C., and Baptista, P. (2006). Research Methodology (3rd ed.). Mexico: McGraw-Hill.

Annexes

[1] http://www.de.gobierno.pr/files/PPT_RESULTADOS_METAPR_2016.pdf

[2] The Mathematics Impact Tests (MIP) developed by Global Education Exchange Opportunities, Inc.

[3] Circular Letter No. 07-2016-2017 Public Policy on the Organization and Curricular Offering of the Mathematics Program at the Elementary and Secondary Levels of the Public Schools of Puerto Rico.

[4]http://intraedu.dde.pr/Materiales%20Curriculares/Forms/AllItems.aspx?RootFolder=%2FMateriales%20Curriculares%2FMatem%C3%A1ticas&FolderCTID=0x012000FD4344A3AA05134AA793F1EFE40EB099&View={20BE0ADC-980C-4B35-8826-DB6381045F03}