E V A L U A T I O N B R I E F BU R E A U O F PR O G R A M EV A L U A T I O N Volume 1, Issue Volume 7, Issue 2, January 2013 Examining Teacher Perceptions and Student Performance of Teachers Mentored by Teacher Development Specialists By Kathy Terry, Ph.D. In August of the 20112012 school year, teacher development specialists (TDSs) were hired to provide teachers in the Houston Independent School District (HISD) individualized, professional development opportunities to support the new Teacher Appraisal and Development System. This brief summarizes the perceptions of teachers regarding their experiences with their TDS and assesses student performance of a sample of TDS mentored teachers on content area tests. The perceptions of the TDS mentors will also be discussed. Background To align with the dist of having an effective teacher in every classroom, the Houston Independent School District (HISD) introduced the Teacher Appraisal and Development System (A & D System) at the beginning of the 2011 2012 school year. The A & D System was developed for two main reasons: (1) t the classroom and (2) to provide teachers with timely, accurate feedback that links them to individualized professional development opportunities (HISD 2011). The teacher development specialist (TDS) position was created in an effort to support classroom teachers and provide personalized, job embedded trainings. At the beginning of the 2011 2012 academic year, HISD administrators hired approximately 130 TDSs. Nearly every HISD teacher was assigned a TDS. Based on the TDS assignment rosters, TDS caseloads averaged 75 teachers. The roles of the TDS included providing teachers with professional development opportunities and trainings that align with the new A & D System. TDS responsibilities included classroom observations, identification of development areas, and helping teachers with everything from planning lessons to asking questions that improve student performance (HISD, 2011). Teacher development specialists started their training in the summer of 2011. The 5 day training included learning the A & D System, an instructional practice rubric, as well as learning new coaching techniques that best help teachers improve student outcomes. Data and Methods In order to explore the perceptions of HISD teachers about experiences with their TDS, four questions were included in a larger online survey administered to teachers about the A & D System. Approximately, 27 percent (n=3,017) of HISD teachers completed the survey items in February 2012 and 43 percent (n=4,714) responded in May 2012. The larger survey was conducted by The New Teacher Project (TNTP). In addition, an online survey to access the sentiments of the TDS about their job responsibilities and interactions with teachers was administered in May 2012. Sixty seven percent Research and Accountability Department
Teacher Development Specialists, 2011 2012 2 (n=87) of the teacher development specialists completed the survey. Percentages reported in this brief were based on the total number of responses. included in calculating the percentages. To assess the impact of teachers on student performance, the 2012 Education Value Added Assessment System (EVAAS) data in reading, language, math, science, and social studies were gathered for a sample of the 2011 2012 teachers that were directly identified by teacher development specialists who received concentrated mentoring. Since the State of Texas Assessments of Academic Readiness (STAAR) was a new testing program, no data were available for comparison. Value added status was available for teachers instructing students in grades three through eight and indicated to what level the average district performance. How did teacher respondents rate their experiences with the teacher development specialist on their campus? During the 20112012 school year, all HISD teachers were asked about their experiences with the TDS on their campus. Teachers were administered the survey twice; once in the winter of 2012 and in the spring of 2012. Table 1 displays the mid year and end of year responses in percentages. Approximately, 52 percent of teacher understood the role of the TDS at their school at the mid year point while 51 percent of respondents statement at the end of the year. Nearly 63 percent of teacher respondents either strongly agreed or agreed that they had a positive relationship with their TDS at the mid year administration. Forty three percent of teachers strongly agreed or agreed that the TDS served as a useful resource in their ability to grow as a teacher. Although the level of agreement (strongly agreed and agreed responses combined) decreased for all survey items at the end of the year, the responses were very comparable to those expressed by teacher respondents at the mid year point. How did teacher development specialists feel about their job responsibilities and interactions with campus teachers? Ninety three teacher development specialists responded to the online survey accessing their experiences in their new role as a coach mentor for campus teachers. Approximately 86 percent indicated that their teaching experience was greater than five years, with 54 percent with Table 1. Teacher Perceptions of their Experiences with Teacher Development Specialists, 2012 Strongly Disagree Disagree Somewhat Disagree Somewhat Agree Agree Strongly Agree n Mid Year % % % % % % 3,017 I understand the role of the TDS at my school. 7.2 7.5 8.2 25.0 38.3 13.9 End of Year 8.4 7.5 8.8 24.3 37.3 13.7 4,646 I receive useful feedback aligned to the Instruction Practice rubric from my TDS. Mid Year 12.8 10.0 9.1 22.7 32.3 13.1 2,987 End of Year 12.9 10.4 10.2 23.3 29.9 13.3 4,603 I have a positive professional relationship with my TDS. Mid Year 7.5 4.9 5.3 19.5 40.1 22.7 2,992 End of Year 7.9 5.1 6.0 19.4 38.7 22.9 4,609 My TDS serves as a useful resource in my ability to grow as a teacher. Mid Year 13.8 10.1 10.5 22.5 28.4 14.7 2,988 End of Year 15.2 11.9 11.5 24.4 25.6 11.3 4,714 Source: The New Teacher Project (TNTP), 2012.
Teacher Development Specialists, 2011 2012 3 teaching experience ranging from 6 to 15 years. In addition, 89 percent of the TDSs noted that they had served as a mentor to fellow teachers in the past. Table 2 presents the responses from teacher development specialists about their experiences. Over 87 percent felt they received adequate training for the position, while approximately, 86 percent felt supported by campus principals to complete their job responsibilities as the TDS. Fifty six percent of the TDS respondents disagreed or strongly disagreed with the statement, I was satisfied with the number of teachers I was assigned. How did teacher development specialists divide their time among teachers assigned to them? Figure 1 displays the results of how TDS respondents divided their time among the teachers assigned to them. The majority of TDS respondents (64 percent) reported that they spent their time mentoring with teachers who needed their assistance the most. Approximately 16 percent worked with the teachers who were the most receptive to their mentoring, while 10 percent of TDS respondents spent their time working with teachers in hard to staff schools. What were the EVAAS results of teachers directly mentored by teacher development specialists during the 20112012 school year? On the online survey, teacher development specialists identified teachers that they spent the most time coaching and providing professional development. Figure 2 (page 4) presents the 2012 value added results of those identified teachers who were directly mentored by a teacher Table 2. Response Rates of Teacher Development Specialists about their Experiences as a TDS, 2012 Strongly Disagree Disagree Agree Strongly Agree n I received adequate training to accomplish my job duties. % % % % 86 4.7 8.1 53.5 33.7 I felt supported by the campus principals. 1.1 12.6 57.5 28.7 87 I felt supported by the Professional Support and Development administrators. 3.4 9.2 49.4 37.9 87 I was satisfied with the number of teachers I was assigned. 17.2 39.1 39.1 4.6 87 The TDS meetings were helpful. 3.5 19.8 46.5 30.2 86 8.0 64.4 16.1 10.3 1.1 020406080100 Evenly across all teachers Most time with those who needed it Most time with those most receptive Most time with teachers from hard to staff schools Most time with beginning teachers Percentage
Teacher Development Specialists, 2011 2012 4 development specialist during the 20112012 school year. Value added data were available for teachers instructing students in grades three through eight. For reading, the majority of teachers (52 percent) had no detectable difference (NDD) from the district average, while approximately 10 percent fell well below the district average. Twenty nine percent of math teachers directly assisted by teacher development specialists fell well below the district average growth and 32 percent had NDD. For language, science, and social studies, the majority of teachers (58, 46, and 44 percent, respectively) were NDD from the district average growth. Limitations Given that the position of teacher development specialists was in its first year of existence, there were limited ways to measure the effectiveness of their work. Within this report, the value added results were reported for teachers identified as receiving the most assistance from a teacher development specialist; however, no comparison data were available. Furthermore, the teachers with whom the TDS spent the most time were teachers identified as lower performing. Conclusions HISD is committed to providing its teachers with personalized, job embedded professional development experiences. The role of the teacher development specialist is to provide the needed mentoring and professional development to district teachers. During the first year of implementation (20112012), teacher development specialists worked with teachers to help them strengthen their instructional practices. Approximately, 37 percent of teacher respondents agreed or strongly agreed that their TDS served as an useful resource in their growth as a teacher. Future examination of the impact of teacher development specialists on district teachers is needed. Currently, teacher development specialists are providing teachers with welcomed classroom support, mentoring, and professional development experiences. References Houston Independent School District (2011). Teacher Development Specialists. Professional Development Services website. 10.2 29.1 6.0 12.513.1 21.6 11.410.7 23.219.7 52.3 31.7 58.3 46.444.3 11.413.916.710.713.1 4.5 13.9 126.96.36.199 0 10 20 30 40 50 60 70 80 90 100 ReadingMathLanguageScienceSocial Studies Well BelowBelowNDDAboveWell Above Figure 2. Value Added Results by Content Area for Teachers Directly Mentored by a Teacher Development Specialist, 2012
1 E V A L U A T I O N R E P O R T BU R E A U O F PR O G R A M EV A L U A T I O N Volume 1, VolumVolume 7, Issue 2, May 2013 The Effect of the New Digital Energy (NDE) Game on SWXGHQWV Science Energy Knowledge Acquisition, Interests, and Teacher Instructional Practices, 2012 2013 By Venita Holmes, Dr.P.H. science game (NDE) and science energy knowledge and science interest. A survey was used to explore teacher perceptions of the game on their instructional practices. Outcome measures were collected at the individual level and the game was played at the team level. The results of the study indicate that for secondary students, post slightly, and for elementary students, post science knowledge decreased slightly. These results suggest that NDE does not appear to be strongly associated with improved science knowledge and interest, at least for the subset of students who participated in this study, although increases were statistically significant. Further analysis is needed to d . The majority of teachers whose students participated in the game indicated changing their was fun and expanded their understanding of science. Longitudinal tracking of elementary student science course selection and grades as they progress through school could provide alternative measures of NDE program impact. Background In its third year, Chevron expanded funding of the NDE game from middle and high school to include elementary school students in the Houston Independent School District (HISD). Tasks were modified to accommodate all student academic levels. Students played as teams against artificial intelligence, competing across three levels of difficulty. Variations in difficulty were incorporated into lessons that students must master to open options within the game. Lessons and questions led students to game play decisions that required understanding of physics, chemistry, earth science, and math concepts. A meta site supplemented their learning and success for subsequent game play. The game combined strategy, construction, and game management, requiring players to build energy companies, gain dominant market share, and meet the needs of cities throughout the United States. The program included field trips to the Ocean Star Offshore Drilling Rig & Museum in Galveston and the +RXVWRQ 0XVHXP RI 1DWXUDO 6FLHQFHV :(,66 Energy Hall and Hall of Paleontology. Prizes were awarded by level (elementary and secondary) for high scores and participation in the game. As technological competencies increase, U.S. students continue to struggle in science achievement (Sparks, 2013; U.S. Department of Education, 2011). Research based on international science assessments found a decline in fourth to tenth grade science performance as students progress through school (Leinward & Pollock, 2007). Psychology learning theorists explored the role of social and cultural experiences in gaining knowledge and the development of learning through socially mediated activities (Cole, 1978). 9JRWVN\n SURSRVHV KXPDQV DUH DFWLYH YLJRURXV participants LQ WKHLU RZQ H[LVWHQFH WKURXJKRXW WKH developmental process (Cole, p. 123). Practical experiences in science education through active participation in social milieu may provide a Research and Accountability Department
Digital Energy Program: The Use of Games, 20123 2 1200 1549 3406 815 1192 1996 0 1000 2000 3000 4000 2010 20112011 20122012 2013 Number of Students RegisteredNumber of Students Playing favorable environment for learning (Driver, Newton, & Osborne, 2000). The National Science Foundation (n.d.) emphasizes the importance of designing and testing innovative approaches to learning by applying technology to meet the needs of all learners. Research postulated that computer games enhance the educational experience of students, partly due to the multifaceted process required to learn science (National Research Council, 2011). Jones (1996) noted that the interaction between the use of technology and student learning may be influenced by various factors, including studentV LQWHUHVW LQ science. Bulunuz and Jarret (2009) identified the relationship between interest and effort, concluding that the more effort, the more interest, and interested individuals are more engaged and engrossed by activities because of their value. Ogunkola (2011) supported relationships between attitudes, interests, study habits, and the use of technology among a sample of 300 high school students in Barbados. The importance of stimulating studentV interest has also been documented among middle school students, noting that students who are disengaged from school are more likely to have poor attendance and more likely to drop out (Balfanz, R., Herzog, L. & Mac Iver, D, 2006). Mundie (2008) maintains that technology has the potential to help reinvent the education process, and excite and inspire young learners to embrace science, matK DQG WHFKQRORJ\n S In efforts to add to the body of knowledge, Plass (2011) cites Resnik in noting the best educational activities connect not only to important concepts but also to VWXGHQWV LQWHUHVWV DQG SDVVLRQV (p.1). Computer games are inherently interactive and have, increasingly, become a new and innovative approach to teaching and learning science as they capture the attention of youth. In addition, computer games allow students to be transported into another reality and engage in activities that simulate the real world while exploring natural phenomena that cannot be directly observed (Games Research, 2011; Mundie, 2011). Previous research conducted on an NDE secondary level student sample found higher overall mean ratings on a survey that measured VWXGHQWV LQWHUHVW and attitudes in science following participation (Holmes, 2012). Nevertheless, there continues to be a gap in the research concerning the use of games, which features contribute to student learning, and whether social factors influence student outcomes. This study will build on the body of knowledge by considering background characteristics of students who participated in the game and assessing whether outcomes and perceptions persist in both elementary and secondary student populations. Methodology All HISD elementary and secondary science teachers were invited to recruit students to participate in the NDE game during fall 2012. Teachers were required to register and sign a letter of commitment. As shown in Figure 1, the number of students who registered and the number of students who played the game in 2012 2013 more than doubled from the first year of implementation in 2010 2011 possibly due to the SURJUDP V expansion from secondary to include elementary schools. Sixteen secondary schools and 26 teachers were represented in the 2011 2012 data considering 85 secondary schools were invited to participate. Comparatively, 22 schools and 29 teachers were represented in the 2012 2013 data. This included 10 elementary and 12 secondary schools. Measures and Variables Students were administered a web based, 17 item multiple choice science energy assessment using a pre and post test design. HISD science curriculum staff and the 7LHWURQL[ V ,QF JDPH developers compiled the science assessment questions to align to the Texas Essential Knowledge and Skills (TEKS) standards and content covered in the game. The instrument had been used in previous research conducted in 2011 2012 with a comparable secondary student sample (Holmes, 2012). Based on the 2012 2013 data, the Cronbach alpha coefficient was .71, indicating Figure 1. Number of all students who registered and participated in the NDE game, 2010 2011 through 2012 2013.
Digital Energy Program: The Use of Games, 20123 3 good internal consistency (Pavot, Diener, Colvin, and Sandvik, 1991). Another variable analyzed in the study was sWXGHQWV science interests. Data were captured during pre and post game participation. Survey items were adapted from the Program for International Student Assessment 2009 (PISA). The science interest survey was used in previous research with secondary students (Holmes, 2012). The instrument had good internal consistency in 2012 2013, with a Cronbach alpha coefficient of .91. Links to both instruments were accessed at the game V website. Completion of the assessment and survey were voluntary. However, students were offered incentives (e.g., tablets, trips to museum, trophies) to participate. Elementary student data were analyzed independently from secondary student data. This was the first year elementary students participated in the game, thus, they had less exposure to the survey and assessment content and data collection methods. Assessment questions were not modified due to an expectation that content would be covered in the game. Students were organized in teams. It is estimated that teams spent approximately 100 hours playing the game (F. Hughes, Tietronix Software, Inc., personal communication, April 10, 2013). Given the team format of the game, the actual number of hours each team member spent playing the game is unknown. This is important because individual student outcomes could be related to exposure to science content and experiences during the game. Data Analysis Descriptive statistics, including means and standard deviations, were calculated based on student responses to the science energy assessment and the science interest survey items using IBM SPSS software. Paired sample data measuring the percent correct on the pre and post science energy knowledge assessments and the mean ratings on the science interest survey were described in this evaluation. Pearson correlation coefficients measured the association between pre and post science energy knowledge as well as pre and post interest in science (p<.05). Linear mixed effects modeling examined the effect of social factors on VWXGHQWV science interest and science energy knowledge after participating in the game. This model took into account students nested in schools. The analysis was conducted at the school level because most schools had only one teacher. Teacher level rather than school level analysis would have reduced sample sizes, further compromising the reliability of the results. Student Sample Data for 462 secondary students and 78 elementary students who completed the four measures (pre and post science energy assessments as well as pre and post science interest surveys) were included in this analysis. Students in the matched samples were enrolled at six elementary and seven secondary schools (five middle and two high schools). Table 1 presents demographic characteristics of the matched student samples. The majority of elementary and secondary students were female (56.4 and 53.0 percent, respectively) and gifted and talented (G/T) (75.6 and 58.4 percent, respectively). A higher proportion of secondary students received free or reduced lunch compared to elementary students (72.9 vs. 46.2 percent). Compared to students district wide, both the elementary and secondary student samples were more likely to be female and G/T, and less likely to be at risk, or LEP. The demographic characteristics of district level students were 51.1 percent male, 15.6 percent G/T, 62 percent at risk, 29.8 percent LEP, and 79.7 percent economically disadvantage (PEIMS, 2012 2013) analysis. What was the effect of the NDE game on knowledge and interest? 'HVFULSWLYH VWDWLVWLFV 3HDUVRQ V FRUUHODWLRQV paired t test analysis, and linear mixed effects modeling were conducted to evaluate the effect of Table 1: Demographic Characteristics of Matched Paired Elementary and Secondary NDE Student Samples, 2013 Elem. (4th 5th) (n = 78) Secon. (6th 12th) (n = 462) District Gender % % Male 43.6 47.0 51.1 Female 56.4 53.0 48.9 Free/Reduced Lunch 46.2 72.9 79.7 At Risk 21.8 12.3 62.0 Gifted & Talented (G/T) 75.6 58.4 15.6 LEP 15.4 17.1 29.8 Elementary Grade Fourth 38.5 Fifth 61.5 Secondary Grade Middle (6th 8th) 89.4 High (9th 12th) 10.6