Hot topics

Numerous, diverse survey instruments are being used by Teaching American History (TAH) grantees to measure the impact of TAH programs. These instruments range from project-developed instruments that measure teachers’ self-reported perceived impact of professional development to standardized achievement tests measuring impact on student achievement.  The instruments also vary greatly in terms what specifically is being measured. For example, topical areas covered by the instruments include: student achievement, student interest in history and/or American History, students’ historical thinking skills, teacher content knowledge, teacher instructional practices (both specific to teaching history and more general teaching skills and abilities), teacher self-efficacy, changes in curriculum and lesson plans, and teacher attitudes/motivation.

Specifically in terms of measuring impact related to the two TAH Government Performance and Results Act (GPRA) measures (for more information related to GPRA indicators see www.sei2003.com/TAH/about/GPRAindicators.asp), CEEP conducted a review of data reported on the 278 TAH Annual Performance Reports (APRs or 524Bs) submitted in Fall 2006. The reviewed reports included 91 reports from 2003 TAH grantees, 101 reports from 2004 TAH grantees and 86 reports from 2005 TAH grantees. Findings related to measuring changes in teacher content knowledge and student achievement include the following: 

  • For those grantees reporting using a student content knowledge test, approximately three-fourths (76%) reported using a state level test, 16% reported using a national test (most commonly the Advanced Placement Exam, the National Assessment of Educational Progress, and the Stanford Achievement Test Series, Ninth or Tenth Edition), and approximately 10% reported using an instrument developed by the evaluator or project.

 

  • For those grantees reporting using a teacher content knowledge test, approximately one-fifth (19%) reported using a state level test, 19% reported using a national test (most commonly the Advanced Placement Exam, the National Assessment of Educational Progress, and the SAT Subject Tests), and approximately half of all grantees reported using an instrument developed by the evaluator or project.
     
  • Approximately one-third of grantees report using teacher self-report surveys to measure impact on teacher content knowledge (i.e., measuring teachers’ perception of impact on their content knowledge).
     
In addition to the GPRA measures, TAH grantees often also reported measuring various other student-level changes. For example, approximately 17% of TAH grantees specifically reported measures of change in student attitudes/motivation on their 2006 APR, and 10% reported using measures of change in student historical thinking. Approximately 13% also reported various other student level changes, including changes in student critical thinking skills. In terms of teacher-level changes, approximately half of all grantees reported using measures of change in general teaching skills/strategies or teacher effectiveness; and approximately half of all grantees also reported including measures of pedagogy change specific to history.  Approximately one-fifth of grantees also noted in their 2006 APRs measures of teacher attitudes/interest specific to history.

These findings do have an important limitation that should be noted: the review and synthesis conducted relied completely and exclusively on the 2006 APR reports and supplemental materials provided by TAH grantees, and given the nature of the APRs it is possible that some grantees used instruments that were not specifically noted on the 2006 APRs. Regardless, these findings do provide a basis for understanding the types of instruments being used by current TAH grantees to measure impact of the TAH programs, as well as a general sense of the frequency of use for various instrument types.

The following links on the TAH Evaluation Technical Assistance website provide some basic information for grantees looking for potential instruments to measure impact on teacher and student content knowledge (see www.sei2003.com/tah/eval_resources/measurInstru.asp). In addition, however, many TAH grantees have suggested that it would be helpful to have access to instruments that have been developed by TAH grantees.  Towards this end, CEEP/SEI will be sending out an e-mail in the coming weeks to request TAH grantees to submit any project-developed instruments they are willing to share with other grantees, as well as the names of any other instruments or surveys that they believe might be useful in evaluating TAH programs.  For example, the website’s “Spotlight” this month includes information on an instrument that measures “Teaching American History Perception and Behavior.” With prior permission of the authors of this instrument, a copy of the instrument will be made available on this website  for TAH grantee use. Anyone reading this article is also invited to submit potential instruments directly to us at TAHeval@indiana.edu. Be sure to include in your message a very brief description of the purposes of the survey instrument, and any specifications related to its use (e.g., permission needed/not needed, associated costs, citation, etc.). 

With the cooperation and participation of TAH grantees, we will create a central location for sharing and accessing these instruments on this TAH Evaluation Technical Assistance website, with the hope that these potential TAH evaluation instruments in turn will aid in the effective and efficient evaluation of TAH programs!