Teaching American History: Evaluation Resources & Technology Assistance

Over the years evaluation has been viewed as an adversarial process. Much like a visit to the doctor’s office for a physical can be viewed as either a positive or a negative experience, a program’s experience of evaluation is largely dependent on a sound understanding of the helpful nature of evaluation.

The mere word “evaluation” has negative connotations for many people. Program directors and staff might become nervous and worried over the very mention of the word. There are several factors playing into these persons’ negative feelings about the idea of an evaluation, some of which may be:

However, evaluation can be a very positive experience for program directors and staff, much like that visit to the doctor can be positive when we see the physical exam as a way of ensuring the optimal health of our body’s program. A thorough and well-done evaluation can be very beneficial. Among other benefits, program directors and staff commonly experience the following:

Evaluation should be an integral part of the program you are planning. It should not be an afterthought: “preventative medicine” is preferred to calling on an evaluator once a program is long underway and perhaps suffering from too many pitfalls or undetected flaws. Evaluation will help you ensure from the outset and over the lifespan of your project that it achieves maximum effectiveness, continuously improving the program’s performance, and ensuring accountability to funders and other stakeholders.

What do you want to evaluate?

Before starting any evaluation you need to address two fundamental questions:

What do you want to know?
Who will use the information, and how?

What do you want to know?

Understanding the purpose of your evaluation is crucial to any good evaluation. To answer this you will need to understand a conceptual model of your program. To return to our running metaphor, it is helpful to understand a conceptual model of the human body and its functions to understand the purpose of your physical examination.

A conceptual or logic model provides a visualization of the parts of the project, the desired outcomes and the connections between all of them. A conceptual model depicts what you plan to do and what you plan to accomplish. A basic model has four areas with arrows showing relationships between them. The four areas are: project inputs, activities, short-term outcomes, and long-term outcomes.

Inputs: The various funding sources and resource streams that provide support to the project.

Activities: The services, materials, and actions that are the project’s emphasis.

Short-Term Impact: The immediate results of the activities.

Long-Term Outcome: The broader and more enduring impacts.

FIGURE 1. Logic Model Framework

The conceptual model will establish the bigger picture for the program.

To create this logic model you will need to receive feedback from a variety of stakeholders. Stakeholders are those who are impacted by the program. Conversations with stakeholders will ensure that you receive all perspectives of your program.

Based on the completed logic model you should begin to develop some questions that you want answered through the evaluation. Again you should receive feedback and input from stakeholders. This can be a long and tedious process as many different people will have many different ideas; however, it is important to narrow down this list and begin to focus on what evaluation needs best serve your program or organization.

Who will use the information and how?

Once you have some idea of what you want to know you need to ask yourself who will use the information produced by the evaluator and how it will be used. Sometimes clients ask or request questions be asked in an evaluation for the simple reason of it seeming an interesting question to ask, even if they have no actual plan for using the information. Questions of of this purely inquisitive sort should be saved for research, while questions that provide information of calculated use to the program are appropriate to evaluation.

After creating a conceptual model of your program and deciding what you want out of the evaluation you should begin to think about the evaluation questions. At this time you should secure the help of an evaluator, if you do not already have one.

Who should conduct the evaluation?

There are two types of evaluations: formative and summative.

Formative evaluations are conducted during the development, implementation, or improvement of the program. They are generally conducted for program staff with the intention of improving the program. Sometimes these evaluations are done by an internal evaluator, that is, someone within the program. However, more often than not they are conducted by an external evaluator, someone outside of the organization.

Summative evaluations are conducted at the conclusion of the program and are generally conduced for an external audience, those outside of the program, so that they can make decisions about the program. These are almost always conducted by external evaluators.

Benefits of an external evaluator

There are a great number of advantages to using an external evaluator to conduct the summative (and formative) evaluation.

The external evaluator has no vested interest in the program and is thus more likely to be impartial.
The evaluation is more likely to be trusted and believable. This is especially the case if the evaluation was controversial in any way.
The program can hire an external evaluator with the skills they need at the time they need them.
An external evaluator brings a fresh pair of eyes into the organization and has a more open mind about the program than an internal person.
Program participants and staff may trust that the information and experiences they share with the external evaluator will be held in greater confidence than they would with an internal person, giving a more accurate or truthful view of the program.

How do I select an external evaluator?

It is important to select an evaluator who is both knowledgeable in evaluation and with whom you can work closely throughout the term of your project.

There are a number of items that an ideal evaluator should be knowledgeable of:

The project’s goals and objectives
Evaluation theory and methodology (formative and summative)
Differences between research and evaluation procedures
How to plan, design, and conduct an evaluation
Statistical procedures and analyses
Database management
Report-writing
Communication strategies for internal and external stakeholders

The evaluator should be an integral member of the project team. As a result, the ability to have a close working relationship with the evaluator will have an enormous impact on the resulting evaluation and consequently the program itself.

Locating an evaluator:

Local colleges or universities
Independent contractors
Recommendations (from other program directors, etc.)

No matter how you find an evaluator, it is important to have a clear description of the project so that the evaluator can prepare an evaluation plan and a proposal. You will have to give potential evaluators information about your program. Along with the logic model they may ask you to explain your goals, objectives and potential benchmarks. For information on this please see the handout “Creating an Evaluation Framework”. You should explain to them what it means to you to accomplish your goals and the types of data and information that will be meaningful to you. Evaluators will also need to have a general idea of the evaluation budget. The scope of work an evaluator proposes can be quite large or small depending on the resources available.

If you have requested proposals from a number of evaluators then this will be a chance for you to review each evaluator’s plan, writing style, and budget to determine who fits best with your organization’s style and needs. The proposal chosen will ultimately serve as the basis for the evaluation contract.

Prior to choosing one evaluator it would also be a good idea to have an interview with each of the potential evaluators. This provides a chance for you to assess your ability to work with this person, their communication style, and provides them a chance to expand on and further explain their evaluation plan.

While most program staff and managers are not proficient in evaluation terminology and methods; it is important to have a basic understanding of evaluation methods and approaches so that you can choose an evaluator and an evaluation plan that best fits your needs.

Some things to consider when choosing an evaluator for your program:

Academic credentials/preparation – What kind of degrees does this person hold? What kind of preparation in evaluation?
Evaluation experience – How long has the person been in the evaluation field? What types of evaluations have they worked on previously?
Capacity – Are their staff and resources sufficient to meet your needs?
References – What types of evaluations have they worked on in the past? Ask to see some work samples.
Location – Are they local or do they live far away? Can you work with an evaluator who you only see once or twice a year?
Personal style – Is this someone with whom you could establish a close working relationship?

What should I know to contract with an evaluator?

Once you have chosen an evaluator you need to write an evaluation contract. There are a number of things that you should make sure are included with this contract.

Scope of Work. This describes exactly what will be done for the evaluation including a timeline and a list of deliverables. This is important for both the evaluator and the project staff/board and will ensure that everyone is on the same page with the same expectations.

Payment Schedule. In the contract you should include a statement of how the evaluator will be paid. It can be done on a regular interval, after a deliverable, or as a lump sum.
Any of the choices are fine options, but it is best to schedule payments after some deliverable (data analysis, report, presentation).

Draft Reports. The contract should ask for the evaluator to submit all items in draft report and allow program staff to review them and make any modifications prior to the final product.
Ownership of Data. It is important to outline who has ownership of the data and to whom information can be given. The contract should specify if the evaluator has any right to publish evaluation results.
Confidentiality. The contract should clarify how the evaluation will work with the issue of confidentiality.
Staff for Project. The contract should list the person (s) working directly on the evaluation. If staff needs to change then the contract should stipulate that both parties have to agree; otherwise the contract is null and void. This will protect the program board and staff from contracting with an evaluator with specific credentials, but having the work done by someone else.
Flexibility. Finally the contract should allow for some flexibility in the scope of work and the timeline. Evaluation needs and methods change with the program and what was planned originally may no longer be needed or feasible. This is especially the case with a multi-year contract. If changes do need to be made the contract should stipulate that both parties need to agree to the changes.

How do I work with an evaluator?

At this point in the process, you have chosen an evaluator and finalized a contract. Now what can you expect from the evaluator and how should you work with them?

Working with an evaluator

Regular contact is very important. Various forms of communication can be useful for maintaining regular contact. Make sure you keep your evaluator “in the know.” Any changes or adaptations to the program is important information that will impact the evaluation.

Decide early on how often you will have this contact, who will be responsible for making the contact, and what the expectations are from both sides.

Monitor the work of your evaluator. Make sure they are accomplishing the evaluation as you planned and to your liking.
Stick with the contract. Do not add extra items or time. The evaluator was contracted for a specific reason and usually for a specific number of hours. If you do need extra deliverables or the work is above and beyond what you contracted with them for then be prepared to pay extra.

Expectations

The evaluator should make all deadlines outlined in the scope of work and the contract. They should be reliable. Confront the evaluator to determine the reason for the tardiness. If it is due to a problem on your end do your best to address this. If it is a problem on the evaluator’s end you need to decide if this was a breach in contract or a one-time miss.
The evaluator should keep in constant communication with you. Just as it is important that you keep your evaluator abreast with program news it is also important that your evaluator touch base with you and let you know what is happening with the evaluation.
The evaluator should stick with the scope of work and timeline as outlined in the contract. An evaluation is a living document. There will always be some changes in the proposed evaluation based on new information, implementation issues, or numbers of participants. However, there should not be constant and ongoing changes to the evaluation plan. The evaluator should stick with the plan and encourage you and your organization to do the same, avoiding constant tinkering.
The evaluation should provide you with a wealth of information to use both during the implementation of the program and after the end of the evaluation contract. An evaluation report gathering dust on a bookshelf is a wasted opportunity; the last step in any successful evaluation is for you to use this information as a tool to guide positive changes to your program and organization