We hope this guide helps practitioners and others strengthen programs designed to increase academic achievement, ultimately broadening access to higher education for youth and adults.
We believe that evaluation is a critical part of program design and is necessary for ongoing program improvement. Evaluation requires collecting reliable, current and compelling information to empower stakeholders to make better decisions about programs and organizational practices that directly affect students. A good evaluation is an effective way of gathering information that strengthens programs, identifies problems, and assesses the extent of change over time. A sound evaluation that prompts program improvement is also a positive sign to funders and other stakeholders, and can help to sustain their commitment to your program.
Theories of change are conceptual maps that show how and why program activities will achieve short-term, interim, and long-term outcomes. The underlying assumptions that promote, support, and sustain a program often seem self-evident to program planners. Consequently, they spend too little time clarifying those assumptions for implementers and participants. Explicit theories of change provoke continuous reflection and shared ownership of the work to be accomplished. Even the most experienced program planners sometimes make the mistake of thinking an innovative design will accomplish goals without checking the linkages among assumptions and plans.
Developing a theory of change is a team effort. The collective knowledge and experience of program staff, stakeholders, and participants contribute to formulating a clear, precise statement about how and why a program will work. Using a theory-based approach, program collaborators state what they are doing and why by working backwards from the outcomes they seek to the interventions they plan, and forward from interventions to desired outcomes. When defining a theory of change, program planners usually begin by deciding expected outcomes, aligning outcomes with goals, deciding on the best indicators to evaluate progress toward desired outcomes, and developing specific measures for evaluating results. The end product is a statement of the expected change that specifies how implementation, resources, and evaluation translate into desired outcomes.
Continuously evaluating a theory of change encourages program planners to keep an eye on their goals. Statements about how and why a program will work must be established using the knowledge of program staff, stakeholders, and participants. This statement represents the theory underlying the program plan and shows planners how resources and activities translate to desired improvements and outcomes. It also becomes a framework for program implementation and evaluation.
Conducting a needs assessment is often the first step in developing a program. This can be done through surveys, focus group discussions, observations, and interviews to identify problems and concerns in a community or segment of the population. A typical needs assessment also describes resources that are available to solve the problem, and helps stakeholders make informed decisions about program goals and what is required to accomplish them. After defining needs, stakeholders design a program to address those needs and clarify goals. Goals are the desired outcomes of a program — the end toward which a program or set of activities is directed.
Evaluators typically use at least three types of evaluation strategies for monitoring program implementation: monitoring, implementation, and outcome evaluations. A fourth type, impact evaluation, is typically undertaken for in-depth and long-term programs.
1. Monitoring evaluation is usually conducted internally by program staff to strengthen program design and management by measuring how accurately a plan has been executed; how well program resources have been invested; and whether adjustments in investments, resource allocations, or activities are necessary. Monitoring evaluations are conducted once or twice a year and answer the question: Is the intervention implemented consistently with the program’s design and goals? If not, why?
2. Implementation evaluation (sometimes called process evaluation) examines the implementation process, e.g., whether activities have served the expected number of people and provided an appropriate amount and quality of services. It explores how activities, programs or groups of programs have been conducted. This type of evaluation is often ongoing, enabling program staff to see the results of activities against process indicators and determine if the program is progressing towards its indirect goals.
3. Outcome evaluation focuses on the program’s impact on the participants. Ideally, outcome evaluations occur annually to answer the question: Is the intervention achieving its intended effect? An outcome evaluation might look at whether completing a college preparatory program encourages more students to enroll in four-year universities. Outcome evaluations are often conducted by an outside evaluator and result in lessons learned that can be generalized beyond program participants.
4. Impact evaluation answers the question: What outcomes did participants achieve in this program over and above what would have occurred had the program not intervened? This type of evaluation usually uses formal scientific methods including control groups, sampling strategies, and statistical procedures to clarify the role of various program features and generalize the value of the program beyond its immediate impact.
Outcomes are the valued result of programs. There are many ways to measure them and different kinds of information to gather about them. Sometimes, however, it takes a long time to achieve ambitious outcomes. In those cases, it may be important to define and measure both early and interim outcomes, while giving long-term outcomes/results more time to root. Measuring incremental changes can show if a program is on target and whether the changes occurring are the desired ones. Interim outcomes also indicate the validity of a program’s theory of change and its relationship to goals, activities, and benchmarks. The most useful evaluations gather both quantitative and qualitative data. Both contribute to understanding program implementation and its effects. Evaluators use different strategies for collecting, analyzing, and reporting different types of data.
Quantitative data is information that can be counted and measured, i.e., test scores, number of people who participate in a program, enrollment levels, and graduation rates. Qualitative data are descriptive and interpretive. They include words, pictures, graphs, and designs that describe how the program works and affects participants. This type of information is typically gathered through interviews, focus groups, observations, and surveys with open-ended responses.
Depending on what resources are available and the type of evaluation being planned, you may choose to use an internal evaluator or an external evaluator. An internal evaluator is a member of the program staff or an individual hired expressly to oversee the evaluation process. This person may be assigned to work exclusively on evaluation or may have other program responsibilities as well. An advantage to this type of evaluator is that she or he may be more familiar with the staff and program activities, have greater access to program resources, and wider opportunities for ongoing evaluation activities. An external evaluator is a person or organization contracted from outside the program. External evaluators may have greater access to outside resources. They may also have more expertise and experience in evaluation.
The kind of evaluation expertise best suited to a particular program will vary depending on the type of evaluation to be conducted, internal expertise, and resources. For instance, a program that relies primarily on surveys and participant interviews for evaluation data might prefer to hire an evaluator with experience designing and analyzing surveys and interview protocols. If the evaluation will focus on student progress as measured by test scores, an evaluator with strong quantitative skills and knowledge about student assessment will be most helpful.
Before beginning an evaluation, there should be consensus about why it is occurring, what will be evaluated, and what all stakeholders and participants expect to learn from it. Evaluation proposals, designs, and memoranda of understanding are useful devices for communicating and clarifying these issues.
Stakeholders typically have an investment in program evaluation, making communication an essential ingredient in the evaluation plan. Inviting stakeholders, including grantee staff members, community advisors, and key foundation staff to contribute to the design of an evaluation insures it will address everyone’s questions. The same questions and data collection strategy can satisfy varied purposes and audiences. Participation by stakeholders strengthens data collection, self-assessment and information management — all of which contributes to a stable, sustainable program.
A number of considerations go into deciding on the best evaluation strategy. Planners need to know if there are adequate financial and human resources available to accomplish the task. They should also consider how they will use the information to be collected and determine if the burden placed on participants or other stakeholders is appropriate. A carefully planned evaluation strategy should inform program staff and funders, as well as promote program improvement without overwhelming either participants or program staff.
A program evaluation strategy usually involves a variety of data collection methods and sources. When designing your evaluation plan, consider whether the techniques you plan to use are appropriate for your program population. If participants are not fluent in English, for example, a written survey will not be the best way of collecting useful information. A focus group might be a better method.
Clarity of language is essential in any data collection instrument. Make sure the instructions and questions asked are easily understood. It is helpful to pilot test a survey or interview on a subset of participants to determine if the questions asked generate accurate and complete information.
The outcomes to be evaluated and the purpose of the evaluation will determine the target population. Evaluation planners should consider who will benefit from the program, then decide who or what to evaluate. Will all participants be asked to provide information about the program at some point? Will supportive stakeholders be invited to contribute to the evaluation? Will staff, participants, or some combination of both be interviewed or assessed, or will only a sample be involved? Will results from participants and non-participants be compared?
Evaluating a program usually involves using a variety of data collection methods and sources. Data collection methods should be customized to the program and the population it serves. If a program is implemented for the first time, for example, it is proper for implementation concerns and indirect or early outcomes, rather than direct or long-term outcomes, to dominate start-up. If participants have low literacy levels or speak little or no English, conducting a focus group in participants’ native language or asking a native-language speaker to translate will surely generate more accurate information than administering a written survey.
Data sources can be grouped into several categories: 1) documents and administrative records; 2) focus groups and interviews; 3) surveys; 4) observations; and 5) assessments of student knowledge. A combination of these sources usually provides the range of information that a program team and other stakeholders need.
Grantees and program staffs are sometimes surprised to learn they do not have to collect all new data for an evaluation. Much of the information needed may be available in existing documents and administrative records. The following types of records can inform an evaluation.
Routine program documents often detail a program’s processes and outcomes, indicating how a program changes over time. Relevant documents include mission statements, budgets, participant records, flyers, curriculum guides, schedules, staff guidelines, and annual reports. Examining documents and administrative records may also signal important questions for evaluators to ask in interviews or surveys. Administrative records, e.g., lists of participants, activity descriptions, budgets, and attendance records contain a wealth of quantitative information. They are especially useful in collecting information about program characteristics or events that evaluators cannot readily observe like the planning that takes place when the evaluator is not on site, or the effects of policy decisions on program changes. These records may come from multiple sources including the program and its agency collaborators or clients.
Before beginning an evaluation, it is a good idea to decide what records you may need and allow time to negotiate access to them. Trying to obtain records at the last moment without the prior approval of administrators or clients can cause delays that will threaten the execution of even the best evaluation plan.
A focus group is usually composed of 6 to 8 people with similar roles (e.g., teachers, students, parents) who meet to discuss topics posed by a facilitator. The group meets for one or two hours. One advantage to this method is that the group interaction may prompt people to think about things they might not have thought of on their own. A disadvantage is that some people may be uncomfortable expressing their opinions publicly or may not want to contradict what someone else says or share what they consider to be sensitive information. Giving individuals an opportunity to speak privately with the facilitator can offset this disadvantage.
Surveys are a direct way of measuring program outcomes. A survey can elicit data through open-ended questions, closed-ended questions, or a mix of the two. Closed-ended questions ask respondents to choose the most appropriate answer from a multiple-choice list. While closed-ended questions can be easier to analyze statistically, the pre-determined answer choices may not reveal subtleties in respondents’ opinions. Open-ended questions give participants more flexibility in sharing their thoughts about a program. Consequently, they make analysis more complex and take more time for respondents to complete.
Observation is another useful form of data collection, giving evaluators contextual information about a program and insights into its strengths and weaknesses. Observation can be a first step to designing relevant focus group questions, interview guides, and surveys. It also provides an opportunity to observe the program setting, activities, and interactions between staff and participants. A checklist of interactions and activities is a useful method of structuring observation.
Unobtrusive measures consist of systematic, but not obvious, observations of program activities and interviews with participants. They use no formal surveys or questionnaires. For example, they rely on periodic documentation of classroom activities; evidence of the need to reorder project materials and books; the thickness or number of pages completed in journals; or changes in the number of participants who use the library or log on to a web site. It is important to note for ethical reasons, however, that program staff and participants must be aware that an evaluation is in effect, even if they agree to not knowing specifically when data are being collected.
Data is easier to collect, more usable, and easier to analyze when evaluators plan and pilot test their instruments, sampling procedures, sources, and collection procedures. In any data collection instrument, the clarity of the instructions and questions will affect the quality of the information gleaned. It is helpful to pilot test a survey or interview protocol on a subset of participants before using it on a large group to get feedback on whether the questions are clear and relevant. Evaluators should have confidence that data collection methods and tools will produce the needed information before using them in a formal evaluation.
At this point, a decision should be made about sampling procedures. Systematically sampling a small number of participants or clients can reduce the data collection burden and expense associated with an evaluation. Sometimes evaluators want to compare data on a program’s results with data reported by a “comparison group” to see if there are systematic effects on participants. In order to draw valid conclusions, comparison groups should be similar to the population being studied, although obtaining records from groups that did not benefit from a program’s services may prove difficult. Additionally, using a comparison group can clarify and validate findings, but it adds complexity and expense to the evaluation process.
Response rate is a critical factor in the validity of data collection. Information needs to be obtained from an adequate proportion of clients or participants. Fifty percent or more of an initial sample should respond to the data collection. The ideal is 80 percent, but it is unusual to achieve this level of return. Experts consider a higher response rate from a small sample more valid than a large number of returns from a non-representative population. Response rates lend credence to findings and should always be included when reporting evaluation findings.
An evaluator has an ethical responsibility to assure confidentiality to anyone who provides information for an evaluation. Promising confidentiality also increases the likelihood that participants will refrain from giving what they think is the “right answer,” making the information collected more accurate and unbiased. In addition to these precautions, evaluators may be requested to obtain permission from program participants before they begin to gather data from them. Student test scores, grades and attendance records cannot legally be collected without explicit permission from the students, or in the case of youth under 18, from their parents. Once data has been collected, the next step is to summarize and analyze the findings in a report to program officials and other stakeholders. Evaluations do not necessarily have to be statistically complex analyses to be effective. Simple approaches often work just as well.
Do not over-generalize results. When interpreting outcome data, it is a mistake to assume the program is responsible for all of the positive changes. Many factors contribute to change and success in an education environment. It is important to acknowledge the possibility of other factors impinging on results.
Non-programmatic factors can influence data too. For example, did only the most involved participants respond to a survey? What happened to non-respondents? If an evaluation only tracks participants who succeeded in the program and stayed in touch, what can be learned about those who failed to complete the program? And what program changes could be made to insure a higher completion rate?
Finally, consider qualitative and quantitative results together to suggest what grantees can do to improve a program. Effective evaluation assesses whether a program is successful in achieving its goals. It identifies effective practices and informs future planning, including strategies for program improvement.
The evaluation process is central to improving programs by indicating what activities should be added, eliminated or modified. Evaluation can also be used as an advocacy tool. Sharing information gleaned from an evaluation with all stakeholders can be a springboard for feedback and strategic action, convincing audiences of the value and need for a project, and ultimately generating greater support for a program.
The most useful evaluations target their results to specific audiences: program staff, foundation boards or trustees, managers, stakeholders, grantees who direct similar or complementary programs, researchers or scientists, and policymakers and other practitioners. Audiences for evaluations have proliferated in the nonprofit world where there is great interest in disseminating lessons learned.
Megan Wilson is a teacher, life strategist, successful entrepreneur, inspirational keynote speaker and founder of https://Ebookscheaper.com. Megan champions a radical rethink of our school systems; she calls on educators to teach both intuition and logic to cultivate creativity and create bold thinkers.