Program evaluation in academic projects

Ross Woods, 2022

When theses and dissertations have projects, the last stage of the project is an evaluation. In other dissertations, the whole dissertation is an evaluation of a complex program.

Churches often evaluate their programs with a discussion in which people say whether or not they liked it and then make a decision. However, better information and more analysis will produce better conclusions.

In some ways, evaluating a program is like making soup. You get a full range of ingredients, mix them together, and see how good the soup tastes. The ingredients are the different kinds of information and the taste-test is the analysis and conclusions.

A program evaluation typically comprises the stages below, and in a thesis or dissertation are usually put in specific chapters:

1. Define the program, done in the introduction chapter
2. Plan your methods for collecting information (data), done in the methodology chapter
3. Collect information, done in the methodology chapter
4. Interpret and analyze information, done in the analysis chapter
5. Draw conclusions, done in the analysis chapter.

Define the program

Defining the program is usually fairly easy. It needs to include aspects such as the following:

• What are its goals and objectives?
• What activities will it use to meet those goals and objectives?
• What is the rationale for the program? Why is it necessary? How does it work?
• Who are the stakeholders? A stakeholder is anybody who depends in some way on the success of the program, such as the participants, the leaders, the administrators, and the funders. Their relationship to the program differs, with some having more at stake than others.
• How will it end? Some programs are closed down when they end, perhaps with an option of re-commencing if the evaluation is favorable. Other programs are intended to continue and the student’s role is to bring them to a stage of maturity that give hope of a bright future. In fact to close them down is to disappoint the participants and to fail.

Methods

Choose a set of methods that is appropriate to your particular program. Do not try to use all possible methods. When you can, use information that already exists, such as documents. Generally speaking, information is of two different kinds: the views of stakeholders, and “harder” more objective information.

Stakeholder views and perceptions

You have a choice of various methods to gather views, for example:

• You can hold interviews with individual stakeholders or focus groups with groups of stakeholders.
• You can hand out feedback forms. Be careful; the comments might tend to be either overly positive or overly negative. For some respondents, the forms simply ask, “Are you happy?” while others see the form as an opportunity to complain and criticize.
• Stakeholders sometimes make comments in casual conversation that you should follow up. These comments are valuable because they might represent the views of many other stakeholders. They might also contain a helpful but counter-intuitive, unexpected insight.
• Another kind of feedback is complaints as long as you have recorded them. Very big programs often routinely receive complaints and see them positively as a way to improve; they create an “Issues register” so that they can plan strategically for solutions and improvements.

Whichever method you choose, don’t just ask the questions, but also explore further to find people’s reasons for their answers. Just knowing “what” is not very helpful without knowing “why.”

“Objective” information

You have a choice of methods to gather more objective information, for example:

• The simplest measure for many programs is attendance. The program has probably succeeded if lots of people attend and stay the course. Likewise, it has probably failed if nobody shows up or if most people drop out.
• Another simple measure is financial. If it goes broke or runs at a major loss, it has probably failed.
• Did the program meet its original aims and objectives? You can use these as a standard to evaluate the program.
• You can use external standards to evaluate the program if any are applicable. For example, government and accreditors often have standards for educational programs.
• You can examine any existing documents. What does the program paperwork tell you? Many organizations do internal reviews and audits. (This option might have limited use to you if you created the program and its documentation.)
• Some industries keep track of defective products and refunds. This information is like complaints and are useful for creating an “Issues register” for planning solutions and improvements.

Interpreting and analysing information

If you have collected enough information, it will eventually form a pattern and you will have enough information to confirm that pattern.

If the dissertation is a professional project, the conclusion will related to the original research question or hypothesis. If the dissertation is original research, the conclusion will relate to original research question or hypothesis.

Interpreting stakeholder views and perceptions

Respondents’ feedback falls into three indistinct categories, which tend to follow a normal curve. Some are not useful, for example, some people don't fill out the feedback forms or give unhelpful comments, and some people who don’t want to speak in an interview or a focus group. Some are of only moderate insight, but their value is that they represent the majority of respondents. Some reflect excellent insight, but they represent only a minority of respondents.

Interpreting “objective” information

Some supposedly “objective” information is not completely free of personal bias. For example, documents normally reflect the biases of their writers. Internal reviews reflect personal judgments and can contain errors and omissions. Moreover, none of these kinds of information are definitive of success. Consider this:

• Attendance numbers don’t indicate the level of commitment of attenders. A small group of committed people might be much more effective in achieving program goals than a large group of less committed people. Numbers alone mean little without evidence of quality.
• A financial criterion might not be definitive. A highly effective program that loses money might be worth giving extra funding.
• Were the original aims and objectives appropriate? You also need to evaluate these, and they might have been inadequate or inappropriate in some way. In fact, program leaders often need to adjust them is some way while the program is running.

Risks

During a program evaluation, you should identify risks, that is, aspects of the program that could easily go wrong in future even if they have not done so.

Comparing information

Next, compare the different kinds of information. For example, you might find that stakeholders’ behavior is inconsistent with their opinions. For example:

• Person A says, “Loved the course. It was excellent.” His behavior, however, was that he dropped out after two weeks.
• Person B stays for the whole course but does not put any of the lessons into practice.

“Politics” and conflict of interest

In your analysis, you need to consider the role of “politics” and conflict of interest. The definition of stakeholder already means that none of them is neutral. Some of them also have much more political power in the organization than others. Be careful to protect those with little power, who are vunerable to becoming victims.

Various factors put pressure on you to get particular evaluation conclusions. The questions are:

• What will someone gain or lose by making positive comments about the program?
• What will someone gain or lose by making negative comments about the program?

Stakeholders, including you, can easily rationalize their views of the program. For example:

• Student A runs a program and then evaluates it. He feels, “I’ll fail the thesis program if I make negative conclusions about the program, so I’ll draw try to make it look good.”
• Student B evaluates a large program, and feels, “Its a very good program, but I’ll fail the dissertation if I can’t find enough things wrong with it, so I’ll look for more faults.”
• Student C evaluates a program, and feels, “It’s a very weak program, but it will lose funding if I say that, so I’ll try to make it look good.”

In the face of these pressures, you need to be as neutral and objective as possible. If your thesis/dissertation assessors percieve your analysis to be unjustified and biased, they will most likely return your thesis or dissertation to you for major corrections.

💡 Tip 1. You probably won’t be able to please everybody.

💡 Tip 2. Observe the group dynamics.

Conclusions

The result of a program evaluation is seldom “Good” or “Bad.” It is more likely to indicate the program’s strengths, weaknesses, and recommendations for improvements. In a thesis or dissertation, it will often include a list of lessons learned. These are most valuable when they could not have been anticipated during the planning phase. If the dissertation is original research, the conclusion will relate to the original research question or hypothesis.

Something unexpected

It is possible that a program successfully achieves its goals, but people do not like it. This usually means something is wrong, but consider the following example:

Small Church planned to grow, and it did, even though it planted daughter churches. However, some members were dissatified when the church was too big for them to know everyone else in the church, so they decided to move to other churches that were smaller.