Evaluation is the art of asking questions. Questions to determine the needs for our program, questions that examine the program’s role in creating change, questions to describe the change or impact made by our program and questions to prompt action as a result of what the evaluation uncovers.

Put simply, evaluation gives us the framework and tools to understand the value of our program process and outcomes. Often the primary role of evaluation is seen as the method by which we demonstrate that we have accomplished what we said we would accomplish. This requires that the dimensions of performance are examined to determine the extent to which expected (and unexpected) results are being achieved. Having an evaluation framework in place allows organisations to embed continuous learning into systems and process. Evaluation can have further-reaching impact as well. Evaluation can provide a voice to the community about the type of service or community they want. If done well, it can make people feel valued and can empower them to be part of the decision-making process.

Why do we need to evaluate?

There are many possible reasons to evaluate a program, including to:
»» monitor whether a program is meeting its goals and objectives
»» monitor whether a program is being implemented as planned
»» to understand how a program works to achieve change
»» to learn and make improvements to the program
»» to ensure it is efficient use of public expenditure
»» to ensure the program is appropriate for the participants

In the situation where there is high need or demand for a service, there can be a dilemma between putting resources towards program delivery or evaluation. However, there are some situations where evaluation is critical:
»» where an existing program is being applied in a new context or with a new population
»» where a program is being piloted and has not been evaluated before
»» where the program needs to ensure ethics criteria is met and the program does no harm
»» where it would be unethical to continue a program that is not effective, such as where the program is dealing with high risk or vulnerable groups.

Funders of family and social services throughout Australia are actively encouraging a move toward outcomes-based accountability. In addition to monitoring program effectiveness and improving outcomes for clients, evaluation activities give an organisation competitive edge by successfully telling their program story. It also provides a mechanism for feedback and accountability to funding bodies, and helps organisations develop a business case for further funding.

When should evaluation happen?

Program evaluation can occur at any stage of the program rollout. Ideally, monitoring, evaluation and learning plans should be developed during the initial stages of program design and planning. The table below shows why evaluation might occur at different stages, and examples of questions it might seek to answer.

Evaluation approaches

There are many different approaches to evaluation depending on the purpose of the evaluation. The approaches do not operate in isolation and there might be a combination of approaches applied. The methods that are then chosen will align to the overall approach, for example if the approach is objectives-based a program logic may be the chosen method to guide the evaluation.
This list provided some examples, but is not exhaustive.

Evaluation planning

An evaluation plan is used to design the evaluation and guide the implementation. The plan will outline why and how evaluation will occur in the organisation, including ethical issues and dissemination of information.
There are a few key differences between evaluation plans and frameworks.
Evaluation Plan:
»» For the evaluation of a specific program or project
»» Includes more detail for the implementation of a specific evaluation

Evaluation Framework:
»» For multiple evaluations and usually different forms of evaluations across a program
»» Outlines a general approach and measures to evaluate and aggregate results across multiple projects or time scales

Whilst there are key differences, an evaluation plan and an evaluation framework often include the same sorts of information, except that the scale and level of detail in each will be different.

Evaluation planning steps

When planning an evaluation it is useful to have some basic steps to follow. The Wissemann six steps for evaluation planning structure the evaluation planning process to make sure all of the important elements are covered. These steps include:
1. Clarifying the evaluation request and what is being evaluated
2. Focusing the evaluation
3. Determine information needs
4. Data Collection
5. Analysis and Reporting
6. Sequencing the evaluation activities

These steps can be applied to both evaluation plans and evaluation frameworks and set up the backbone for the documented plan. If the intention is to commission an external evaluator, the planning steps can be used to inform the brief sent to consultants. For example:

Writing a brief checklist

At CFRE we believe the best way to work with agencies or programs is to take a collaborative approach. We work with agencies in partnership to tailor any training or evaluation services to meet their needs and to build lasting capacity so that they can use these skills to enhance their services in future.

For more information about how we can work with your team, contact cfre@ds.org.au