Step 3: Identifying evaluation purposes and formulating evaluation questions

In the third step of the evaluation process, the evaluation purposes are identified and evaluation questions are formulated in a way that enables the collection of the information needed to achieve the evaluation purposes.

Apfelbaum mit Most, Geld und Äpfel

Evaluation purposes can be categorised according to function and intended use.

Five evaluation functions can be distinguished:

  • A proactive evaluation is intended to determine, to what extend the object of evaluation is actually necessary and to what extend it can be implemented using existing resources within the given timeframe.
  • A clarificative evaluation is intended to ascertain to what extend the concept used for the object of evaluation is compelling and coherent with identified needs and requirements on the basis of available data.
  • An interactive evaluation is intended to determine the strengths and weaknesses of a given object of evaluation and identify any range for optimisation.
  • A monitoring evaluation focuses on the standardised collection of data at regular intervals over an extended period of time to monitor key performance indicators.
  • An impact evaluation is intended to determine whether the goals established for the object of evaluation have been reached and, if applicable, how effective the object of evaluation has been in achieving the established goals.

While proactive and clarificative evaluations are carried out before the possible start of the object of evaluation, interactive evaluations take place while the object of evaluation is ongoing, especially in the development phase when the object of evaluation has just started for the first time or is undergoing a redesign phase. Monitoring and impact evaluations, on the other hand, can only be carried out if the object of evaluation has reached sufficient maturity.

When classifying evaluations according to intended use, a distinction is drawn between a focus on improvement (i.e. creating an information basis to enable optimisation or stabilisation), accountability (i.e. providing evidence of quality to persons/institutions responsible), fundamental decisions (i.e. collecting data to enable a decision to be made in favour or against execution/continuation of an object of evaluation) and knowledge creation (i.e. generating new, generally valid knowledge).

There are two evaluation roles where functions and intended uses are combined:

  • Formative evaluations tend to connect interactive or clarificative functions and are focussed on improving an object of evaluation.
  • Summative evaluations tend to connect proactive, monitoring or impact functions and are focussed on providing accountability or guiding fundamental decisions.

There are also hybrid forms between these two evaluation roles. However, in the interests of ensuring the utility and fairness of evaluations, clear priority should be given to at least one of the two roles. For longer-term evaluations, an initial formative evaluation can be followed by a second summative evaluation after the object of evaluation has reached maturity.

In some cases, evaluators may even be confronted with hidden evaluation purposes (e.g. marketing aims or legitimising a decision already made). Hidden evaluation purposes are particularly difficult to identify and clarify.

When developing evaluation questions, the aim is to determine what information the intended users require in order to satisfy the evaluation purpose. This is usually decided in discussions between evaluators, clients and relevant stakeholders.

When formulating evaluation questions, it is useful to recall that good evaluation questions are:

  • ‘focused’ (on a specific part of the object of evaluation)
  • ‘open’ (seeking to answer to what extent something is present so as to avoid yes-no answers)
  • ‘realistic’ (related to facts associated with or possibly caused by the object of evaluation)
  • ‘clear’ (formulated in an understandable way)
  • ‘empirical’ (may be answered on the basis of data)
  • ‘resource-friendly’ (may be answered using available resources)
  • ‘useful’ (relating to the informational interest of stakeholders)

There are three different types of evaluation questions:

  • Descriptive questions are intended to shed light on the conditions and characteristics of the object of evaluation.
  • Evaluative questions are intended to determine the extent to which the object of evaluation satisfied previously established criteria.
  • Causal questions are intended to determine the extent to which changes or stabilisations may be attributed to the object of evaluation.

Because different actors often have different interests with regard to the apple as an object of evaluation, an evaluation can also serve different informational interests and thus be conducted for different purposes. While processors, traders (wholesalers, retailers, the farmers who sell the apples directly) and end customers are mainly interested in quality control and accountability, it is often more important for producers and apple tree nurseries to know how they can optimise cultivation or breeding of the apple as the resulting end product. This results in very different evaluation questions, such as:

  • What is the relationship between the variety of apples produced and the environment in which they are grown?
  • What is their relationship to overall aspects such as the environment and resource consumption?
  • How can the cost of apple cultivation be optimised?
  • To what extent do apples satisfy the needs of consumers?
  • How well suited are the apples for further processing (e.g. apple juice, apple pie)?