Step 5: Selecting data collection design and methods

In the fifth step of the evaluation process, evaluators plan the concrete methodological procedure and implement it. For this intention, it must first be decided whether new data must be collected or whether the existing data basis is sufficient to answer the evaluation questions. If (additional) data collection is necessary, a data collection plan is drawn up to specify in writing the type of data to be collected, the data collection method to be used, the target group and timeframe.

Frau an Marktstand

Depending on the evaluation questions, time constraints, organisational aspects, data protection requirements, access to the field, sampling possibilities and financial resources, different design decisions have to be carefully weighed up:

  • If, for example, achievement of goals or changes will affect a certain group of people, it may be useful to carry out several successive data collections on the same target group, e.g. an initial data collection before the start of the object of evaluation and follow-up data collection during, upon completion or even at a subsequent stage.
  • If the aim is to demonstrate effects beyond this, i.e. to prove that it was really the object of evaluation that led to achievement of the goals, it may be necessary to collect data among groups of persons who did not take part in the object of evaluation.

Data collection designs where data are obtained from multiple groups at multiple points in time involve a great deal of effort and require precise planning and considerable resources.

The data collection method chosen should also be based on the evaluation questions and underlying conditions and also be closely linked to design decisions. In principle, the repertoire of methods in the social sciences may be categorised into three main groups:

  1. Use of existing data (behaviour traces) or process data.
  2. Systematic observation, covert or participating, especially of verbal and non-verbal interactions.
  3. Oral or written questioning; these can take place face-to-face, on the telephone or via computer, whereby the specific questions can be asked in a more or less open or structured and standardised manner.

Data collection methods can be used to collect or generate both qualitative and quantitative data. Since each data collection method has different strengths and weaknesses, each one is differently suited to the evaluation purpose, the evaluation questions and the evaluation budget, etc. In every concrete evaluation, the aim is to find the most appropriate methodological approach. It is therefore essential that evaluators or evaluation teams have a broad repertoire of methods at their disposal.

In this step of the evaluation process, it may be necessary or advantageous to involve the stakeholders in methodological decisions. More than in the other steps, however, the responsibility here lies with the evaluators, as they have expertise and extensive experience with data collection methods and tools.

In order to obtain the information required to answer an evaluation question, we first have to determine what data should be collected, what data collection method(s) should be used, whom we should obtain the data from and when. For example, a stand could be set up in a pedestrian zone where these apples could be tasted and consumers could be asked to answer the evaluator's questions about the apple. Other data collection methods (e.g. interviews with sellers or producers) could be used to gather information about the regional origin of the apples and the cultivation methods used. Economic analyses could be carried out to estimate the cost-benefit ratio of apple production.