Content
The content of a questionnaire must be derived from the purpose of the survey. For example, if the goal is to compare the attractiveness of one product with another, established instruments such as AttrakDiff or AttrakDiff2 can be used (Südmeyer et al., 2007). Such instruments are already theoretically grounded and are usually empirically tested (Baur & Blasius, 2014).
Even when adapting an existing questionnaire, caution is still required. Validated questions should not be adopted uncritically. Each item must make a clear contribution to the study objective. Ambiguous or unnecessary questions increase respondents’ cognitive load and can reduce data quality (Porst, 2014).
Questionnaires can also be developed from the academic literature or through expert discussions. Both approaches aim to identify relevant constructs systematically and translate them into suitable items (Diekmann, 2018).
Deriving Items from the Literature
When deriving a questionnaire from the literature, existing studies are analyzed to determine which theoretical concepts, open research questions, or empirical relationships are relevant to the topic at hand. This analysis can be used to derive constructs, dimensions, and possible hypotheses, which are then translated into concrete questions (Baur & Blasius, 2014, Diekmann, 2018).
Expert Discussion
An expert discussion is particularly useful when a topic area is not yet sufficiently structured or when practical judgments are needed. First, suitable experts must be identified. One supervisor can provide valuable input, but usually does not replace a sufficiently robust expert group (Rea & Parker, 2015).
The goal of the discussion is to delimit the topic area precisely and identify the most relevant aspects. To do so, guiding questions, thematic blocks, and initial formulations should be prepared in advance. This helps structure the discussion and keep it aligned with the research question (Porst, 2014).
As a general rule, only content that is necessary to achieve the study objective should be included in the questionnaire. The longer a questionnaire is, the greater the risk of dropouts, fatigue effects, and incomplete responses (Rea & Parker, 2015).
Length
The length of a questionnaire depends on the study objective, the target group, and the survey mode. As a practical guideline, the completion time should ideally not exceed 15 minutes. As duration increases, willingness to participate tends to decline, especially if respondents do not perceive any direct benefit or incentive (Rea & Parker, 2015, Porst, 2014).
In practice, the rule is therefore simple: shorter and more precise is usually better than long and unwieldy. A compact questionnaire often improves both response rates and answer quality (Diekmann, 2018).
Procedure
The survey procedure is determined not only by the length of the questionnaire, but also by the timing of data collection. Certain target groups are only partially accessible at specific times, for example students during semester breaks or teachers during school holidays. Such constraints should be taken into account early on (Rea & Parker, 2015).
The survey mode also affects the process. Online surveys are usually efficient and quick to implement, but they carry the risk of unclear responses if it is not certain who actually completed the questionnaire. Postal surveys, by contrast, require additional time for mailing, delivery, and return (Diekmann, 2018).
Target Group
The target group should consist of people who are actually able to provide informed answers on the subject of the survey. Asking students about strategic business decisions may be interesting, but it is usually not suitable for drawing reliable conclusions. More appropriate respondents are those who work in the relevant context and have corresponding experience, such as executives or IT leaders like CIOs (Porst, 2014, Rea & Parker, 2015).
It should also be noted that the higher the hierarchical level, the more difficult it often is to recruit participants. Senior professionals are usually busier and respond more selectively to survey invitations (Diekmann, 2018).
Question Design
Once the objective, content, length, procedure, and target group have been defined, the actual question design begins. In general, a distinction is made between open-ended and closed-ended questions (Porst, 2014).
Open-Ended Questions
Open-ended questions allow respondents to answer freely. The response options are not predetermined. Evaluation is usually qualitative, for example through coding, categorization, or content analysis, in order to identify patterns, similarities, and differences (Baur & Blasius, 2014).
Closed-Ended Questions
With closed-ended questions, the response options are predefined. The choice of response format has a major influence on later analyzability and interpretive value. The scale level is crucial here. A distinction is made between nominal, ordinal, interval, and ratio scales (Diekmann, 2018, Porst, 2014).
Nominal scales allow only statements about equality or difference. One example is the categorical measurement of gender (Diekmann, 2018).
Ordinal scales additionally allow statements about rank order. A typical example is a Likert scale ranging from “very poor” to “very good.” The order is known, but the distance between the levels is not interpretable (Porst, 2014).
Interval scales additionally allow comparisons of differences. A classic example is temperature in degrees Celsius: the difference between 20°C and 15°C is the same as the difference between 10°C and 5°C. However, there is no true zero point, which means that ratio statements are not possible (Diekmann, 2018).
Ratio scales finally also allow statements about proportions. Income is a typical example: $4,000 is half of $8,000. Such scales have a true zero point (Porst, 2014).
Response scales should generally be balanced. A bipolar scale, such as strongly disagree to strongly agree, is usually preferable to a one-sided scale because it reduces bias (Rea & Parker, 2015).
Guidelines for Wording Questions
When wording good questions, several basic principles should be followed:
- Questions should be simple and unambiguous. Nested sentence structures should be avoided because they reduce comprehension (Porst, 2014).
- Each question should address only one issue. Double-barreled questions that combine multiple issues with and or or should be avoided. For example, instead of asking, “What is your gross income and are you satisfied with it?” two separate questions should be asked (Rea & Parker, 2015).
- The language should match the target group. If in doubt, a brief comprehension check can help, for example: “Did you experience any language difficulties? If so, which ones?” (Porst, 2014).
- The wording should fit the chosen method. “How” questions are often suitable for process-related aspects, whereas “what” questions are more appropriate for concrete facts or effects (Diekmann, 2018).
Each question should have a clear connection to the survey objective. If necessary, it must be possible to justify why it was included. Data protection must also be considered. The more personal a question is, the lower the willingness to answer may be. At the same time, the requirements for transparency, data minimization, and responsible handling of the collected information increase (Diekmann, 2018).
Before the questionnaire is deployed, a pretest should be conducted. It helps check whether the questions are understandable, estimate the completion time realistically, identify technical issues in the survey tool, and assess the target group’s general willingness to participate (Porst, 2014, Rea & Parker, 2015).
If the response rate is already low during the pretest, the questionnaire should be revised or the recruitment strategy adjusted (Diekmann, 2018).
References
- Baur, N., & Blasius, J. (Eds.). (2014). Handbook of empirical social research methods (2nd ed.). Springer VS. https://doi.org/10.1007/978-3-531-18939-0
- Diekmann, A. (2018). Empirical social research: Basic principles, methods, and applications (12th ed.). Rowohlt.
- Porst, R. (2014). Questionnaires: A workbook (4th ed.). Springer VS. https://doi.org/10.1007/978-3-658-01269-8
- Rea, L. M., & Parker, R. A. (2015). Designing and conducting survey research: A comprehensive guide (4th ed.). Jossey-Bass.
- Südmeyer, A., Hassenzahl, M., & Diefenbach, S. (2007). AttrakDiff: A questionnaire for measuring the hedonic and pragmatic quality of interactive products. In Mensch & Computer 2007. Oldenbourg.