Assessing Health Outcomes:
Part 4, Writing Health Survey Questions
Writing Health Survey Questions
A major flaw of most surveys for assessing health outcomes is a lack of the appropriate level of specificity in defining exactly what is being measured. This can lead to the use of instruments that are vague or confusing to both survey administrators and to respondents. The resulting data can be hard to analyze and ultimately a significant amount of time and resources are wasted when the final survey results are of minimal use to the organization.
Once a survey design team has a clear statement of purpose, they must write data collection questions that will allow them to gather accurate information that reflects the assessment domains. The most difficult aspect of this process results from the fact that very few assessment domains are concretely measurable. For example, while member weight or blood pressure can be measured objectively with little difficulty, assessment domains like “health literacy” cannot be measured so easily. In addition, any given area to be assessed may have any number of dimensions (discussed in Part 2 of this series). This entry focuses on the process of writing health survey questions by which survey designers prepare to accurately measure these abstract and complex assessment domains. These processes are the elaboration of conceptual and operational definitions.
A conceptual definition is very important for the accurate measurement of an abstract assessment domain. Conceptualization is the process by which the survey design team clearly defines what their assessment domain means to the healthcare organization. For example, to measure the success of a smoking cessation program a health care organization must know exactly what they mean by smoking cessation. Does it simply mean, “Has the member quit smoking?” It could also mean, “Has the member reduced his or her smoking?” Or it could even mean, “Has the member decided to quit smoking?” Of course, a very complete conceptual definition of smoking cessation might include them all. Depending on the complexity of the definition of smoking cessation one uses, very different data collection collections are necessary.
Once a conceptual definition has been developed, the next step for the outcomes design team is to create an operational definition. Operationalization is the process by which researchers define exactly what indicates the presence or absence of the various elements of the conceptual definition they have created. For example, the survey designers can operationalize outcomes related to smoking cessation in a variety of ways. What data can be collected to actually show how much smoking cessation has occurred for the health care organization’s members? One survey design team may operationalize “quit smoking” as “the member has not smoked a single cigarette in at least 3 months,” while another may choose to operationalize it as “the member has not smoked a single cigarette in at least 6 months.”
So, what are the consequences of poor conceptual and operational definitions? If a survey design team simply writes a question to measure smoking cessation that says, “Since enrolling in our smoking cessation program have you quit smoking?” the member is left to choose what it means to quit. Some members who have reduced their smoking significantly but still smoke on rare occasions could answer “yes” to this while other members who have not had any cigarettes for two weeks may still answer “no” because they are not yet confident in the success of their cessation. They can ultimately compromise the validity of the survey. Essentially the health survey questions may not actually be measuring what they seek to. Proper conceptual and operational definitions are a major step to guaranteeing a valid survey that can provide reportable results.
Read more on this topic:
Aday, Lu Ann and Llewellyn J. Cornelius, 2006. “Designing and Conducting Health Surveys: A Comprehensive Guide,” Jossey-Bass: San Francisco, CA.