Year of Publication
2008
College
Martin School of Public Policy and Administration
Date Available
8-27-2014
Abstract
Questionnaires can be useful tools for evaluating program outcomes in human services programs, in that they provide quick, easy-, and inexpensive-to-gather information about the program's success in meeting its objectives. Questionnaires may be easy to use, but they are not necessarily easy to design. If developed using sound methodology, the data gathered from questionnaires can be valid and of use to administrators, board members, direct-service staff, and funders in improving a program's services without a more resource-intensive program evaluation.
An exit questionnaire was developed to evaluate the Bluegrass Domestic Violence Program's (BDVP) achievement of short-term outcomes. Short-term outcomes for the BDVP were defined using Kentucky Victim Service Standards mandated service provision guidelines for Kentucky Domestic Violence Shelters, as well as through conversations with the Bluegrass Domestic Violence Program's Executive Director, Darlene Thomas. The questionnaire was then pre-tested using cognitive interviewing techniques.
Results of cognitive interviews revealed several classes of problems with the original questionnaire, including lexical, temporal, and computational problems with questionnaire items. As well, specific problems emerged in the single interview where the respondent was primarily Spanish-speaking and an interpreter was used.
Based on these findings, the following recommendations are made:
- Change the wording of questions to more explicitly elicit the information about short-term outcomes desired, based on interview responses.
- Determine how important it is to know the specific time frames in which some services were provided. Several of the questions involved very specific time frames (e.g. “within the first 2 days of arriving at shelter…”), and respondents found these questions almost impossible to answer as originally written.
- Have the questionnaire document translated into other languages needed before engaging in cognitive testing, since this is the form it will be administered in once it is ready for the field.
- Explore other structural issues that might weaken the instrument. Particular attention should be paid to question sequencing and methods for administering, collecting, and analyzing the questionnaire data.
- Format the instrument to reflect current changes in programming, and complete additional iterations of the pre-testing exercise.
This study has several limitations, though it provides assessments of the existing exit questionnaire that can produce a stronger questionnaire instrument and that may be more likely to capture how well the BDVP meets its short-term outcomes.
Recommended Citation
Walden, Andrea, "Developing an Outcome Evaluation Instrument for the Bluegrass Domestic Violence Program: The Use of Cognitive Interviews in Questionnaire Development" (2008). MPA/MPP/MPFM Capstone Projects. 163.
https://uknowledge.uky.edu/mpampp_etds/163
