Sexually Transmitted Diseases  >  Program Guidelines  >  Program Evaluation

Program Operations Guidelines for STD Prevention
Program Evaluation

Previous Page

Table of Contents

Next Page

Sections on this page:

• PRACTICAL CONSIDERATIONS
• Overcoming Barriers
• Resources
• Ethics
• Standards to Consider in a Practical Evaluation

PRACTICAL CONSIDERATIONS

Overcoming Barriers

The different motivations and goals of stakeholders can bring conflicts to the evaluation process. Likewise, there can also be tensions between the program staff and evaluators, especially if the evaluators are from "outside" and were not a part of the project from the beginning. There may be mutual misunderstanding about the purposes and conduct of the program evaluation. Plans for conflict resolution should be determined at the beginning of any evaluation effort (Short, 1996). Conflicts in any of these areas can result in one or more of the stakeholders or evaluators erecting barriers to the process and almost guaranteeing that the results, however well founded, will not be used.

Resources

It is very important to make sure that appropriate resources are available for doing the evaluations throughout the various steps and stages of the program. If the organization has no monetary resources dedicated for evaluation and is unwilling to free up staff time, the feasibility of conducting an adequate evaluation must be questioned. Therefore, it is important that programs budget for evaluation in the planning stage. It can also be helpful to look for other options for doing evaluations, such as volunteers or students from local universities. Volunteers can help train staff and mobilize a process of self- evaluation useful to the organization. Students may be interested in conducting evaluations to gain work experience, develop professional relationships with health departments, or work in collaboration to obtain data to assist in educational requirements, such as completing a thesis.

Another issue of real importance is that of "overkill" in evaluation efforts. Some evaluations can cost as much as the program to be evaluated. While there are no hard and fast rules about the proportion of a program's budget to be spent on evaluations, it is useful to tailor the size of the evaluation to the importance of the decisions which need to be made, or the importance of the intervention to the overall public health objective.

The amount of time an evaluator can devote to the project is dependent on the available budget. Available time, combined with other resources, significantly influences the choice of methods. Site visits, for example, incur costs in terms of staff time as well as travel. Special outcome measures require substantial staff time for development, pilot testing, and analysis. Assessing more rather than fewer program participants, as another example, has significant cost implications for the program. Abundant resources available for evaluation are generally not found in STD and other public health programs, so there must be an effort to maintain a reasonable balance between the demand of scientific rigor and credibility and those of the budget (Herman, 1987).

At a minimum, the number and type of people involved in conducting an evaluation are the program manager, who understands the program and evaluation and knows of the resources available, the program staff who have experience with the program's activities and their target groups, and an evaluator, who knows the evaluation process and has expertise with specific methods and technical issues (Porteous, 1997). Another option is to collaborate with the evaluation team of an equivalent or sister program, or to join forces in a multi-site evaluation effort.

As the scope and limits of activities are defined, financial and staff resources, as well as expectations of the evaluation must be kept in mind. Organizations with an in-house evaluation staff of experienced epidemiologists, behavioral scientists, health planners, and evaluation specialties can be expected to conduct a more comprehensive, sophisticated, resource intensive, and wider array of evaluation activities than organizations with smaller budgets and fewer in-house staff resources (Schechter, 1993). However, an outside source for the evaluation should be considered when objectivity is needed, the necessary skills are not available in the organization, there is a lack of time or interest among staff, and if the budget is available (Marris, 1998).

Conducting an evaluation takes time and resources that are easily forgotten or overlooked. It is essential, when designing programs, that detailed evaluation steps and costs are included as part of the project action plan and budget. Consider a budget and time line for each of the following steps:

• Designing the evaluation
• Developing the measurement instruments
• Pilot testing measurement instruments
• Revising measurement instruments
• Collecting the data
• Processing the data
• Analyzing the data
• Writing the report
• Disseminating the results
• Critiquing the whole process.

Ethics

Many evaluations have the potential for putting individuals or groups and their work into a bad light or a good one. In addition, some evaluations have the potential for doing real or perceived harm to individuals or groups, both in the target population and in the staffs who are part of the program being evaluated. Evaluations which are conducted without regard to the people involved can also cause harm to the relationship between the communities and the health department or within parts of the agency itself. Those collecting information should be vigilant in keeping the confidentiality of participants and staff and be respectful of cultural values, language differences, and be as least disruptive as possible (Sanders, 1994). Data collection should be conducted in light of ethical considerations (e.g, it may not be ethical to randomize a subgroup not to receive an intervention which has proven to be effective), resources (e.g., data collection should not compromise delivery of critical program services), and sensitivities of program personnel (e.g., staff may feel concerned that evaluation results may shed a negative light on the program or their own work).

Standards to Consider in a Practical Evaluation

CDC has accepted the set of standards developed by the Joint Committee on Educational Evaluation (1994) which have been endorsed by a number of other professional organizations. As the steps in evaluation are undertaken, the standards are used to help determine the soundness of public health evaluation efforts. These standards are intended to ensure that:

• an evaluation will serve the information needs of its intended users. (Utility standards)
• an evaluation will be realistic, prudent, diplomatic, and frugal. (Feasibility standards)
• an evaluation will be conducted legally, ethically and with regard for the welfare of those involved in the evaluation, as well as those affected by the results. (Propriety standards)
• an evaluation will reveal and convey technically adequate information about the features that determine worth or merit of the program being evaluated. (Accuracy standards) (MMWR, 1999)

 

Previous Page

Table of Contents

Next Page