As you may recall from the definition provided in Chapter 1, evaluation research is research conducted to assess the effects of specific programs or policies. Evaluation research is often used when some form of policy intervention is planned, such as welfare reform or school curriculum change. The focus on interventions and social problems makes it natural fit for social work researchers. It might be used to assess the extent to which intervention is necessary by attempting to define and diagnose social problems in social workers’ service areas, and it might also be used to understand whether their agencies’ interventions have had their intended consequences. Evaluation research is becoming more and more necessary for agencies to secure and maintain funding for their programs. The main types of evaluation research are needs assessments, outcomes assessments, process assessments, and efficiency analyses such as cost-benefits or cost-effectiveness analyses. We will discuss two types in this section: outcomes assessments and process assessments.
An outcomes assessment is an evaluation designed to discover if a program achieved its intended outcomes. Much like other types of research, it comes with its own peculiar terminology. Inputs are the resources needed for the program to operate. These include physical location, any equipment needed, staff (and experience/knowledge of those staff), monetary funding, and most importantly, the clients. Program administrators pull together the necessary resources to run an intervention or program. The program is the intervention your clients receive—perhaps giving them access to housing vouchers or enrolling them in a smoking cessation class. The outputs of programs are tangible results of the program process. Outputs in a program might include the number of clients served, staff members trained to implement the intervention, mobility assistance devices distributed, nicotine patches distributed, etc. By contrast, outcomes speak to the purpose of the program itself. Outcomes are the observed changes, whether intended or unintended, that occurred due to the program or intervention. By looking at each of these domains, evaluation researchers can obtain a comprehensive view of the program.
Let’s run through an example from the social work practice of the wife of Matt DeCarlo who wrote the source material for much of this textbook. She runs an after-school bicycling club called Pedal Up for children with mental health issues. She has a lot of inputs in her program. First, there are the children who enroll, the volunteer and paid staff members who supervise the kids (and their knowledge about bicycles and children’s mental health), the bicycles and equipment that all clients and staff use, the community center room they use as a home base, the paths of the city where they ride their bikes, and the public and private grants they use to fund the program. Next, the program itself is a twice weekly after-school program in which children learn about bicycle maintenance and bicycle safety for about 30 minutes each day and then spend at least an hour riding around the city on bicycle trails.
In measuring the outputs of this program, she has many options. She would probably include the number of children participating in the program or the number of bike rides or lessons given. Other outputs might include the number of miles logged by the children over the school year, the number of bicycle helmets or spare tires distributed, etc. Finally, the outcomes of the programs might include each child’s mental health symptoms or behavioral issues at school.
Outcomes assessments are performed at the end of a program or at specific points during the grant reporting process. What if a social worker wants to assess earlier on in the process if the program is on target to achieve its outcomes? In that case a process assessment is recommended, which evaluates a program in its earlier stages. Faulkner and Faulkner (2016) describe three main goals for conducting a process evaluation.
The first is program description, in which the researcher simply tries to understand how the program looks like in everyday life for clients and staff members. In our Pedal Up example, assessing program description might involve measuring in the first few weeks the hours children spent riding their bikes, the number of children and staff in attendance, etc. This data will provide those in charge of the program an idea of how their ideas have translated from the grant proposal to the real world. If, for example, not enough children are showing up or if children are only able to ride their bikes for ten minutes each day, it may indicate that something is wrong.
Another important goal of process assessment is program monitoring. If you have some social work practice experience already, it’s likely you’ve encountered program monitoring. Agency administrators may look at sign-in sheets for groups, hours billed by clinicians, or other metrics to track how services are utilized over time. They may also assess whether clinicians are following the program correctly or if they are deviating from how the program was designed. This can be an issue in program evaluations of specific treatment models, as any differences between what the administrators conceptualized and what the clinicians implemented jeopardize the internal validity of the evaluation. If, in our Pedal Up example, we have a staff member who does not review bike safety each week or does not enforce helmet laws for some students, we could catch that through program monitoring.
The final goal of process assessments is quality assurance. At its most simple level, quality assurance may involve sending out satisfaction questionnaires to clients and staff members. If there are serious issues, it’s better to know them early on in a program so the program can be adapted to meet the needs of clients and staff. It is important to solicit staff feedback in addition to consumer feedback, as they have insight into how the program is working in practice and areas in which they may be falling short of what the program should be. In our example, we could spend some time talking with parents when they pick their children up from the program or hold a staff meeting to provide opportunities for those most involved in the program to provide feedback.
A third type of evaluation research is a needs assessment. A needs assessment can be used to demonstrate and document a community or organizational need and should be carried out in a way to better understand the context in which the need arises. Needs assessments focus on gaining a better understanding of a gap within an organization or community and developing a plan to address that gap. They will often precede the development of a program or organization and are often used to justify the necessity of a program or organization to fill a gap. Needs assessments can be general, such as asking members of a community or organization to reflect on the functioning of a community or organization, or they can be specific in which community or organization members are asked to respond to an identified gap within a community or agency.
Needs assessments should respond to the following questions:
In order to answer these questions, needs assessments often follow a four-step plan. First, researchers must identify a gap in a community or organization and explore what potential avenues could be pursued to address the gap. This involves deciphering what is known about the needs within the community or organization and determining the scope and direction of the needs assessment. The researcher may partner with key informants within the community to identify the need in order to develop a method of research to conduct the needs assessment.
Second, the researcher will gather data to better understand the need. Data could be collected from key informants within the community, community members themselves, members of an organization, or records from an agency or organization. This involves designing a research study in which a variety of data collection methods could be used, such as surveys, interviews, focus groups, community forums, and secondary analysis of existing data. Once the data are collected, they will be organized and analyzed according to the research questions guiding the needs assessment.
Third, information gathered during data collection will be used to develop a plan of action to fill the needs. This could be the development of a new community agency to address a gap of services within the community or the addition of a new program at an existing agency. This agency or program must be designed according to the results of the needs assessment in order to accurately address the gap.
Finally, the newly developed program or agency must be evaluated to determine if it is filling the gap revealed by the needs assessment. Evaluating the success of the agency or program is essential to the needs assessment process.
Evaluation research is a part of all social workers’ toolkits. It ensures that social work interventions achieve their intended effects. This protects our clients and ensures that money and other resources are not spent on programs that do not work. Evaluation research uses the skills of quantitative and qualitative research to ensure clients receive interventions that have been shown to be successful.
