Listening to the Community, the Sequel:
Project Evaluation

Thoughtful and focused project evaluation is an important aspect of service provision and is a valuable tool for measuring changes in systems, attitudes, and accessibility. Feedback from the people involved in your project—whether they are collaborators, community members, people with disabilities, other organizations, or those whose services you seek to improve—tells you what you need to know most: whether your efforts are working. Evaluation also helps determine if a project can be replicated in its current state or if further development is required.

Feedback can inform you of public perception and awareness of the agency or entity, accessibility of facilities and services, how programs and processes are working for those served, and the referral process to and from the organizations in question. When you ask for feedback, you run the risk that you will learn something you are not expecting or prepared to hear. However, a successful evaluation will also reveal the areas in which your group’s work is succeeding, changes that are being made, and how your project is affecting people’s lives. Questions that commonly arise when designing an evaluation strategy include the following:

• Who should create and/or administer the evaluation?
• What questions yield the most constructive feedback?
• Who should be asked to respond (e.g., which organizations or individuals)?
• Is verbal or written feedback preferable?
• Is self-evaluation a valid tool?

It is often best to begin your evaluation by asking for feedback from individuals who use your services and those who do not, as well as peer organizations working with the population you serve. Determine what information will be most helpful to you and make it easy for the individuals and groups evaluating you to tell you what you need to know. Remember that a well-executed evaluation is invaluable to those who seek to improve, progress, and bring about positive change.

To assess progress in achieving the project’s goal of increasing crime victims with disabilities’ access to victim services and criminal justice systems, the 10 subgrantees examined the following core questions:

• Is the project effective?
• Is the project meeting its overall goals?
• Are participants benefiting from their involvement?
• Which project components are the most successful?
• Is the project worth sustaining?
• What changes or modifications are needed?
• How can the project be more efficient with its resources?
• Can the project be replicated in other communities?

The following examples illustrate some of the work done by the subgrantees to create and execute their evaluation plans, and the value of each step in the process:

Create an evaluation instrument that measures your project’s effectiveness, achievement of its goals, and benefit to its participants.

Carbondale Police Department staff reviewed all crime reports for indications that a victim had a disability, and the department’s victim advocate developed a survey to use during intakes with people with disabilities. This helped ensure that victims with disabilities received the strongest support possible during the crime reporting and victim service processes. In addition, the data obtained from the survey helped the department refer crime victims with disabilities to the community resources that would best meet their needs. The survey has since been used by other organizations in the Carbondale community, in forums such as group meetings and speaking events, to better coordinate service efforts to persons with disabilities. The information gathered helps identify crime victims who do not report their victimization to the police as well as those who have made a report but are not receiving the assistance they need.

Identify successful project components and those that need to change.

Process evaluation is a form of feedback that can evolve naturally in response to an unexpected barrier. That barrier may require an adjustment in direction, reconsideration of the original plan, or the realization that staff may choose to do things differently next time. At the beginning of the grant, project partners RCCCM and the Center for Living and Working (a disability service agency) held a cross-training session for staff from both organizations. In retrospect, RCCCM staff realize their time would have been better spent if all parties had gotten together beforehand to sort out basic issues such as differences in the agencies’ philosophies and terminology. (The word “client,” for example, is more accepted in crisis centers than in disability service agencies.)

When Ability1st staff suggested partnering with Florida churches to address victimization of people with disabilities, the faith community’s response was poor. Churches in rural counties were already overwhelmed by the needs of their members and the lack of community resources. In the more populated areas, no interest could be generated. In the third year of the project, Ability1st staff reviewed this strategy and shifted their focus more toward training law enforcement and coordinating services with domestic violence programs. The staff eventually discontinued their efforts to engage the faith community.

A few organizations, such as the Ulster County Crime Victim Assistance Program, struggled to interest local law enforcement organizations in training related to crime victims with disabilities. The county’s many law enforcement departments had various reasons for declining training, and staffing issues were frequently cited. In the second year of its project, Ulster County produced and distributed a training video for law enforcement, and hoped to reach greater numbers of officers by using a more versatile medium.

Law enforcement organizations encountered comparable difficulties when they attempted to partner with disability service organizations. The Lafourche Parish Sheriff’s Office worked diligently to let local disability service providers and people with disabilities know that its staff were interested in increasing accessibility to crime reporting and victim service processes, but the initial response was poor. Officers were persistent, however, and eventually community members started to participate.

SACASA’s first major obstacle was the disability services community’s initial distrust of and unfamiliarity with the agency. However, as they began to work more closely with the center’s staff, community members realized that advocating for disability services and advocating against violence had much in common. Both initiatives are based on a model of empowerment and encourage people to exercise their rights and make their own choices. It took several months, however, for the partners to identify that common ground.

Create measures that reveal changes in knowledge, skills, behavior, attitude, or conditions for participants.

Subgrantees providing education or training measured changes in knowledge, skills, or attitudes about an issue through surveys. Some groups measured changes in awareness of issues concerning crime victims with disabilities by surveying trainees both before and after presentations. Project satisfaction surveys allowed clients with disabilities to give feedback on accessibility and other facets of the service provided, as well as the impact any assistance had on their situation or in their lives. Several organizations measured the increase in the number of crime victims with disabilities accessing victim services and the criminal justice system during the project.

Project staff from all 10 subgrantee organizations used training evaluations, feedback, and the results of their efforts to keep the spirit of change “front and center” in their work. They changed their plans and activities as needed; revised information, materials, and process; and refined their training styles. Some reworked the focus or direction of their project. All learned that patience and flexibility are often the keys to success, and to appreciate the small victories and life lessons that come with working for change.

In that spirit, SACASA designed the following three evaluation techniques to help it continually evolve and improve its services for sexual assault survivors with disabilities:

• Every 120 days of service, clients of the center’s mental health crisis program are asked to fill out a questionnaire about their experience. People who identify themselves as having a disability are also asked about their experience with requesting or receiving accommodations. The center’s director of Mental Health Services reviews the questionnaires and relays feedback to program staff so they can modify their service delivery to better meet their clients’ needs.
  
  
• Staff members who do not provide direct services are assigned to interview persons with disabilities about their experience with the center and its services.
  
  
• The center’s Web site features a satisfaction survey with filters that allow the project to review data from respondents who self-identify as having a disability. The center also maintains a Web survey that allows people with disabilities to evaluate gaps in services in the community. This helps the center to measure the impact of sexual violence on this population and identify areas that need improvement.

Create specific outputs that quantify the work being done and outcomes that speak to the changes and progress made in addressing needs.

Outputs help a project measure both the rate and volume of work being done, as well as progress toward meeting established goals. Common examples of outputs used in projects like those described in this toolkit might include numbers of persons with disabilities served, training events conducted and number of people trained, outreach materials developed and distributed, community task force meetings held, or surveys completed by crime victims with disabilities. The 10 subgrantees developed the following strategies for measuring outputs:

• Collect and analyze 50 surveys about experiences in crisis services and criminal justice from community members with disabilities, as part of ongoing project evaluation. (SACASA)
  
  
• Provide 16 hours of training to 35 staff members to increase the capacity of all direct service staff to provide short-term emergency shelter and support to women with disabilities. (PADV)
  
  
• Provide all cadets with 8 hours of training on responding to crime victims with disabilities and working with people with mental illness and cognitive disabilities. (Lafourche Parish Sheriff’s Office)
  
  
• Convert 25 agency forms into Braille, large print, and audiotape. (SACASA)
  
  
• Increase the working group’s membership to include five key targeted representatives working in housing, senior citizen, and protective service agencies. (Ulster County Crime Victims Assistance Program)
  
  
• Train at least 25 people per quarter from law enforcement or the judicial system in how to work more effectively with and provide accommodations for crime victims with disabilities. (Ability1st)
  
  
• Educate 40–60 victims of domestic violence, sexual assault, or caregiver abuse on safety, rights, resources, and the dynamics and effects of abuse. (Safe Passage)
  
  
• Train staff of at least five police departments, three hospitals, and two courthouses in how to better serve crime victims with disabilities. (RCCCM)
  
  
• Provide training to 30 people with disabilities to increase their knowledge of crime, exploitation, abuse, and measures they may take to protect themselves, and to improve their understanding of law enforcement procedures. (Carbondale Police Department)
  
  
• Cross-train and provide networking opportunities for 50 people with disabilities and professionals in criminal justice, investigative social work, mental health, medicine, crime victim service, and disability service through a day of training and networking. (Chadwick Center for Children and Families)
  
  
• Develop 250 palm cards and distribute them to emergency first responders to use as a quick reference guide in a crisis and increase their knowledge of how to support survivors of abuse who have disabilities. (Stavros Center for Independent Living)

Outcomes are the expected results of project efforts. They measure more than a successful step or milestone in attaining a goal—they represent the true impact of project activities on the lives of people, and within organizations, systems, and communities. Outcomes of projects like the ones described in this toolkit are commonly given in percentages, whether related to persons served, conditions improved, crime reported, or knowledge increased. The 10 subgrantees developed the following methods for measuring their projects’ outcomes:

• 85 percent of sexual assault center staff will report that they have an increased understanding of the issues surrounding sexual violence for people with disabilities, as measured by a training evaluation form. (SACASA)
  
  
• 80 percent of sexual assault survivors with disabilities will report that they are satisfied with services provided to them by the sexual assault center, as measured by a client satisfaction survey. (SACASA)
  
  
• 50 percent of law enforcement and judicial trainees will demonstrate increased knowledge of, awareness of, and sensitivity to the needs of crime victims with disabilities, as measured by a training evaluation form. (Ability1st)
  
  
• 25 percent of officers attending training will show increased understanding of the issues and needs of crime victims with disabilities, as measured by training evaluations. (NOVA)
  
  
• Data will show a 25 percent increase in the number of crime victims with disabilities who requested services within a 1-year period. (NOVA)
  
  
• Data will show a 10–15 percent increase in hotline calls by crime victims with disabilities within the first year of operation of a 24-hour hotline. (Stavros Center for Independent Living)
  
  
• At least 25 percent of crime victims with disabilities accessing social and civil service agencies and the criminal justice system will report that they reclaimed their self-respect and dignity, and felt empowered to make their own choices, as measured through client satisfaction surveys. (Safe Passage)
  
  
• A 15–25 percent increase in the rate of reported prosecutions of perpetrators against crime victims with disabilities will be measured through police and district attorney’s office records. (Safe Passage)
  
  
• 60 percent of shelter clients who have disabilities will report a successful transition from emergency shelter to their long-term living arrangement. (PADV)
  
  
• 80 percent of police officers and hospital and courthouse staff who receive project-related training will indicate an increased awareness of the needs of crime victims with disabilities who access law enforcement services. (RCCCM)

Back to Top