Interagency Agreements (IAA) Performance Measurement Training

TINA DIMACHKIEH: Hello and thank you so much for joining us for the OVC IAA Performance Measurement Training hosted by the Performance Management Team. I'm Tina Dimachkieh, I'm the Training and Technical Assistant Specialist providing contractors support to OVC and OVC-funding recipients.

During today's session, we are going to be going over the Performance Management Team. We'll introduce you to our team and let you know who we are and how we can support you. We will be talking about performance measure reporting at OVC, and how OVC uses the performance measure data provided by you all. Then dive into the IAA reporting system that you'll be using to report out your measures, the reporting types and the deadlines as well, and then move on to talking about the performance measures themselves and I will walk you through the different question sets that will be available for your interagency agreement data collection. Then we will provide some contact information so you can reach out to us if you need any additional support.

As I said, I'm Tina Dimachkieh, I'm part of the Performance Management Team and we have an entire team that is available to you all that works with OVC around performance management and performance measures for reporting for federal funding recipients. That includes interagency agreements, federal partners, awards and grants, and all the different programs that OVC supports. We have a team of data analysts as you seen on the screen, here. We have Abria, Finesse, and Aubrey as they are our analysts who review all the data that's coming in from the different OVC programs.

We always get a lot of questions from organizations that receive funding, mostly asking, Does anybody really look at the data? Where is it going? What do you do with it? So, these three individuals are data analysts. They are the ones looking at the data all the time. They are making sure that the responses are logical, they are making sure that their responses are accurate, and then they're also responding to data with requests from a multitude of different organizations and leadership and partners around the data that's being collected and they develop different reports, that I will share examples of later on in the slide.

We also have our own helpdesk specialist response to helpdesk calls and email tickets that we receive from federal partners and our grantees to make sure that we are connected with you, to make sure that we're directing you, to the right point of contact, to respond to any issue that you're experiencing, or to provide additional assistance. Many, as you see, here, myself, TTA Support, we do training and technical assistance. I do individualized technical assistance for funding recipients. As well as larger program trainings around whether it's reporting months or just in general, around any changes that are taking place, across the performance measures for different programs, or refresh trainings, or topic specific trainings as well. Last, we have our system subject matter expert Liz, who works with the developers on the platform side to make sure that any bugs or issues in the system are addressed and to make sure that everything is functioning as it should for our funding recipients, and she really is very, very helpful and supportive in making sure that everything is running smoothly with the platform that we use. We all work under Virginia Ward-Proctor. She's our task lead and she is our main connection between us and OVC grant managers, OVC leadership, and funding recipients as well.

So, what can our team do for you? We provide general support. You can contact us, as I said, through the helpdesk, through RIF by either calling or sending in an email ticket and we will respond to you that way, but we also love to hear from you and love to support you through outreach and through technical assistance requests. We call them coaching sessions so we can do so many different sessions. Some are one-on-one sessions to introduce you to reporting for new staff members, new units that are joining us. We also do refresh sessions after trainings that we've held such as this one. We also can create a session or work on a session together where we discuss a certain measure, a certain case or a situation that you're experiencing as it relates to the funding provided by OVC that you need assistance with when you're collecting data or entering data into the platform. We really do it all around the measures, but, so, the only thing you would need to do is make sure that you are reaching out to us and you're connecting with us, so, we can provide that support to you.

Our sessions we see on the screen, here, we usually schedule them for 30 minutes, but some of them could be as short as 10 minutes just to talk through a question and answer it or they could be an hour. We could schedule follow-up sessions as needed and connect you with the right representatives, if we could not answer certain questions or provide the level of support that you are looking for. We also have a new webpage for our interagency agreement performance measures.

We have a webpage for you where this recording will be housed, but other resources and the measures will be housed there as well. It's part of the OVC performance measures webpage, and it's its own separate page for you all. Each program has their own page and now IAA has their own page as well. We will be including any new resources on that webpage, any other recorded webinars, everything will go on that webpage for you all. It also does include a dictionary and terminology resource that can help you better understand the terms that OVC is using within your measures as you are responding to the performance measure questions. So, please feel free to visit that webpage often scroll around and get familiar with it and with the resources that will be available on that webpage.

So, performance measure reporting at OVC. How does OVC use data that's provided by our federal partners and by other funding recipients as well? The accuracy and timeliness of data reporting is extremely important because we really do look at the data and we really do use it for multiple different things at the OVC level. Reporting data accurately and in a timely manner will help OVC demonstrate the value and the specific benefits of the programs, specifically, the interagency agreements, to leadership, to federal and state government agencies, to Congress, to the victim services field at large, to just the general public, and any other stakeholders that might need that information.

It also helps generate annual reports about the different programs; and demonstrate the output of the different programs and the funding that is being provided; and for you all, specifically, their agency agreement funds that are being provided; and the work completed with those funds based on the data provided. It'll also help us emphasize the progress made towards achieving OVC's strategic and programmatic goals for your agreement, but also, OVC's larger mission as well. And it will help us, and it does help us, reach the target audience of grantees, federal partners, and agencies, and the general public and raising awareness around victim services and the availability of services that are provided with OVC funding.

Here are some examples of data analysis and reports. Our analysts have been able to create program reports as you see on the screen, here, specifically, around victim demographics and the victims that are being served with OVC funding. This report on the left side shows the different demographics that are collected around age, gender, and race and ethnicity. We also have a couple of snapshots around specific signs. This specific one, here, in the middle is around human trafficking and we collect data, report out who of the victims are being served the most, what types of victims, and based on demographics and the types of crimes where there's multi victimization being experienced and then have the third one, where it helps us review the measures and make sure that they are as accurate and that they are as informative as they need to be for the different programs and it also helps us create different data collection tools as well. We really do use that data and it's really important that you report accurately and on time.

So, we'll talk about the IAA reporting system that our federal partners will be using, the reports themselves, and the deadlines for reporting. Partners under IAA are only going to be using one system and that is the performance measurement tool, the PMT. The PMT is a web-based reporting system where grantees and other federally-funded partners can electronically submit quantitative and qualitative program performance data. You'll hear me say those words a few times throughout this presentation, qualitative and quantitative. Those are the two different types of information that you will be providing us through PMT. PMT will be launching for your program on October 1, 2023.

So, as of right now, we're not on October 1, and you're listening to this recording. You may not have access to the PMT just yet. All our federal partners will have access to the PMT by the launch date of October 1st, 2023. By then our team will have reached out to you and provided additional guidance on what you will need to do and different instructions on how to access the system. After we have created an account for you, we will let you know that, there's an email and a password to log in. We also will be hosting additional training on the use and the functionality of the PMT. We will be sure to provide all that information to you and any resources written guidance to you well in advance before the October 1st anticipated launch date.

So, be on lookout for that additional information. We also will be sure to complete reports together and to provide additional support in October for that first reporting period. I will be talking about the due dates in just a moment, but more to come on the platform and the use of the platform. For now, if we are before October 1st, all you will need to know is that, you're only using one reporting system and that is the performance measurement tool, the PMT. We will make sure that all the points of contact that are listed in the interagency agreements do have access. If you would like to add additional staff members, you will have the ability to do so, so, they can access the PMT as well and we will go over that also as we get closer to launch date.

Okay, so report due dates, partners follow a semiannual reporting schedule, so, you will be reporting on your performance measures semiannually, as for a year, every 6 months and as you see on the screen, here, the first reporting period is October 1st to March 31st. That encompasses agreement activity or funding activity from October 1st to March 31st. The submission period for that report is between April 1st and May 30. What this means is that, the PMT will have that report available to you all to enter the data from October 1st to May 30. The deadline of submitting data for award activity or agreement activity that took place from October 1st to March 31st is May 30, so that is the last day that you can enter data into the PMT for that reporting period. The information that is due, as I said before, is that quantitative data report and the semiannual narrative questions, which are the qualitative responses that we'll talk about in a moment.

The second report period out, of the two semiannual reporting periods, is the April 1st to September 30 reporting period. The submission period will be open from October 1st to November for our partners to enter data into the system. The last day to enter data for the April 1st to September award activity is by November 30. That's the deadline, and again, you will be responsible for entering quantitative data and semiannual narrative responses into the PMT by the 30th of November for that second portion. For the first period of reporting, when the PMT launches on October 1st, 2023, we will provide guidance on how to report on two separate reports because you'll have an entire year of reporting.

So, you will be reporting from October 1, 2022, data up until September 30, 2023 data, as applicable. You only are reporting on funding provided by OVC on activities that took place with the funding provided by OVC; but if you did not use any funding under your agreement, you would not have any data to report on and I will speak more to that momentarily. Okay, performance measures themselves, so, all our IAA federal partners will have to respond to narrative questions. However, based on the nature of your agreement and the work that you are completing with OVC, you will be assigned certain question sets. On this screen, here, you will see all the question sets that are available under the IAA program. Those may not all be applicable to your work, and that's okay. We will be providing additional guidance on what's applicable to which agency and which agreement, and you will have different question sets already pre-assigned in the PMT, so, you will not have to find that out on your own.

So, that's our program activity: the first program activity, the second one's IAA-funded staff and services, the third one is emergency funding for victims, the fourth is training, the fifth is technical assistance, the sixth is technology development, the seventh is data gathering, the eighth is collaborative partnerships, the ninth (and second to last) is victim services we ask that everybody will be responsible for their semiannual narrative questions. That's the qualitative portion. The first nine are your quantitative piece and last one is the qualitative one. Again, as you see noted on the screen, here, partners are only going to be reporting on OVC funded activities into the PMT.

Partners only need to respond to question sets and specific measures that apply to their OVC funded work and not anything outside of the funding of OVC. What this means is that your unit or your the staff members working on the agreement with you might be certain additional victims or doing additional work within your organization. We are not looking for those numbers. We are only looking for what work was completed with OVC funding.

If you recall in previous slide I showed you what OVC does with the data. We want to reiterate that when we're reviewing the data and creating our reports and, you know, demonstrating the value of the work and emphasizing the reach of OVC, we're only looking for the funding that OVC had provided and data collected based on that specific funding and not the larger picture of the work that your units or your organizations are completing. Types of questions that you will be completing.

As I've said a few times now, you will be completing the quantitative reporting questions. Those are the numbers, right? Those are the standard performance measure data about the OVC funded program activities that were completed during the reporting period. So, this is where you're just answering based on numbers and services that you provided, the activity that you completed. Then the second set of questions is the narrative questions. Those are the qualitative questions, the qualitative information about the program goals, activities and other factors that have impacted program delivery.

These are open-ended essay questions. They could contain numbers and they could speak to the numbers that you provided in the quantitative portion of the reporting, but they are basically questions to explain to OVC and to partners at OVC what type of work engaged in and to explain what the numbers that you provided actually mean. I'll go over those narrative questions with you as well. Questions set. IAA partners have to respond to the first question set. This one is assigned to all agreements because it asks two very specific questions. The question set is program activity and the first question is, "Is the last reporting period during which the award will have data to report?"

So, unless your agreement has reached its end date or you are unfortunately terminating an agreement for any reason, you will not answer "yes" on this question Your answer should always be "no" unless you have reached the end date of your agreement. This question is purely for the system to recognize to create final report for that agreement. If you say this is the last report period, the system will then generate, compile all of your data, and generate a final report for OVC. But if your agreement is ongoing, you will always answer no, so, you can just respond to the questions without the system entering it to be a final report. Questions one and two are "yes" and "no" responses, so, it's just the radio button that you would be clicking and we will show you when we demonstrate the PMT, we do a demo of the PMT when that training comes up, what I mean by that, but that's at a later date.

Second question asks, if there was program activity during the reporting period? This question is very important and it's very important to note that, it's asking about program activity and what program activity is defined as is, "Activity that occurs when the agreement has obligated, expended, or drawn down program funds to implement objectives proposed in the OVC-approved program application."

So, basically what it means is, "Did you use any OVC funding during the reporting period? Did you use any money from your agreement?" If you did not, you would just answer "no" and then you just need to provide an explanation as to why you did not use the funding over the previous 6 months or the semiannual reporting period. If the answer is "yes," then you would move forward to respond to the performance measures that are applicable to your agreement. But if you answered "no," the system will then be triggered to cancel out all the additional questions sets because answered "no," then we don't need any data from you because you did not use funding and were only collecting again, funding or data based on OVC funding provided any used.

So, if you did not have any activity, you don't have to answer the qualitative questions because there's no data to report on, but if you did have activity you would answer "yes" and then the system would be on the next set of questions, which is the second question set. That's the IAA-funded staff and services questions set. For this question set, our IAA federal partners are asked to provide information on the number of staff whose salaries were either fully or partially paid for with IAA OVC funding during the reporting period.

Partners are instructed to include both full and part-time staff, as applicable, and they're only supposed to count each staff member one time between the three subsections that are listed here for you. The subsections are the number of staff enrolled providing A, direct services to tribal victims, B, direct services to non-tribal victims, C, non-direct service support roles such as administrative, financial and training.

So, this is where you would be entering the number of staff members. If you don't have any staff members or any responses to this because it's not applicable to you, you would then enter "0" or an applicable response. The third question set is the emergency funding for victims. Again, this question set only applies to your agency agreements and federal partners that have agreed to provide or are providing emergency or short-term funding. If your organization is not doing so with your agreement, you would disregard this question set, enter zeros and move forward. For those that are providing emergency and short-term funding for victims, they should be including in the numbers and the data provided, they should be including family members and companions as well.

The data needed, what you need to provide data for, is domestic and international incidents, the number of cases, the number of victims, and the total amount of funding provided. The emergency funding subcategories that are available and measures are travel and transportation, lodging, counseling, scene clean-up, personal effects, cleaning and shipment, food, clothing, medical, conferences room, other. That "other" category is, very important because if you did provide emergency funding and none of these subcategories are applicable to the funding you provided, you would then select "other;" and you would explain what that other is so, we could better understand where the funding was used, and then this also, if you provide an accurate description of what "other" is, that will help us revise the measures at a later time to be sure to include more accurate subcategories as needed.

Through the measures, I will point out where else we have subcategories or is a category for you to do that exact same thing, provide an additional explanation as to why you selected "other" as opposed to any of the other subcategories that are provided. Again, for emergency funding for victims, the second piece of that performance measure or that question set is around travel and the map, if there was any travel funding provided for victims or companions. If so, you will need to provide the number of victims and companions who traveled and the total amount provided for travel during the reporting period.

The travel funding subcategories that are available are for hearing appearance, trial appearance, other court appearance, transportation, meeting with prosecution, meeting with law enforcement, medical exam, testing and treatment or treatment, reunification travel and other reason. Again, this is where you would include a reason if none of the subcategories are applicable and you would explain what that reason is.

The question set is the training question set. This is where our federal partners will be reporting data on training activities that were funded as part of the agreement during the reporting period. I do want to make note that training differs from technical assistance, that it is usually intended for multiple audience types and is not customized for a single group. I'm making this distinction because both training and technical assistance are their own question sets and we will talk about technical assistance in the next slide.

For training, what is included for data is any training events that your agency provided, hosted, coordinated, attended or planned using IAA funding. This includes in-person trainings and presentations and virtual webinar trainings and conferences as well. The data that you'll need to provide for the training question sets is the number of hours of training that was delivered, types of trainings included in the data, the types of participants who attended training hosted by the agency.

This one specific to training hosted by the agency and if feedback surveys were distributed and collected. So, next question set is technical assistance and four technical assistance partners are reporting on technical assistance activities that occurred during the reporting period. Again, technical assistance differs from training in that it is customized for the needs of a particular group providing victim services of some kind. Data that our partners need to provide here is around TTA requests received and complete within that reporting period and the nature of the technical assistance.

Next question set is around technology development; and again, I will repeat, if this not apply to you, if your organization is not providing or engaging in technology developments for your agreement, you would just disregard these questions, enter zeros and move forward to the next question set. The information that partners who are providing technology development data will need to report on is in pursuit of the three goals listed below.

The three goals are: providing support for improved assistance to victims, including information or referrals as well as online or hotline services of national and international level; the second goal is enhancing the state's access to technology, increasing victims access to resources and increasing the accuracy of administrative reporting; and lastly, using technology to effectively navigate the criminal justice system and to connect end users with victim-centered resources and services.

Specific measures or the data that you'll be collecting or providing is on the type of technology that is used. Number of contacts received by each type of technology listed. Contacts are considered to be the targeted unit of measurement in the program, which may be a website hits or contacts such as text, email, phone calls, chat to a hotline, may need to separate out the in-person visits and contact by technology. If the technology is not applicable, again, just enter zeros and move forward.

The technology type subcategories that are available in this performance measure are telephone hotline, text message service, instant message-chat service, website, mobile app, cybersecurity/safety technology, report/data management capacity that is not public-facing and we also have that other option in case you need to use that and explain what that other is. Since that is around data gathering, Partners will need to report data gathering and research activities if they did engage in any of those activities. The goal is to produce and disseminate information resources that improve delivery of services to victims of crime and the data that you'll need to provide for this is around the types of data gathering initiatives that were completed and the feedback or the outcomes from the data gathering initiatives.

The types of data gathering subcategories are: literature searches and reviews, needs assessments, gap analysis, report, and again, other, in case the gathering activity does not fit into any of these subcategories and you would need to put an explanation of what that activity is.

Next, we have collaborative partnerships. Here, you're reporting information on activities completed to establish or maintain partnerships that were funded as part of the agreement. Collaborative partnership data is on the groups and organizations and agencies participating in the program. The frequency of these partner meetings, if meetings are held, and the intended outcome of the meeting. This is the second to last, the ninth question set and this is only applicable for our federal partners who are providing direct services to victims under their agreement.

If your agreement does not encompass providing direct services to victims, this question set would not be applicable to you and you would need to either enter zeros or disregard the question set depending on whether it's listed in the PMT for you. As question set for these measures, we are looking for direct victim services provided using agreement funding, again, only using agreement funding. We're only looking at data from OVC funding provided. That includes data on the number of victims served, the number of victims served who are new and demographics of new individual schools. Demographics are age, race, ethnicity, and gender.

There's a number of underserved victims and it's the services provided and the number of occurrences for each service subcategory. The service categories or those types of services reported are, as you see on this screen, here, information and referral, personal advocacy and accompaniment, emotional support or safety services, shelter/housing service, criminal civil justice system assistance. Within these types of services, we have subcategories. There's about more than five under each type of service and you will need to select the type of service and the subcategory and the number of times it occurred within that reporting period.

I have the semiannual narrative questions. This is the question set that is qualitative and not quantitative. This is the one where you are entering in an open-ended response or essay response to three different questions to explain what the numbers that you entered in the quantitative part of the reporting and the measures part of the reporting, what they mean and the work that you completed over the 6 months has done or what the work looks like. Information that we're looking for is if there were any problems, delays or adverse conditions that affected your ability to reach your goals and objectives. Just really helping us understand what happened under your agreement or within your agreement with the funding and how we can better understand and assist as your partner.

We also are looking for any significant developments as it relates to the project. These may be factors that are internal to your organization or external as it relates to the larger community that you serve or the Nation that positively or negatively affected your project implementation. And the most important piece is we are looking for success stories. We love to hear about success stories at OVC. It's what gives us motivation to continue to do the work, all of us, right?

Success stories are always welcome. Summarizing accomplishments of the program, summarizing the specific case that you worked on with the funding and the outcome of it would be very informative for us to hear about. Of course, when you're speaking about cases and victims, to redact any personally identifying information and only speak to what you can. We totally understand that there are certain things that you are unable to share, especially in ongoing cases.

This is it for narrative and the narrative questions. That's really all we're looking for and they are open-ended. You will have a character limit in the system, around, I want to say 5,000 characters when entering your response, but we will talk about the system in greater detail during the PMT training that we will be providing later on in the year. So, bear with us as we get you more information.

I've reached the end of the presentation and I want to make sure we provide you with contact information so, you can reach out to us through our helpdesk if you have any questions, if you need additional support, or you need assistance walking through a specific measure or understanding the reporting requirements better, we are here to help. We are more than happy to hear from you and want to be supporting your agreement and making sure that reporting is a breeze for everyone. If you need to reach out to us, you could give us a call, you could email us, and also be sure to visit the webpage that we have developed for you for additional resources as you continue working on your agreement.

Thank you again for taking the time to join me today. I know this was a long presentation, there was a lot of information in here for you, but we will be reaching out with additional information and make sure that you have all the resources and the tools that you need in order to report accurately and on time for your interagency agreement. Again, thank you so much and good luck with reporting.