The responses of the three categories of respondents to this question differed considerably.Respondents from departments and statutory agencies believed that the KPIs in the National Drugs Strategy (NDS) could form the basis for measuring the performance of DTFs, and that the annual work plans of the DTFs should be examined to determine the extent to which DTFs have achieved their objectives and delivered outcomes. They also suggested that agencies and projects funded through the DTFs should provide information on their local area to enhance analysis and performance assessment, and that the provision of this data should be a condition of funding. Finally, they suggested data on the following topics could facilitate performance measurement: 

·         outputs and outcomes, including evidence of progression and individual achievement;

·         reasons for exit from treatment;

·         number of treatment centres and barriers to access, if any;

·         number of rehabilitation places;

·         number of prevention projects being undertaken;

·         number of awareness programmes developed;

·         number of arrests; and

·         number of seizures.

 

Respondents from the community and voluntary sectors advocated the development of a valid and reliable outcome measurement system, with targets not just for outputs, but for short and medium term impacts, and KPIs established for each area. They also recommended that DTFs adopt a logic-model approach to their work based on implementation of the NDS. Respondents from the community and voluntary sectors suggested a series of process-oriented KPIs: 

·         extent to which DTFs provide accessible information on drug use and misuse services;

·         extent to which a DTF is community-focused;

·         extent to which a DTF makes strategic decisions in the funding of projects;

·         evidence that the local/regional plan is linked to the NDS;

·         evidence that the DTF network is influencing policy;

·         evidence of how DTFs share information and promote best practice;

·         evidence of DTF involvement in other social inclusion initiatives;

·         percentage of funding spent directly to help drug misusers and recovering drug users;

·         extent of community engagement in DTF principles, goals, plans and strategies;

·         increase in local leadership and local capacity; and

·         level of allocation of resources to community engagement activities.

 

Respondents from among the DTFs noted that while there are KPIs for DTFs in the NDS, these KPIs are designed to measure the effectiveness of the NDS (including the co-ordination function), and not the performance of any one agency. Therefore, DTF respondents proposed the following KPIs, including a mix of both process and content measurements: 

·         adequacy and appropriateness of representation on DTFs;

·         responsiveness of DTFs at local level to national decisions;

·         interagency working, local co-ordination and participation, level of project staff involvement;

·         impact of DTF on communities;

·         qualitative and quantitative measures of Treatment outcomes, e.g. client numbers, waiting times; programme retention and completing times;

·         qualitative and quantitative measures of emerging drugs use trends;

·         process indicators;

·         compliance with QuADS;

·         training results;

·         health monitoring;

·         value for money.

 

Respondents from this category suggested that a small group of task force co-ordinators be established to devise KPIs along the lines of Provan and Milward’s framework.2 It was also proposed that the DTF forms should be evidence-based and follow a logic model.

 

There was a general consensus among respondents from all three categories that a standardised evaluation mechanism for DTF projects is needed; respondents from among the DTFs also called for a common reporting/evidencing framework. It was suggested that the Drugs Programmes Unit in the Department of Health should develop an evaluation tool in ‘close consultation’ with DTFs and the channels of funding, and that, rather than trying to reinvent the wheel, existing tools, e.g. from the EMCDDA best practice portal or the WHO, should be used as a starting point. Other suggestions were that the evaluation tool should be based on the objectives of the core work of the funded projects and linked to the objectives of the relevant DTF, e.g. through use of the LDTF 1 form; that the theory of change and/or logic model should provide the basis for a standard evaluation tool; that a logic-based evaluation template should be developed to reflect the QuADS standards; and that a common web-based computer database with standardised questionnaires and agreed KPIs for DTFs and projects be adopted.

 

Evaluation criteria based on the following types of evidence were suggested by one statutory agency:

          appropriateness of drug-focused interventions;

·         strategic fit with NDS actions, DTF work-plan and fitness for purpose;

·         range and type of activities/services delivered;

·         progression and individual achievements;

·         impact on the individual service user and added value to community; and

·         project capability and sustainability.

 

Respondents from among the DTFs also suggested that it was important to measure the work being done by workers, as well as changes in clients and families. The use of new technology, such as the client relationship management system being researched by some DTFs, was mentioned in this regard.

 

The final report and recommendations are still awaited. In thinking about how performance measurement and evaluation systems may develop in the future, it should be borne in mind that the interim report states that one priority of the review is to identify where it would be appropriate to transfer responsibility for projects to statutory agencies, thus reducing the number of projects supported by DTFs in the future.

 

1. Drugs Programmes Unit (2012) Report on the consultation process in relation to the review of the structures underpinning the National Drugs Strategy. Dublin: Department of Health.  www.drugsandalcohol.ie/17027

2. Provan K and Milward HB (2001) Do networks really work? A framework for evaluating public-sector organizational networks. Public Administration Review, 61(4): 414–423.