Strengthening Indigenous evaluation

Our research identified a plethora of small programs (particularly health and well-being programs) currently being delivered by Aboriginal organisations that are not being evaluated. Out of 1082 Indigenous programs just under half (490) were delivered by Aboriginal organisations; of these, only 20 programs were formally evaluated and made public.  The lack of formal evaluations is a problem that plagues all providers of programs — not just Aboriginal organisations. However, given that the number of programs provided by Aboriginal organisations is so high, increasing their capability to undertake evaluations could have a significant impact on Indigenous outcomes.

While there is general agreement that random control trials (RCTs) are the ‘gold standard’ of research evidence, they are not always appropriate or practical. Estimating the counterfactual — what would have happened in the absence of the program, can be virtually impossible given the myriad of different programs in Indigenous communities. Moreover, there are few people significantly trained to carry out this type of evaluation in Australia.

There is also a difference between a health intervention and a program. While there may be evidence for the benefit of the intervention there may not be evidence on how best to deliver that intervention as part of a program.  Recent research suggests that it can take up to 17 years for health research findings to be adopted into practice. A recent inquiry found $72 million dollars was spent trying to implement reforms following the Gordon Inquiry into child abuse, but this money could not be accurately accounted for. Monitoring was abandoned because it was deemed too difficult to track progress of actions against particular recommendations.

The federal government has announced $40 million over four years to improve the evaluation of Indigenous programs, but this money will not go far if the government relies on external evaluators to conduct the evaluations. Instead of hiring outside contractors, the government should embed evaluation into the delivery of programs and fund organisations to self-evaluate and collect data through online data management systems. A co-accountability framework could monitor the achievement of outcomes and provide a two-way exchange, with the experiences of those delivering the program being used to inform its ongoing implementation.

Sara Hudson is a Research Fellow and Manager of the Indigenous Research Program at the Centre for Independent Studies. Her report Evaluating Indigenous programs: a toolkit for change  was published this week.