Aboriginal and Torres Strait Islander viewers are advised that this website contains the names and images of people who have passed
Aboriginal and Torres Strait Islander viewers are advised that this website contains the names and images of people who have passed
Methodology
Assessment process and methodology for Looking Back, Moving Forward
"Looking Back, Moving Forward" aimed to be culturally responsive, safe, and Aboriginal-led. The project takes a self-determined approach to the assessment of recommendation implementation based on Aboriginal Justice Caucus’s (AJC’s) notions of progress and success.
To achieve this aim, the project was directed by a Project Steering Committee (called the AJC Working Group) comprising AJC members, including representatives from Aboriginal Community Controlled Organisations and Chairpersons from Regional Aboriginal Justice Advisory Committees. The Working Group contributed to project findings through a series of workshops responsible for review and sign-off of all project deliverables and key outputs, which enabled members to focus their efforts on the most impactful aspects of the project, including its design, and development and framing of key findings.
To assist the AJC Working Group, a Project Team of was created at Victorian Aboriginal Legal Service (VALS) (The Project Team). This team of researchers were responsible for delivering on the day-to-day requirements of "Looking Back" and were governed and overseen at all stages by the AJC Working Group.
This approach is consistent with the first principle for ways of working set out in Burra Lotjpa Dunguludja (AJA 4), which is the prioritisation of self-determination and the transfer of power and decision making to Aboriginal groups.
Learning from past reviews
Problems we identified
We identified the following problems in earlier assessment and review processes:
- Too much focus on actions taken rather than on the impact or effect of those actions.
- Failure to assess whether action taken was relevant to the intent of a recommendation.
- Lack of analysis of the legitimacy and/or accuracy of agency self-assessments.
- Failure to consider Aboriginal Community experiences and views.
- No prioritisation of recommendations requiring further response.
How we overcame them
This project aimed to overcome these noted deficiencies by:
- Gathering evidence on the impact and outcomes of action/s taken to implement recommendations.
- Analysing government/agency self-assessments in terms of the relevance of action taken to the intent of the recommendation, and the accuracy of claims.
- Prioritising recommendations requiring further action and response according to the degree to which they could reduce incarceration rates, increase safety in custody and advance Aboriginal self-determination.
Project phases
Phase 1: Establishment
Establish the foundations of the project, including through development of a high-level project plan and research protocol, and information gathering.
Phase 2: Quick Scan
Identify and prioritise thematic areas for more in-depth examination. Engagement with ‘justice system’ stakeholders was required to identify priority themes.
Phase 3: Thematic Deep Dive
Undertake thematic analyses of RCIADIC and coronial recommendations, implementation assessments, and progress updates from stakeholders responsible for implementation. This information was compared with qualitative and quantitative data (policies, manuals, evaluations, statistics, anecdotal reports, community experiences) that reflect the effectiveness of implementation.
Phase 4: Assessment and Reporting
In this final phase, the Project Team and Working Group came together through a series of workshops to interpret the information, make assessments of implementation progress, identify and frame key findings and develop recommendations for reform. This process drew together the findings and recommendations, and was endorsed by the Aboriginal Justice Caucus.
The assessment process
Principles for developing assessment criteria included:
- Aboriginal values – criteria reflect Aboriginal values and perspectives on the most important aspects of implementation
- Consistency – criteria can be consistently applied by various AJC members, and can be consistently applied across recommendations (RCIADIC and coronial)
- Simplicity – criteria can be easily applied, easily understood and interpreted by various audiences (Aboriginal Community, agencies responsible for implementation etc)
- Replicable – criteria can be applied at different points in time to assess progress in future
The AJC Working Group developed the assessment criteria and approach, which was endorsed by the AJC. The Project Team then provided detailed information for each recommendation which included:
- assessment status from previous reviews
- actions taken to implement the recommendation
- evidence of impact and or outcomes (we anticipate limited evidence of outcomes)
- the frequency with which the recommendation appears to have been breached (based on coronial findings, and concerns raised by Community members)
- related recommendations from other reviews, inquiries and Commissions.
Recommendations, grouped by theme, were presented to AJC members for assessment. AJC members provided an individual assessment and determined a consensus view of the implementation status for each recommendation.
Applying assessment criteria
There were four key questions to be answered to assess each recommendation:
- Does the action taken align with the intent of the recommendation?
- Is there clear evidence the desired change has happened?
- Is the recommendation still relevant today?
- Would full implementation reduce incarceration, increase safety in custody and/or progress Aboriginal self-determination?
These questions were assessed on several important criterion that reflect various aspect of implementation. A score of 0-3 was then applied (none, low, medium, high), with high scores representing a high priority.
Prioritising the recomendations
Given the vast number of recommendations examined in this project–339 RCIADIC recommendations and approximately 70 coronial recommendations–it was important that where further action was required to achieve full implementation, those recommendations were graded in order of priority so that efforts can be initially focused on those with the greatest potential to achieve significant improvements in justice outcomes for Aboriginal people.
Plotting the total score for the ‘action and outcomes’ criteria against the total score for the ‘relevance and potential impact’ assisted to prioritise recommendations for future work:
- Green zone: moderate to high degree of action taken towards implementation, with further action unlikely to have a significant impact = no need for further work
- Yellow zone: moderate to high degree of action taken towards implementation but further action likely to have additional impact on desired outcomes = some need for further work
- Red zone: low to moderate degree of action taken towards implementation, with further action necessary and likely to have a significant impact = highest priority for further work. These recommendations remain highly relevant, with great potential to contribute to key Aboriginal justice outcomes but there is little evidence of action taken. These are the highest priority for further work to complete implementation.
- The Project Team transcribed the assessment and discussion of each recommendation, providing a historical account for transparency and accountability.