By Type:
Table of contents:
A mixed method approach, using both quantitative and qualitative techniques, was utilised for this evaluation. This included an online survey of all participants in the SMG programme to gain baseline data on the overall effectiveness of the programme; key informant interviews with a sample of Māori organisations, assessors and Te Puni Kōkiri staff to gain more ‘rich’ information on achievement of outcomes and specific examples; and a review of all relevant documentation and assessors’ summary reports.
Evaluators used the judgement of key informants to assess the level of contribution of the SMG programme to the achievement of outcomes or to a change in how organisations operated. This approach is considered subjective because key informants may be likely to attribute any change that takes place to the SMG programme rather than to other factors. However, it provides some evidence of the change.
A more detailed discussion of the methodology is provided in Appendix one.
The evaluation focused on all Māori organisations that participated in the SMG programme from 2003/04 through to the 2007/08 period.
All 110 organisations that had participated (as at March 2008) in the SMG programme were selected as part of the population cohort for the online survey. An email list was generated from the Te Puni Kōkiri administrative database and validated by programme and regional staff. As a result of the validation process, the survey frame was reduced to a population of 89 organisations.
From the 110 organisations identified, a random list of 20 organisations was selected. A purposeful sample of six organisations was then drawn based on the:
The six organisations were drawn from five Te Puni Kōkiri regions: Te Taitokerau, Tāmaki Makaurau, Te Tairāwhiti, Te Whanganui-a-Tara and Te Waipounamu.
Three of the six assessor organisations were randomly selected to be part of the key informant interview process. All assessors interviewed had conducted over 10 organisational assessments and/or undertaken remedial work through the SMG programme.
Te Puni Kōkiri developed a draft outcomes model (see Figure 1 below)4 to inform the overall evaluation design, test the assumptions around the outcomes and, ultimately, inform Te Puni Kōkiri’s investment strategy. The model describes the key outcomes required to achieve healthy, vibrant and excellent Māori organisations. For this to be achieved, a series of low level outcomes needs to be present. These low level outcomes (such as appropriate financial management, relevant governance structure and strong management) provide the focus of the evaluation, and some attempt was made to assess (to the extent possible) whether the higher order outcomes, such as strong Māori organisations and high performing Māori organisations, were achieved. (Refer to the third tier of boxes in Figure 1).
All evaluation questions were derived from the draft outcomes model and the programme outcomes for the SMG programme. The key evaluation questions were:
Document analysis, an online survey and key informant interviews were undertaken to inform the objectives of the evaluation.
SMG programme summary reports, briefing materials, SmartFund5 and the 2005 process evaluation were analysed to understand the SMG programme and to inform the evaluation design. These documents were also used to develop the detailed question guides for the key informant interviews and the online survey.
The online survey was initially run for two weeks, from 3 to 17 March 2008, with a 46% response rate. The evaluation team extended the survey for another three weeks (until 7 April 2008), resulting in a small increase in the response rate.
Overall, a 51% response rate was achieved, which is above the average response rate6 for online surveys.
The evaluation team conducted semi-structured interviews with a range of stakeholders including organisation managers, directors, operational staff, board members and trustees, as well as assessors and Te Puni Kōkiri regional directors and kaiwhakarite. Interviews were held either individually or in group settings (two or more participants) within the organisations.
The interviews provided a ‘rich’ source of information on stakeholders’ perceptions and experiences of the SMG programme.
The qualitative information was analysed using a thematic approach noting the ‘significant’ findings from the interviews. The quantitative information was analysed using a descriptive statistical tool and the findings integrated with the key themes identified through the qualitative findings.
Information collected is presented as aggregated results and quotes are attributed anonymously to maintain the confidentiality of participants. However, where quotes are attributed to assessors or Te Puni Kōkiri staff, this is noted in brackets.
4 The draft outcomes model has not been validated by Te Puni Kōkiri. 5 Te Puni Kōkiri administrative contracts management database. 6 The literature noted that the average response rate for online surveys is between 30% and 40%. See Ritter & Sue (2007).
