Part 1: Introduction

Reporting the progress of defence acquisition projects.

1.1
By this time in 2008, we expected to have produced a report for Parliament providing assurance about how effectively the Ministry of Defence (the Ministry) and the New Zealand Defence Force (the NZDF) were managing acquisition projects. We wanted to provide this assurance by carrying out a performance audit to identify and report changes to costs, time frames, and essential user requirements for specific projects managed by the Ministry and the NZDF (the defence agencies). However, we had difficulty getting all of the necessary information, particularly from the Ministry’s information systems, and it was not possible to complete the audit.

1.2
On 20 March 2008, we briefed the Foreign Affairs, Defence and Trade Committee (the Committee) about these difficulties and proposed a new approach. This interim report is part of that new approach – it sets out what we have found so far, and describes why the existing reporting by the defence agencies is problematic. It also describes how we will work with the defence agencies during the next two financial years to encourage them to make the changes needed to improve the quality, transparency, and usefulness of their reporting about defence acquisition projects.

1.3
This interim report does not make any conclusion about how well the defence agencies are managing specific defence acquisition projects, or on the causes or justifications for any changes in costs, time frames, and essential user requirements. We do not have enough information or the right information to form a judgement on that larger question. The purpose of the audit was to improve the quality of reporting by the defence agencies, not to assess the quality of the decisions made in managing the projects.

1.4
In this Part, we describe:

  • the purpose and approach of our original audit;
  • the change in our approach; and
  • the structure of this interim report.

Purpose and approach of our original audit

1.5
We expect the defence agencies to monitor and be able to report on the progress of major defence acquisition projects as part of its accountability to Ministers, Parliament, and other stakeholders. We started to audit 10 of the largest and highest-priority defence acquisition projects against that expectation. The projects are listed in the Defence Long-Term Development Plan (LTDP) that the defence agencies regularly publish.

1.6
The purpose of our audit was to identify and report changes to costs, time frames, and essential user requirements for the 10 defence acquisition projects. The selected projects, and their forecast acquisition costs (excluding GST) according to the 2006 LTDP, were:

  • Medium Utility Helicopter, $771 million;
  • Multi-Role Vessel and Patrol Vessels (Project Protector), $500 million;
  • P-3 Systems Upgrade, $373 million; • C-130 Life Extension, $234 million;
  • Boeing 757 Acquisition and Modification, $220 million;
  • Training/Light Utility Helicopter, $110 million;
  • Light Operational Vehicle, $93 million;
  • Medium Range Anti-Armour Weapon, $24 million;
  • Improvised Explosive Device Disposal, $22 million; and
  • Very Low Level Air Defence Alerting and Cueing System, $14 million.

1.7
We were particularly interested in measuring the progress of these projects against the costs, time frames, and essential user requirements at two main project approval points (that is, the “Approval to Commence” and the “Approval to Commit” points in the defence acquisition process).

Approval points for defence acquisition projects

1.8
The process for acquiring defence capability is set out in the New Zealand Defence Capability Management Framework (CMF). The defence agencies first implemented the CMF in May 2004 and last updated it in March 2008. There are five stages (including four approval points) described in both the 2004 and 2008 versions of the CMF:

  1. Ministerial Note;
  2. Approval to Initiate;
  3. Approval to Commence;
  4. Approval to Negotiate; and
  5. Approval to Commit.

1.9
Eight of the 10 selected projects were given Approval to Commence before the 2004 CMF was implemented. However, the 2004 CMF set out practices and procedures that had been in place since mid-2002, after which most of the projects obtained Approval to Commence.

1.10
We consider that the 2004 and 2008 versions of the CMF are not materially different in their requirements, particularly the requirement for accurate estimates at the Approval to Commence point. Both versions require the defence agencies to submit to Cabinet estimates of cost and time at the Approval to Commence point that have a high level of confidence. By this approval point, the 2004 CMF states that:

…risk should have been sufficiently reduced and the project should have reached a sufficient degree of maturity for user and systems requirements, [expected introduction into service date] and costs to be set with confidence. (Note that [the Approval to Commence] for major projects could be used for external reporting and performance measurement of the [defence agencies].
…
The highest acceptable cost will normally be set at the 90% confidence level. This cost and the expected cost of the demonstration and manufacture phases will be derived from a three point cost estimate.

1.11
In comparison, the 2008 CMF says:

…accurate cost estimation (to 90% level of confidence) is critical at the [Approval to Commence] stage. Moreover, investment proposals should set out the costs of the project derived from a three-point estimate (high, low, and expected cost).

1.12
We have used the 2008 version as our source because it describes the defence agencies’ current required acquisition process. Paragraphs 1.13-1.17 describe the stages of the process for acquiring defence capability in more detail, reflecting the requirements from the 2008 CMF.

Ministerial Note

1.13
The defence agencies identify new defence acquisition projects through the Defence Strategic Plan process and list the projects in either the LTDP (which looks ahead 10 years) or the Long Range Capability Forecast Review (which looks more than 10 years ahead). Cabinet is asked to note and approve the projects’ listing. To begin one of the listed projects, the Minister of Defence is notified through a Ministerial Note about such matters as the capability being considered, the expected level of capital and operating funding required and available, and the proposed year of delivery and entry into service. The Minister is asked to note that the defence agencies are to start assessing and identifying options for meeting the capability in more detail.

Approval to Initiate

1.14
The NZDF is then primarily responsible for defining the capability required, and getting Cabinet Approval to Initiate a project to acquire the capability. Cabinet is presented with options for delivering the desired capability and asked to approve the defence agencies initiating a capability project and conducting a detailed assessment of those options. At this point, the defence agencies can consult with suppliers to clarify the range of options to meet the capability need, to provide information on emerging technologies, and to provide information that will support the development of robust cost estimates. Once that work is completed, the defence agencies then seek Cabinet Approval to Commence the acquisition.

Approval to Commence

1.15
The CMF describes the Approval to Commence point as the “Main Gate”. At this point, the defence agencies seek Cabinet approval on the basis of the Main Gate Investment Case, which describes in detail the proposal for investment, supported by links to strategy, quantified risk analysis and options for mitigating risk, cost-benefit analysis, an implementation or procurement plan, and a specification of expected performance. The CMF states that accurate cost estimation is critical at this point. Cabinet notes the estimated costs, and sometimes public announcements on the details of the project are made at this point.

1.16
The Approval to Commence gives approval for the acquisition phase to begin. During this phase, the Ministry’s Acquisition Division acquires the equipment component of the capability and the NZDF acquires the other functional components of the capability (such as personnel and infrastructure).

Approval to Negotiate and Approval to Commit

1.17
The Approval to Commit point is when the defence agencies seek final Cabinet approval to commit funds for the project before signing a contract with a supplier. Between the Approval to Commence and Approval to Commit points, the Ministry’s Acquisition Division carries out activities including tendering, acquisition risk management, tender evaluation, and contract negotiation. Depending on the nature and complexity of the project, the Acquisition Division may seek an Approval to Negotiate with a preferred tenderer. When evaluations of tenders and negotiations with potential contactors are complete, the defence agencies seek Approval to Commit. Once this approval is given, the contract is awarded and the Acquisition Division manages the contract until the NZDF accepts the contract deliverables and services.

1.18
The defence agencies work together throughout these five stages, but the balance of responsibility changes from the Approval to Commence point. Before that point, the NZDF does most of the work. After that point, the Ministry’s Acquisition Division is solely responsible for the equipment acquisition phase until the NZDF accepts the equipment. Once the NZDF accepts the equipment, it is responsible for bringing the equipment together with the functional components of the capability that it had acquired, to introduce the capability into service.

1.19
We used the process described in the CMF as the basis for our original audit and set out to:

  • compare estimated costs, time frames, and essential user requirements at the Approval to Commence point with those at the Approval to Commit point, and identify the reasons for the changes; and
  • compare the latest forecasts of costs, time frames, and essential user requirements with those approved at the Approval to Commit point, and identify the reasons for the changes.

1.20
We expected the defence agencies to be able to report the reasons for changes in estimated costs, time frames, and essential user requirements from those defined at the Approval to Commence point.

1.21
We expected the tender and negotiation process to cause some changes to estimates between the Approval to Commence and Approval to Commit points. Because the CMF states that accurate cost estimation is critical at the Approval to Commence point, we expected that these changes would not be significant.

1.22
During our audit, we were told by the defence agencies that the cost estimates submitted to Cabinet at the Approval to Commence point had not been set with the high level of confidence described in the CMF. The defence agencies told us that they can only make general enquiries of potential suppliers before the Approval to Commence point to support the development of robust estimates, as described in the CMF. This means that the submitted estimates were, at best, an “intelligent guess”.

1.23
We expected there to be few changes to costs, time frames, and essential user requirements after the Approval to Commit point. The defence agencies advised us that the forecast costs and time frames at the Approval to Commit point are much more accurate because the Ministry has been able to get costed proposals from potential suppliers and is about to award a firm fixed-price contract.

Problems with the information about each acquisition project

1.24
We sought information from the defence agencies on cost and time frame changes and changes to essential user requirements for all of the selected projects. We wanted to present information in our report on:

  • the extent to which estimates changed between the Approval to Commence and Approval to Commit points, and why they changed;
  • the total and average cost changes since the Approval to Commit point, the variations making up the totals, and the reasons for them (including any common reasons);
  • the total and average time frame changes since the Approval to Commit point, the variations making up the totals, and the reasons for them (including any common reasons); and
  • the extent to which essential user requirements at the Approval to Commit point were met, and the reasons for any changes to those requirements.

1.25
Our audit work mainly involved reviewing project files and documents held by the Ministry and the NZDF at Defence Headquarters in Wellington. For the most part, we focused on information held by the Ministry. We tried to complete a standard set of information for each acquisition project. This proved extremely difficult and time-consuming. It led to a number of queries and information gaps, which we tried to resolve by working closely with the defence agencies’ staff.

1.26
However, it became clear that the Ministry’s project monitoring and reporting systems in particular were not able to readily produce much of the detailed information needed to explain changes to costs, time frames, and essential user requirements for each project. This was particularly so for historical information about projects that have spanned several years.

1.27
When the defence agencies’ systems were introduced, they were not designed to monitor the progress of projects in the way that we set out to. Nevertheless, we expected the sort of information we were looking for to have been more readily available and easier to extract.

Change in our approach

1.28
We advised the Committee in March 2008 that it was not possible for us to report accurately and completely to Parliament in the way we originally intended. Even if we continued the time-consuming work to extract the project information, it would be subject to so many assumptions and qualifications that its value and usefulness would be questionable.

1.29
We told the Committee that we would produce this interim report to explain in more detail the problems we had, and set out our proposed approach to working with the defence agencies to resolve them. We are committed to finding a way to provide effective assurance to Parliament on this area of spending.

1.30
The Treasury has carried out a Capital Asset Management Review to assess the effectiveness of capital asset management regimes, practices, and performance in government departments and Crown entities. We will ensure that any work we do on a revised reporting framework aligns with the Treasury’s work.

Structure of this interim report

1.31
Part 2 provides a high-level analysis of cost and time frame changes for the 10 selected acquisition projects. This analysis builds on the information provided by the Ministry to the Committee in December 20071 and by the NZDF to us during our audit. We do not report on changes to the essential user requirements, because of the difficulty we had in getting that information. In preparing our analysis, we checked the information reported by the Ministry to the Committee in December 2007 against Cabinet approvals and LTDPs, where appropriate.

1.32
Part 3 summarises the difficulties we encountered, particularly with the Ministry’s information systems. These difficulties have prevented us from identifying and properly reporting the details of – and reasons for – the changes to project costs, time frames, and essential user requirements.

1.33
Part 4 presents our planned new approach – how we will work to encourage the defence agencies to improve their reporting on the progress of defence acquisition projects.

1.34
In this performance audit, we did not verify the accuracy of the individual financial transactions within the procurement phase of each of the selected acquisition projects. We review a sample of those transactions as part of the annual financial audit of the Ministry. This performance audit examined other aspects of the Ministry’s reporting on the projects in more detail than we have done in the financial audit. The results of the respective audits can be different, depending on their scope and level of detail.

1.35
We do not include recommendations in this interim report, because we have not yet formed a view on the best way for the defence agencies to address the problems we have identified. We will complete the work needed to do this as part of the first stage of our planned new approach.

1 Ministry of Defence response to the Foreign Affairs, Defence and Trade Committee’s Supplementary Questions from the Ministry’s Financial Review 2006/07 (response dated 7 December 2007), pp 5-6.

page top