Audit of the Bombardier CSeries Contribution Agreements

Final Audit Report May 2014

Audit and Evaluation Branch

Recommended for Approval to the Deputy Minister by the Departmental Audit Committee on June 17, 2014

Approved by the Deputy Minister on June 19, 2014

Table of Contents

• List of Initialisms and Acronyms Used in Report
• 1.0 Executive Summary
  • 1.1 Background
  • 1.2 Audit Objective and Conclusion
  • 1.3 Main Findings and Recommendations
  • 1.4 Audit Opinion
  • 1.5 Conformance with Professional Standards
• 2.0 About the Audit
  • 2.1 Background
  • 2.2 Objective and Scope
  • 2.3 Audit Approach
• 3.0 Findings and Recommendations
  • 3.1 Introduction
  • 3.2 Program risk management approach
  • 3.3 Performance measurement, monitoring and reporting
  • 3.4 Program internal controls over claims
  • 3.5 Safeguarding and handling sensitive information
  • 3.6 Management Response and Action Plan
• 4.0 Overall Conclusion
• Appendix A: Audit Criteria

List of Initialisms and Acronyms Used in Report

1.0 Executive Summary

1.1 Background

Industry Canada's Bombardier CSeries Program provided repayable contributions through two agreements to Bombardier Aerospace Inc. to fund the development of (i) generic technologies applicable to a variety of aircraft platforms, and (ii) technologies for a new fixed‑wing commercial aircraft, the Bombardier CSeries aircraft.

The Terms and Conditions of the Program, signed in 2008, provided funding up to $350 million over six years. As of December 31, 2013, the entire $350 million of committed funds were disbursed. The remaining project costs are being funded by Bombardier and multiple private sector and government organizations.

The program's repayable contributions provided funding for the development of new commercial aircraft technologies. Specifically, the objective was to encourage R&D that will result in:

• the development of generic technologies applicable to a variety of aircraft platforms including advanced materials, technologies and manufacturing processes; and,
• the development of technologies for a new fixed‑wing commercial aircraft, the Bombardier CSeries aircraft.

The Program is managed and delivered by Industry Canada's Aerospace, Defence and Marine Branch (ADMB) in the Industry Sector. The verification of claims was carried out by the Industrial Technologies Office (ITO) and the approval of claims for payment was performed by ADMB.

With the R&D phase largely complete, the Program will be entering the repayments phase, during which the Recipient (Bombardier) will repay Industry Canada. The Program is slated to be transferred to the Repayments and Recoveries Directorate (RRD) of the Corporate Management Sector by the time that the aircraft enters into service.

During the repayments phase, RRD will be responsible for ensuring compliance with repayment requirements, and collecting repayments.

The Bombardier CSeries aircraft family consists of narrow‑body, twin‑engine, medium‑range jet airliners. The CSeries has two models: the 110‑seat CS100 and the 135‑seat CS300. Bombardier expects the CSeries to burn 20% less fuel per trip than competitors, such as current versions of the Boeing 737 series, the Airbus A320 family, and the Embraer 195.

The CS100 took its first test flight on September 16, 2013.

1.2 Audit Objective and Conclusion

The objective of the audit was to provide assurance that the Program's governance, risk management and internal control processes support the delivery of the Program's mandate and priorities, specifically in the areas of risk management, performance measurement, program monitoring and claims process.

The scope of the audit included an assessment of the Program's activities, processes and controls in the specific areas identified in the audit objective. The audit covered Program activities and processes up to December 31, 2013.

The results of the audit revealed that, with exceptions, the Program's governance, risk management and internal control processes support the delivery of the Program's mandate and priorities. Improvements are needed to address low risk specifically in the areas of claims review, risk management practices and documentation of performance measurement, monitoring and reporting.

1.3 Main Findings and Recommendations

Program risk management approach

An approach to managing risks was established at the outset of the Program.

Risks are discussed, analyzed and monitored as part of ADMB's ongoing activities. There is an opportunity to strengthen risk management practices by implementing an approach to systematically update and document risk assessments and mitigation strategies.

Recommendation 1

The Director General of the Aerospace, Defence and Marine Branch should update its assessments of risk and mitigation strategies prior to transferring the file to the Corporate Management Sector.

Performance measurement, monitoring and reporting

A performance measurement strategy was established in November 2008 and outlined in the Results‑based Management Accountability Framework and Risk Based Accountability Framework (RMAF/RBAF) and later updated in the Performance Measurement (PM) Strategy Report (2013–14).

The Program collects information and reports outlined in the contribution agreements, with one minor exception.

The Program monitored the progress of projects. However, there is a lack of documentation to demonstrate the Program's review of information and regular monitoring against the performance indicators identified in the performance measurement strategy.

Recommendation 2

The Director General of the Aerospace, Defence and Marine Branch should ensure that a review of performance information against performance indicators identified in the PM Strategy is completed and documented prior to transferring the file to the Corporate Management Sector.

Program internal controls over claims

The required internal recipient claim forms and verification checklists were completed and included in the claims packages.

The audit found that all claims underwent a verification process. In some instances additional documentation and review could have demonstrated the application of the risk‑based methodology to claims review and improved the quality of the verification carried out.

Recommendation 3

For future contribution program claims, the Director General of the Aerospace, Defence and Marine Branch should ensure that the risk‑based methodology is applied and documented.

Safeguarding and handling sensitive information

Program Management understands the importance of managing sensitive information.

Program Management has undertaken activities to ensure that all employees understand the importance of managing potential conflicts of interest in accordance with the Industry Canada Values and Ethics Code.

1.4 Audit Opinion

In my opinion, overall, the Program's governance, risk management and internal control processes support the delivery of its mandate and priorities with exceptions noted.

1.5 Conformance with Professional Standards

This audit was conducted in accordance with the Internal Auditing Standards for the Government of Canada, as supported by the results of the Audit and Evaluation Branch's quality assurance and improvement program.

Susan Hart
Chief Audit Executive, Industry Canada

2.0 About the Audit

2.1 Background

In accordance with the approved Industry Canada (IC) 2013–2014 Multi‑Year Risk‑Based Audit plan, the Audit and Evaluation Branch (AEB) undertook an audit of Bombardier CSeries Program's contribution agreements.

The Canadian aerospace industry is one of Canada's largest employers, employing approximately 73,000 and contributing $12 billion to Canada's gross domestic product^{Footnote 1}. It plays a key role in driving innovation; aerospace firms are often early users of innovative technologies including composite materials, information and communications technologies, nanotechnology, and advanced manufacturing techniques.

As the world's third‑largest fixed‑wing commercial aircraft manufacturer, Bombardier Aerospace is an important member of the aerospace sector. The firm is a world‑leading manufacturer of regional and business aircraft, with 35,500 aerospace employees worldwide including approximately 20,000 in Canada.

Until recently, Bombardier had focused on building smaller regional aircraft with less than 100 seats. With the maturing of the smaller regional aircraft market, Bombardier made the strategic business decision to develop a new family of larger regional aircraft to compete in the 100–149 seat market.

Baseline seating will be 110 for the CS100 and 130 for the CS300. The new aircraft are expected to respond to market demand for quieter aircraft, lower fuel consumption, and lower operating costs per passenger mile. The new CSeries aircraft will have transcontinental capability and are expected to have 20% lower fuel consumption and 15% lower operating costs than current platforms available for purchase.

In May 2005, the Government of Canada announced a $350 million repayable contribution in support of the CSeries program, and in 2008, the Terms and Conditions of the Program were signed. The Terms and Conditions provided for reimbursement of eligible retroactive costs starting from May 1, 2005.

The Bombardier CSeries Program advances research and development in the aerospace industry and maintains and enhances the technology base and technological capabilities of Canadian aerospace firms. Specifically, the Program funds the development of generic technologies applicable to a variety of aircraft platforms, and development of technologies for a new fixed‑wing commercial aircraft, the Bombardier CSeries aircraft. The repayable contributions for R&D activities associated with the Program are expected to result in the development of new technologies, increase the competitiveness of Canada's aerospace sector and foster the growth of a competitive, knowledge‑based Canadian economy.

Two projects make up the CSeries program:

• The Generic Technologies Project: This project involves the development of a range of aircraft technologies (e.g., composite materials, fly‑by‑wire technologies, noise abatement technologies, etc.) that will be applied to the CSeries aircraft and to other aircraft platforms.
• The CSeries Project: This project includes the development of technologies that will contribute to the unique design and specifications of the CSeries aircraft family. This includes systems integration activities, systems testing, and specific process and production engineering efforts that apply only to the CSeries aircraft family.

The total project cost was funded by Bombardier and multiple private sector and government organizations. Under the Program, Industry Canada contributed $350 million to Bombardier for R&D over the six‑year period from 2008–09 to 2013–14. As of December 31, 2013, the entire $350 million of committed funds were disbursed. The Program is now entering the repayments phase, during which the Recipient (i.e. Bombardier) will repay Industry Canada.

Development of the CSeries aircraft is reaching its final stages. In September 2013, Bombardier successfully completed a test flight, and it aims to complete its testing program in 2014. The planes are now expected to enter into service in 2015.

The Program is managed and delivered by IC's Aerospace, Defence and Marine Branch (ADMB) in the Industry Sector. The verification of claims was carried out by the Industrial Technologies Office (ITO) and the approval of claims for payment was performed by ADMB.

The Office of the Auditor General of Canada (OAG) conducted an audit of Transfer Payments to the Aerospace Sector – Industry Canada in fall, 2012. The audit included the Bombardier CSeries Program and analyzed contribution agreements, files, records and reports. Program Management subsequently took steps to address the OAG's recommendations:

• Bombardier and ADMB clarified, in writing, the content and timing requirements of existing reporting;
• ADMB continues to review, and now documents its review, of the progress reports submitted with the claims; and
• ADMB maintains increased documentation of the results of the annual Performance Review Meeting with the recipient through meeting notes.

The audit team considered the objective, scope, findings and recommendations of the OAG audit during the planning phase and performed follow up assessment on the actions taken by management in response to the findings of the audit.

Now that the R&D phase is largely complete, the Program is slated to be transferred to the Repayments and Recoveries Directorate (RRD) of the Corporate Management Sector by the time that the aircraft enters into service. RRD manages repayable contribution agreements that are in the benefits (repayment) phase, and supports Industry Canada's compliance with the Treasury Board's Policy on Transfer Payments, issued under subsection 7(1) of the Financial Administration Act.

RRD's responsibilities in relation to monitoring and collection will include:

• Liaison with Recipient representatives;
• Verifying the Recipient's compliance with the contribution agreements (by monitoring reporting and repayment requirements and undertaking other follow up actions as required);
• Assessing risks associated with the contribution agreements;
• Reviewing and making recommendation on requests for amending the contribution agreements (if applicable); and
• Forecasting and collecting repayments and closing files when obligations are met.

2.2 Objective and Scope

The objective of this audit was to provide assurance that the Program's governance, risk management and internal control processes support the delivery of the Program's mandate and priorities, specifically in the areas of risk management, performance measurement, program monitoring and claims process.

The scope of the audit included an assessment of Program activities, processes and controls in the specific areas identified in the audit objective. The audit covered Program activities and processes up to December 31, 2013.

2.3 Audit Approach

The audit was conducted in accordance with the Internal Auditing Standards for the Government of Canada. Sufficient and appropriate audit procedures have been conducted and evidence gathered to support the accuracy of the conclusion and opinion provided and contained in this report. The opinion is based on a comparison of the conditions, as they existed at the time, against pre‑established audit criteria that were agreed on with management. The opinion is applicable only to the areas examined and within the scope described herein.

The audit was performed in three phases: planning, conduct and reporting. A risk assessment was conducted during the planning phase to confirm the audit objective and identify areas requiring more in‑depth review during the conduct phase. In addition to the risk assessment, the audit considered the Treasury Board Secretariat's Management Accountability Framework tool for assessing Core Management Controls (CMC).

Based on the identified risks and the CMC assessment, AEB developed audit criteria that linked back to the overall audit objective (refer to Appendix A).

The methodology used to address the audit's objectives included:

• Documentation review
• Interviews with key Program personnel
• File testing: 11 of 27 claim files were tested.

For sampling purposes, sample testing of controls over the claims review processes was carried out. The testing period covered completed claims submitted between May 8, 2010 and December 31, 2013. A sampling strategy was prepared documenting the nature and extent of testing to be performed. The sample size selected for the claims review process controls testing was based on the overall population size during the sampling period and the level of evidence required to conclude on the overall audit criteria.

Debrief meetings were held with ADMB and ITO management to validate the accuracy of the findings in the report.

3.0 Findings and Recommendations

3.1 Introduction

This section presents detailed findings from the audit of Bombardier CSeries Program's contribution agreements. The findings are based on evidence and analysis from both the initial risk assessment and the detailed audit work.

In addition to the findings below, AEB has communicated to management, either verbally or by management letter, findings for consideration that were non‑systemic, of low risk or not directly related to the audit objective and criteria.

3.2 Program risk management approach

The Treasury Board (TB) Framework for the Management of Risk defines risk management as a systematic approach to setting the best course of action under uncertainty by identifying, assessing, understanding, making decisions on and communicating risk issues. Effective risk management adds value as a key component of decision‑making, business planning, resource allocation and operational management.

Based on interviews and a review of program documentation, it was determined that the Program established an approach to managing risks at the outset of the Program. This approach included a risk management strategy for the Program which was outlined in the Results‑based Management and Accountability Framework and Risk Based Audit Framework (RMAF/RBAF). The RMAF/RBAF incorporated the Performance Measurement Strategy, a strategy for identifying and collecting information to monitor and evaluate program performance, and a risk‑based approach to monitor and manage risks associated with the two projects under the Program.

In the RMAF/RBAF the key risks associated with the Program's environment, design, implementation and achievement of results were identified and described and the risk level of each was identified. For each key risk, the issues and existing mitigating measures were examined and the likelihood and potential impact of the undesirable effect were assessed. An overview of key risk factors was also included in the Performance Measurement (PM) Strategy for the Program when it was updated in 2013–14.

The Program also prepared Project Summary Forms (PSF) in 2008 for both projects under the program. These forms, which are used to document the assessment of projects and information critical to project decisions, included an analysis of the risk factors associated with the projects.

Risk management considerations are incorporated in the strategies presented in the previously noted documents (RMAF/RBAF, PSF, PM Strategy) and are consistent with the TB Framework for the Management of Risk.

In a comprehensive risk management process, key program risks and mitigation strategies are assessed annually, at a minimum, and any decisions, changes or no changes, are documented. The audit assessed whether the program periodically reviewed risks and mitigation strategies affecting the program.

Program Management indicated that risks were communicated during meetings and briefings to senior management. The audit team reviewed a sample of these briefings, interviewed Program Management and noted that:

• Key risk factors, as they relate to the Recipient and to the project, were discussed during the annual site visits;
• Financial risks were monitored by the Program on a quarterly basis and reported to senior management;
• Order status updates were provided to senior management on a regular basis.

Through interviews with Program Management, it was identified that:

• risk management/assessment discussions were taking place within the Program;
• mitigation measures were adopted when two amendments were made to the contribution agreements (June 2012 and December 2013).

The audit noted that the program discussed, analyzed and monitored risks as part of ADMB's ongoing activities. The program did not systematically update and document its original risk assessment, which entails considering and documenting, on an annual basis, the risks originally identified in the RMAF/RBAF, any new risks and the continued relevance of the mitigation strategies. This would strengthen the risk management approach.

Recommendation 1

The Director General of the Aerospace, Defence and Marine Branch should update its assessments of risk and mitigation strategies prior to transferring the file to the Corporate Management Sector.

3.3 Performance measurement, monitoring and reporting

The TB Policy on Evaluation requires the Program to develop and implement ongoing performance measurement strategies so that sufficient information is available to effectively support evaluation of the Program.

A performance measurement strategy was developed as part of the RMAF/RBAF in November 2008. The Program updated the strategy in a Performance Management Strategy Report in 2013–14. These strategies are consistent with the elements of a performance measurement strategy outlined in TBS guidance.

The updated performance measurement strategy defines who has overall responsibility and accountability for the management and delivery of the Program during the Program's phases (research and development/construction phase, repayments phase), as well as which internal organizations provide other supporting roles. It includes the identification of performance indicators such as the number and type of research and development positions maintained and created under the agreements, as well as the number and type of collaborations with research institutions, universities and other companies. It focuses on the collection of relevant information to inform the assessment of the Program's achievements.

Under the TB Policy on Transfer Payments the administrative requirements of recipients are to be proportionate to the risk level. In particular, monitoring and reporting requirements should reflect the risks specific to the program.

For this Program, the RMAF‑RBAF reporting requirements reflected the Program risk level, and the reporting requirements were included in Schedule 5 of both contribution agreements. Based on a review of Recipient reporting, the following was noted:

• The Recipient provided performance measurement information through the Annual Information Update (AIU), and during annual Performance Review Meetings.
• At the annual Performance Review Meetings, the Recipient provided reports in presentation format that included the required reporting information.
• In response to a 2012 Office of the Auditor General (OAG) audit that included the Program, ADMB requested, and the Recipient agreed to provide, additional performance information in advance of the Performance Review Meetings.
• The Recipient made available the Annual Financial Statements within the required four‑month time frame. Program Management obtained these statements through the Recipient's website.
• The quarterly progress reports submitted with each claim did not include the required cash flow projections as required. However, the Recipient provided cash flow forecasts and projections for fiscal periods, with no attribution to specific claims, electronically when requested.

The Program fulfilled monitoring and reporting requirements in a number of ways.

• The Assistant Deputy Minister, Director General, Director and Program Manager have all participated in annual Performance Review Meetings and site visits. Notes of these meetings were kept by the Program.
• Program Management indicated, in interviews, that key performance indicators were reviewed during the annual site visit. However, there is no documentation of this review.
• The Program shares reporting information with senior management, which facilitates oversight and decision making. Financial information is provided to senior management through quarterly advice to the Deputy Minister.
• Debriefs to senior management cover the Program, the company as a whole, financial information and analysis about company financial results. They also include the order status for Bombardier CSeries planes.
• For the sample of claims tested that occurred prior to the 2012 OAG audit, the files contained limited documentation demonstrating that the Program had reviewed progress reports that accompanied the claims. Subsequent to the OAG audit, monitoring of information has improved as e‑mails to the Program Director with a brief summary of the claim progress reports demonstrate that claims and progress reports were reviewed before issuing payments.

The Program reviews reports provided by the Recipient but does not document the adequacy of the information, potential issues or any follow‑up actions taken. Also, there is a lack of documentation to support that the Program reviews the performance information provided in the Annual Information Update (AIU) against the targets set out in the performance measurement strategy.

More systematic assessment and documentation of performance against performance indicators identified in the PM Strategy would minimize the risk that the program would not be able to effectively measure, document and report on the benefits and outcomes resulting from the contribution agreements.

Recommendation 2

The Director General of the Aerospace, Defence and Marine Branch should ensure that a review of performance information against performance indicators identified in the PM Strategy is completed and documented prior to transferring the file to the Corporate Management Sector.

3.4 Program internal controls over claims

Internal controls over claims reduce the risk that the Program makes payments for ineligible claims. Interviews and documents reviewed demonstrated that ADMB has process flows and descriptions identifying key controls for the claim receipt, review, and approval process. These process documents also identify roles and responsibilities.

The internal form and checklist that ensure the completion of all required steps comprise a key component of the internal controls for this Program. To test for this control, a sample of claim files was selected and checked to determine if all required forms and checklists were present.

Based on the review of the sample files, every claims package included:

• A completed and signed claim submission form;
• A progress report, detailing work performed and progress achieved on the two projects under the Program;
• A completed and signed claim verification checklist indicating the review procedures carried out by the Verification Officer.

The verification process should be consistent with the TB Policy and Directive on Transfer Payments and serves as an internal control to ensure payments are in accordance with the Terms and Conditions of the contribution agreements. Any significant decisions, adjustments, reconciliations or exceptions to the normal process should be documented and supported by evidence.

The claims process normally includes reviewing summaries, supporting reconciliations, obtaining invoices for individual amounts above a determined threshold and verifying eligibility of costs which are reviewed based on the assessed level of risk. AEB was informed by management that the application of the verification process depends on the past experience and judgment of the reviewer and on the results of the analysis of previous claims. The invoices sampled for further verification are also based on the experience with the Recipient and represents the element of risk assessment to the program. For example, in the judgment of the reviewer, a history of issues could result in a larger or smaller sample size. Where the reviewer exercised judgment, the audit expected the reasons supporting the use of judgment would be documented.

Evidence of the claim verification process and the documented review of claimed amounts were noted during the audit. Although there is a claim processing checklist and guide, the audit noted situations where additional documentation of the methodology followed would have better demonstrated compliance to the prescribed process. For example, it was noted that the reasons for not always obtaining supporting documentation for amounts exceeding the verification threshold were not documented.

In some instances, decisions made during the verification of claims were not documented. For example, once documentation is received to support individual expenditures the decision made as to which expenditures, and how many, to sample for detailed verification purposes is not documented.

The audit also noted instances where further review could have improved the quality of the verification carried out. For example, where explanations were received from the Recipient to enquiries resulting from the claims review, there was not always a review of the explanations or evidence that the explanations were acceptable to Program officials.

Although all claims sampled during the audit were reviewed by the Program, without documentary evidence, there is a risk that the program may not be able to demonstrate that sufficient review was carried out to ensure payments were made for eligible expenditures. Risks are low, given that claims were reviewed and that the program plans to conduct a recipient audit.

Recommendation 3

For future contribution program claims, the Director General of the Aerospace, Defence and Marine Branch should ensure that the risk‑based methodology is applied and documented.

3.5 Safeguarding and handling sensitive information

As a knowledge‑based organization, IC relies on sound information management (IM) to achieve the Department's mandate and strategic objectives. IC's IM Governance and Accountability Framework and its Guide to the Identification, Categorization and Marking of Protected and Classified Information establishes the framework for the protection of sensitive information and provide guidance for the implementation of effective security measures throughout the Department. Training is available and provides additional guidance to employees in this area.

Through interviews, Program Managers indicated that they understand the importance of managing sensitive Recipient or Program information in accordance with IC's IM Governance and Accountability Framework and the Guide to the Identification, Categorization and Marking of Protected and Classified Information. Sensitive information includes the Recipient's confidential business, commercial and intellectual property information. Financial information, on the other hand, is not proprietary as the Recipient is a public company, and the quarterly financial results are available to the public.

According to the Values and Ethics Code for the Public Service, Deputy Ministers are required to determine the appropriate method for a public servant to comply with the Code in order to avoid conflicts of interest. The Code requires deputies to ensure that public servants in their organization are informed of the requirements of the Code on an annual basis.

Industry Canada's annual performance review process requires that all staff acknowledge that they are subject to the requirements of the Industry Canada Values and Ethics Code and will continue to comply with these terms and conditions of employment, including the disclosure of any real, apparent or potential conflict of interest situations.

Through interviews, Program Management indicated to the audit team that all individuals involved in managing the Program had participated in the mandatory training and understood the importance of managing potential conflicts of interest in accordance with IC's Values and Ethics Code.

3.6 Management Response and Action Plan

The findings and recommendations of this audit were presented to ADMB management. Management has agreed with the findings included in this report and will take action to address all applicable recommendations by September 2014.

ADMB will update assessments of risk and mitigation strategies and review and document performance information against performance indicators identified in the PM Strategy prior to transferring the file to the Corporate Management Sector. ADMB will also document and share with the management team key lessons learned with respect to the application of risk‑based methodologies for claims review in order to inform decisions on future contributions agreements.

4.0 Overall Conclusion

The results of the audit revealed that, with exceptions, the Program's governance, risk management and internal control processes support the delivery of the Program's mandate and priorities.

Improvements are needed to address low risk specifically in the areas of claims review, risk management practices and documentation of performance measurement, monitoring and reporting.

Appendix A: Audit Criteria