Audit of the Strategic Aerospace and Defence Initiative – Phase I (Final Audit Report)

This publication is available online at https://ic.gc.ca/eic/site/ae-ve.nsf/eng/h_03841.html.

To obtain a copy of this publication or an alternate format (Braille, large print, etc.), please fill out the Publication Request Form at www.ic.gc.ca/Publication-Request or contact:

Web Services Centre
Innovation, Science and Economic
Development Canada
C.D. Howe Building
235 Queen Street
Ottawa, ON K1A 0H5
Canada

Telephone (toll-free in Canada): 1-800-328-6189 risk-based approach is applied to recipient monitoring and reporting.
TTY (for hearing-impaired): 1-866-694-8389
Business hours: 8:30 a.m. to 5:00 p.m. (Eastern Time)
Email: ic.info-info.ic@canada.ca

Permission to Reproduce

Except as otherwise specifically noted, the information in this publication may be reproduced, in part or in whole and by any means, without charge or further permission from Innovation, Science and Economic Development Canada, provided that due diligence is exercised in ensuring the accuracy of the information reproduced; that Innovation, Science and Economic Development Canada is identified as the source institution; and that the reproduction is not represented as an official version of the information reproduced, nor as having been made in affiliation with, or with the endorsement of, Innovation, Science and Economic Development Canada.

For permission to reproduce the information in this publication for commercial purposes, please fill out the Application for Crown Copyright Clearance at www.ic.gc.ca/copyright-request or contact the Web Services Centre mentioned above.

© Her Majesty the Queen in Right of Canada, as represented by the Minister of Innovation, Science and Economic Development Canada, (insert year of publication).

Cat. No. Iu4-209/2017E-PDF

ISBN 978-0-660-07558-7

For publications produced separately in English and French, add the following: Aussi offert en français sous le titre French publication title in italics.

Table of contents

List of initialisms and acronyms used in report

A&D
Aerospace, Defence, Space and Security
AEB
Audit and Evaluation Branch
CA
Contribution Agreement
FAA
Financial Administration Act
ISED
Innovation, Science and Economic Development Canada
ITO
Industrial Technologies Office
R&D
Research and Development
RRD
Repayments and Recoveries Directorate
SADI
Strategic Aerospace and Defence Initiative
TB
Treasury Board

1.0 Executive summary

1.1 Introduction

Launched in 2007, the Strategic Aerospace and Defence Initiative (SADI) is one of the largest contribution programs at Innovation, Science and Economic Development Canada (ISED). SADI provides repayable contributions to support industrial research and pre-competitive development (R&D) projects in the Canadian aerospace, defence, space and security (A&D) industries.

SADI's objectives are to:

  • encourage strategic R&D that will result in innovation and excellence in new and improved products, services and processes;
  • enhance the competitiveness of Canadian A&D companies; and,
  • foster collaboration between research institutes, universities, colleges, and the private sector.

The delivery of SADI comprises the following three phases:

  1. Application Phase
    ITO conducts due diligence to help guide project funding decisions.
  2. R&D (Work) Phase
    Company performs R&D, submits claims and receives SADI assistance.
  3. Repayment (Benefits) Phase
    R&D is complete and the company makes repayments according to the terms of the Contribution Agreement.

In 2007, SADI was provided an initial budget of $900 million over five years. Budget 2013 announced that the Government would continue to provide stable funding for SADI, close to $1 billion over five years. To date, $1.32 billion in authorized assistance has been approved for 39 SADI projects.

SADI is delivered and managed by the Industrial Technologies Office (ITO), a Special Operating Agency housed in the Science and Innovation Sector of ISED.

1.2 Audit objective and scope

The objective of this audit was to provide assurance that SADI's management control framework is adequately designed and implemented to support the delivery of SADI.

The audit scope was SADI's control framework in place at the time of conducting this audit, and was limited to an assessment of control design and implementation in the following areas:

  • Recipient risk management and monitoring
  • Claims verification procedures

The audit of SADI is being conducted in two phases given a recent transfer of repayment files from the Corporate Management Sector to ITO. Phase I (this audit) focuses solely on the application and R&D phases of the program. Phase II (timing to be determined) will focus on the management of repayments for the SADI program. This phased approach will allow for the reintegration of repayment files into ITO.

1.3 Overview of audit results

Strengths

ITO has established a strong framework to support the delivery of SADI. Key controls are designed and in place to support good management practices in the areas of recipient risk management and claims verification.

There are processes in place to ensure that proposed projects meet ITO's established evaluation criteria. As part of the due diligence process, a detailed project risk assessment is conducted that considers a variety of risk factors and helps guide funding decisions. ITO also uses both standard and targeted risk mitigation measures to address risks for each project.

ITO has established procedures to support the day-to-day administration of SADI with defined roles and responsibilities. Steps have also been taken to ensure that a risk-based approach is applied to recipient monitoring. Through its risk mitigation and recipient monitoring efforts, ITO is able to monitor recipient progress and performance on a regular basis.

Areas for Improvement

The audit identified opportunities for ITO to enhance the existing framework through revisions to its current approach to recipient risk management and monitoring.  Specifically, within the application phase, ITO should update program documents to reflect how the compliance risk factor is currently being assessed, and use an overall risk indicator that reflects the assessment of all individual risk factors to determine the initial level of monitoring required in the work phase.

In addition, ITO may be able to achieve operational efficiencies by aligning some of its reporting requirements (i.e. recipient progress reporting required as part of claim submissions) to be more risk-based. In doing so, ITO may also be able to alleviate some of the administrative burden currently placed on recipients.

1.4 Audit opinion and conclusion

In my opinion, with some exceptions, ITO has established and implemented controls and processes to support the delivery of SADI. Opportunities for improvement and associated recommendations were identified in the areas of recipient risk management and claims verification.

1.5 Management response

Management has agreed with the findings included in this report and will take action to address all recommendations by June 30, 2017.

1.6 Statement of conformance

This audit was conducted in accordance with the Internal Auditing Standards for the Government of Canada, as supported by the results of the Audit and Evaluation Branch's quality assurance and improvement program.

space to insert signature

Michelle Gravelle
A/Chief Audit Executive, Innovation, Science and Economic Development


2.0 Background

2.1 Program context

In the global marketplace, research and development (R&D) is a key driver of economic growth, and innovative companies are more likely than others to be part of that growth. R&D allows Canada to compete in new markets and industries, and helps Canadian businesses offer their customers new or improved products, processes and services. With annual R&D investments of more than $1.6 billion each year, the aerospace sector is the second most research-intensive industry in Canada.  

2.2 Program overview

The Strategic Aerospace and Defence Initiative (SADI) was launched in 2007, and is one of the largest contribution programs at Innovation, Science and Economic Development Canada (ISED). SADI provides repayable contributions for industrial research and pre-competitive development projects in the Canadian aerospace, defence, space and security industries (A&D).

SADI's objectives are to:

  • encourage strategic R&D that will result in innovation and excellence in new and improved products, services and processes;
  • enhance the competitiveness of Canadian A&D companies; and,
  • foster collaboration between research institutes, universities, colleges, and the private sector.

Industrial research: refers to planned research or critical investigation aimed at discovery of new knowledge, with the objective that such knowledge may be useful in developing new products, processes or services, or in bringing about a significant improvement to existing products, processes or services.

Pre-competitive development: refers to the translation of industrial research findings into a plan, blueprint or design for new, modified or improved products, processes or services whether intended for sale or use, including the creation of a first prototype. It may further include the conceptual formulation and design of products, processes or services and of initial demonstration or pilot projects.

Through the delivery of SADI, ISED supports and encourages R&D investment in the private sector. This helps Canadian companies achieve technological breakthroughs and increase their competitiveness, which, in turn, contributes to economic, technological and other benefits for Canadians.

SADI is managed and delivered by the Industrial Technologies Office (ITO), a Special Operating AgencyFootnote 1 of ISED. ITO is led by an Executive Director and is made up of five directorates, employing 75 FTEs that manage various aspects of ITO's three main programs: SADI, the Technology Demonstration Program and the Post-Secondary Institutions Strategic Investment Fund.  ITO also manages legacy programs including the Technology Partnerships Canada Program and the Defence Industry Productivity Program.

Program Delivery Life Cycle

The delivery of SADI comprises the following three phases:

Application phase: This phase involves three main steps for approving applications for funding:

  • Submission of the proposal: ITO officers screen each application received to ensure that the application contains adequate information upon which to start a due diligence review, and that the applicant and project meet the eligibility requirementsFootnote 2.
  • Due diligence review: ITO officers conduct a detailed review of each project proposal to ensure it meets ITO's evaluation criteria. Through an assessment of various risk factors, (e.g. market, technical, managerial capability, etc.), ITO evaluates the applicant's capability to achieve the economic and social benefits that it expects will result from the R&D activities. During this step, site visits by ITO officials and third-party technical and market experts may be conducted.
  • Project approval: Funding decisions are made and applicants are informed of the status of their proposal. The Minister will seek Treasury Board (TB) and Cabinet approval prior to authorizing contributions in excess of $50 million. Following project approval, a contribution agreement (CA) will be prepared for signature outlining the legally binding responsibilities and obligations of both the Crown and the recipient, at which point the applicant becomes a recipient. The recipient is responsible for managing the project, submitting claims and reporting on progress and benefits.

Application processing time is six months for applicants requesting less than $50 million. For smaller applicants (i.e. companies with less than 100 employees and applications requesting less than $2 million), ITO has committed to a four-month service standard.

R&D (Work) phase: During this phase, ITO monitors recipient progress towards meeting the contractual requirements outlined in the CA, including the outcomes and benefits associated with the project.  It is expected that the company will undertake strategic R&D activities as part of the process of transforming ideas, technology and expertise into marketable products, processes and services.

As part of the monitoring process, ITO will conduct site-visits, review recipient progress reports and verify claims for reimbursement. ITO normally reimburses 40% of eligible costs submitted for a SADI project, while not exceeding the maximum limit of 75% government funding (federal, provincial, territorial, municipal). Projects in the R&D phase typically last up to five years, and are sensitive to changing economic conditions.   

Repayment (Benefits) phase: Once projects are ready for commercialization, they move to the repayment phase with repayments commencing two years following the R&D phase. During this phase, in addition to ISED collecting repayments, ITO prepares repayment forecasts and monitors the recipients' overall progress in achieving expected outcomes.

There are two different types of repayment terms for SADI projects. If repayment is unconditional, then there is a defined payment schedule. If repayment is conditional, then repayment is based on the recipient's gross business revenues. In both cases, repayments are typically repaid over a 15 year period.

Program Resources & Project Portfolio

SADI was provided an initial budget of $900 million over five years (2007-08 to 2011-12). Budget 2013 announced that the Government would continue to provide stable funding for SADI, close to $1 billion over five years.

Since its launch, SADI has authorized $1.32 billion in assistance to 39 projects. Within the current portfolio, authorized assistance varies from approximately $0.3 million to $300 million.

SADI Highlights as of November 1, 2016:

  • Total # of active projects: 33
  • Projects in the R&D (work) phase: 17
  • Projects in the repayments phase: 16
  • Total authorized assistance for all projects: $1.32 billion
  • Total disbursements to date: $1.0 billion

SADI is accessible to Canadian incorporated companies of all sizesFootnote 3, located in Canada, that perform R&D in the country. Currently, large companies account for the majority of total authorized assistance provided, with the aerospace sector receiving the largest share.  

Reviews

Emerson Review of Aerospace and Space Programs and Policies

In 2011, the Government of Canada announced that it would initiate "a comprehensive review of all policies and programs related to the aerospace/space industry to develop a federal policy framework to maximize the competitiveness of this export-oriented sector and the resulting benefits to Canadians." This Review was led by the Honourable David Emerson, who produced his two volume report at the end of 2012. The report's seventeen recommendations to optimize federal support for aerospace and space focused on innovation, market access and supplier development, skills and procurement.

For SADI, it was recommended that funding levels be maintained at existing levels, and that SADI modify its terms and conditions to make it a more effective program for stimulating the development of the aerospace and space technologies of the future. Following the review, changes were made to SADI that included expediting the application process for smaller recipients and increasing the share of the total eligible project costs covered by SADI from 30% to 40%.

Auditor General's Report on Transfer Payments to the Aerospace Sector

In fall 2012, the Office of the Auditor General released its audit report on transfer payments to the aerospace sector. SADI was among the transfer payment programs included in the scope of the audit. The Auditor General found that most aspects of aerospace transfer payment programs were managed appropriately. Some recommendations for the Department were raised and have since been implemented.

Evaluation of SADI

ISED's Audit and Evaluation Branch (AEB) is currently evaluating SADI to assess overall program relevance and performance. The evaluation will be completed by March 31, 2017.

2.3 Audit background (objective, scope and methodology)

In accordance with the approved ISED 2015-16 to 2017-18 Multi-Year Risk-Based Audit Plan, AEB undertook an audit of SADI.

Audit Objective

The objective of this audit was to provide assurance that SADI's management control framework is adequately designed and implemented to support the delivery of SADI.

Audit Scope

The audit scope was SADI's control framework in place at the time of conducting this audit, and was limited to an assessment of control design and implementation in the following areas:

  • Recipient risk management and monitoring
  • Claims verification procedures (to support the reasonableness of claims submitted)

The audit of SADI is being conducted in two phases given a recent transfer of repayment files from the Corporate Management Sector to ITO.Footnote 4 Phase I (this audit) focuses solely on the application and R&D phases of the program. Phase II (timing to be determined) will focus on the management of repayments for the SADI program. This phased approach will allow for the reintegration of repayment files into ITO.

Methodology

The audit was conducted in accordance with the Internal Auditing Standards for the Government of Canada. Sufficient and appropriate audit procedures have been conducted and evidence gathered to support the accuracy of the conclusion and opinion provided and contained in this report. This opinion is based on a comparison of the conditions, as they existed at the time, against pre-established audit criteria that were validated with management. This opinion is applicable only to the areas examined and within the scope described herein.

The audit was performed in three phases: planning, conduct and reporting. A risk assessment was executed during the planning phase to confirm the audit objective and identify areas requiring more indepth review during the conduct phase. In addition to the risk assessment, the audit considered the Treasury Board Secretariat's Management Accountability Framework tool for assessing core management controls.

Based on the identified risks, AEB developed the audit criterion and sub-criteria that linked back to the overall audit objective (see Appendix A).

The methodology used to address the audit's objective included:

  • Review of ITO process documentation;
  • Conduct of nine interviews with ITO personnel in the Operations area; and
  • Conduct of three walk-throughs in the areas of recipient risk assessment (application phase), project monitoring and claims verification (work phase).

All of the audit evidence gathered through the above noted processes was synthesized, analyzed and supports the audit findings presented throughout this report.

A debrief meeting was held with ITO senior management on October 4, 2016 to validate the accuracy of the findings contained in this report. This meeting also provided the auditee an opportunity to offer any additional information/clarification regarding the findings.

3.0 Findings and recommendations

3.1 Introduction

This section presents detailed findings from the audit of SADI. The findings are based on evidence and analysis from both the initial risk assessment and the detailed audit work.

In addition to the findings below, AEB has communicated to management findings for consideration that were either non-systemic or not directly related to this audit's objective and criteria.

3.2 Project risk assessment

ITO has a strong framework in place to assess project risks during the application phase. However, ITO's assessment of the compliance risk factor differs from how it is described in program documents. Further, an overall risk indicator that reflects the assessment of all individual risk factors is not used to determine the baseline frequency of monitoring required in the work phase.

During the application phase, ITO performs due diligence on each project application to ensure that proposed projects meet ITO's established evaluation criteria. As part of the due diligence process, ITO conducts a project risk assessment that considers a variety of risk factors. For some risk factors (e.g. technical, market opportunity, project financing), ITO officers are required to engage subject matter experts for their input.

The individual risk factors assessed by ITO have been described in the SADI Risk Rating Guide (the Guide) and consider the following:

  • Management Capability – if the company can be reasonably expected to secure and sustain the requisite level of managerial capability to achieve the stated objectives.
  • Company Financial Stability – the company's expected financial position over the duration of the project.
  • Compliance – the company's capability of complying with the CA terms and conditions.
  • Market Opportunity – if there is a clear understanding of the market problem/opportunity that the planned products/services will address.
  • Technical – if the project is technically feasible and if the company has the technical capability to successfully complete the project.
  • Project Financing – if the company can be reasonably expected to secure the financial resources to finance its share of the project (work phase) costs.

The detailed assessment and corresponding risk score for each of the individual factors is currently documented within a project summary form. The assessment is used by ITO to help guide funding decisions and determine if targeted risk mitigation measures need to be built into the CA.

While there is a strong framework in place for assessing project risks, the compliance risk factor is not being assessed in the application phase as it is currently described in the Guide. The current assessment focuses largely on past history and ITO's ongoing relationship with a recipient, which is ITO's intent in the application phase.

Further, only the compliance risk factor is currently being used to determine the baseline frequency of monitoring (i.e. project management reviews) required in the work phase. An overall risk indicator, which reflects the assessment of all risk factors (e.g. management capability, technical, market opportunity), would be more appropriate to determine the initial level of monitoring. It should be noted that as part of the project risk assessment, ITO establishes an overall risk rating that considers all individual risk factors assessed in the application phase. However, this indicator is currently designed to only be relied upon to determine a project's repayment terms.

In light of the above, the compliance risk factor, as it is currently described is not representative of how it is assessed in the application phase. In addition, the baseline frequency of monitoring performed for each project in the work phase may not be appropriate, since it is not being driven by an overall risk indicator.  

Recommendation 1

The Executive Director, ITO, should update program documents to reflect how the compliance risk factor is currently being assessed in the application phase. Further, ITO should use an overall risk indicator that reflects the assessment of all individual risk factors to determine the baseline frequency of monitoring in the work phase.

3.3 Risk mitigation & recipient monitoring

ITO uses a variety of risk mitigation measures to address individual risk factors. There is also an approach in place that allows ITO to regularly monitor recipient progress and performance and take corrective action when necessary during the R&D phase.

As part of effective risk management, it is essential that appropriate mitigation measures are identified and implemented to respond to identified project risks. It is also important that recipients be monitored on a regular basis to ensure project success and identify changes to the risk environment so that corrective action can be taken where necessary.

The audit noted that during the application phase ITO conducts a detailed assessment of project risk factors. The assessment of each risk factor can result in the following actions being taken:  

  • Rejecting an application due to intolerable risk;
  • Mitigating the risk by including special conditions and/or reporting requirements in the CA; or
  • Accepting the risk.

ITO uses a variety of risk mitigation measures to address individual risk factors. Each recipient is subject to standard monitoring activities and reporting requirements (e.g. site visits, progress reporting, etc.) outlined in the CA. In addition, targeted risk mitigation measures deemed necessary may be included in the CA (i.e. special conditions and/or reporting requirements). In the work phase, officers will also determine if any additional measures or actions need to be taken based on the initial risk assessment performed during the application phase.

ITO also has various mechanisms in place to monitor all risk mitigation measures identified. For example, the ITO database allows for tracking of standard monitoring activities and reporting requirements, while special conditions identified in the CA are monitored as part of the review of each claim and recipient progress report. In addition, monitoring officers are required to update the assessment of individual risk factors (initially assessed during the application phase) on an annual basis or sooner, if new information is made available or a significant event impacts the risk of the project.

Through its risk mitigation and recipient monitoring efforts, ITO is able to monitor recipient progress and performance on a regular basis. Changes in risk levels or project issues identified through on-going recipient monitoring could result in modifications to the monitoring approach or additional mitigation measures being identified. These are brought to the attention of ITO management on a regular basis, who provide advice and direction on how project risks and issues are to be addressed.

Finally, it was noted that ITO has established procedures to support the day-to-day administration of the program. Key monitoring activities (e.g. site visits, review of recipient progress reports and claims) are supported by documented processes and guidelines.

3.4 Risk-based approach to recipient monitoring and reporting

While ITO has taken steps to ensure that a risk-based approach is applied to recipient monitoring, there is an opportunity for ITO to align its reporting requirements to reflect the value of funding and risk profile of each recipient.

The TB Policy and Directive on Transfer Payments require that a risk-based approach be applied to recipient monitoring and reporting. In particular, monitoring and reporting requirements should reflect the value of funding and the risk profile of the recipient.

In light of the TB Policy and Directive requirements, and in response to a recommendation made by the Auditor General in the 2012 Fall Report on Transfer Payments to the Aerospace SectorFootnote 5, ITO has developed a Guideline on Risk-based Monitoring and Reporting. The guideline outlines that the levels of monitoring required be proportionate to the risk level and the value of funding. To illustrate, ITO conducts project management reviews (i.e. site visits or teleconferences) with the recipient to review project progress. At a minimum, each recipient is subject to at least one review annually. The need for additional reviews is dependent on the level of funding and compliance risk rating for each recipient.  

While progress has been made towards addressing the Policy and Directive requirements, the audit identified an opportunity to apply a risk-based approach to recipient progress reporting requirements related to claim submissions. Currently, recipients must provide a detailed progress report with each claim submitted. However, the extent of progress reporting required is the same for each recipient and does not vary based on the recipient risk level and value of funding.

Adopting a risk-based approach to recipient progress reporting required as part of claim submissions could help alleviate some of the administrative burden currently placed on recipients without significantly impacting ITO's ability to monitor recipient progress. Furthermore, this approach could help improve resource utilization, since a significant portion of time is spent reviewing recipient progress reports.

Recommendation 2

The Executive Director, ITO, should review recipient progress reporting requirements related to claim submissions to ensure that they reflect the value of funding and risk profile of recipients.

ITO has established procedures in place to support claims verification. However, there is no formal consideration of recipient risk that helps determine the level of monitoring during claims verification.

Claims verification is another form of recipient monitoring used by ITO to process claims. The purpose behind verifying claims is to help ITO meet its obligations under the Financial Administration Act (FAA).Section 34 of the FAA requires that an authorized officer certify that a recipient is entitled to the payment before a payment is made. It is expected that officers, when reviewing claims, verify that amounts claimed are reasonable and in line with the CA.  

ITO's approach to verifying claims submitted by recipients is outlined in its Claims Process document. The document defines the roles and responsibilities of all officers involved in the review of claims and recipient progress reports. It also provides an overview of general procedures to be performed to verify each cost category claimed (i.e. direct costs, labour and overhead). General procedures for claims verification include:

  • Reviewing claim supporting schedules for mathematical accuracy
  • Substantiating direct costs (e.g. materials) claimed to supporting invoices
  • Reviewing invoices to ensure that invoices are in line with the claim period
  • Comparing hourly rates charged against previous labour claims
  • Verifying the correct application of the overhead rate, as outlined in the CA

While general procedures have been defined, ITO does not formally consider recipient risk, and instead relies on officers' use of professional judgement to determine the extent of procedures performed (e.g. comparing hourly rates charged for labour in the current claim for a sample of employees against previous claims, as opposed to performing the same procedure for all employees). Professional judgement exercised is driven by officers' knowledge of recipients based on past claims and information acquired through on-going communication with recipients. The extent of analysis performed and follow up with recipients may vary from project to project, depending on the officers' conclusions.  Guidance and direction is provided by management, and there is an informal review of officers' work to help determine the adequacy of procedures performed.

In the case of direct costs, ITO has established financial thresholdsFootnote 6 for required invoices that recipients must submit to support claims. While thresholds established are based on total eligible costs, recipient risk is not formally considered.

As outlined in the TB Policy on Transfer Payments, risk should be a factor in determining the extent of monitoring performed. While recipient risk is considered informally through the use of professional judgement by ITO officers, formal consideration of recipient risk would provide a more systematic and clear baseline for officers to conduct their verification of claims. The current design of claim verification procedures may lead to instances where the extent of verification may not be appropriate relative to the recipient risk level.

Recommendation 3

The Executive Director, ITO, should adjust claims verification procedures to allow for formal consideration of recipient risk, which would help determine the appropriate level of verification required for each recipient.

3.5 Management response and action plan

The findings and recommendations of this audit were presented to the Acting Executive Director of ITO and the Acting Senior Director of Strategic Planning and Management Services at ITO. Management has agreed with the findings included in this report and will take action to address all recommendations by June 30, 2017.

Notably, the Executive Director, ITO will take actions that include:

  • updating program documents to re-name and describe how the compliance risk factor is currently being assessed in the application phase
  • using an overall risk profile to determine the baseline frequency of monitoring in the work phase
  • aligning progress reporting requirements related to claim submissions to the value of funding and recipient risk profile
  • determining the extent of claims verification required commensurate to defined risk levels

All changes and updates will be documented and communicated to staff.

4.0 Overall conclusion

The results of this audit revealed that ITO has a well-established framework to support the delivery of the SADI program. Key controls are designed and in place to support good management practices in the areas of recipient risk management and monitoring and claims verification.

The audit identified some exceptions and corresponding opportunities to enhance the existing framework in the following areas:

Project Risk Assessment

  • ITO conducts a project risk assessment that considers several risk factors. However, ITO's assessment of the compliance risk factor in the application phase differs from how it is described in program documents. Furthermore, an overall risk indicator that reflects the assessment of all individual risk factors is not used to determine the baseline frequency of monitoring required in the work phase.

Risk-based Approach to Recipient Monitoring and Reporting

  • ITO has taken steps to ensure that a risk-based approach is applied to recipient monitoring. However, there is opportunity to apply this approach to recipient progress reporting required as part of claim submissions, to ensure that the reporting requirements are reflective of the value of funding and risk profile of each recipient.
  • ITO officers, as part of the claims verification process, exercise professional judgment in determining the extent of their review and analysis of claims submitted to ensure that amounts claimed are reasonable and in line with the CA. However, there is no formal consideration of recipient risk to help determine the extent of verification required.

Appendix A: Audit criteria & sub-criteria

Audit criteria

  • 1. ITO utilizes an approach to program delivery that considers and mitigates program risks.

Sub-criteria

  • 1.1 Appropriate recipient risk factors are considered and monitored on a regular basis during the R&D phase.
  • 1.2 Appropriate capacity exists and resources are utilized effectively to monitor recipient progress and performance.
  • 1.3 Claims verification procedures are appropriately designed to support ITO's assessment on the reasonableness of claims submitted.

Management Response and Action Plan

A - For inclusion in the report

The findings and recommendations of the SADI audit were presented to ITO management. Management has agreed with the findings included in this report and will take action to address all applicable recommendations by June 30th, 2017.

B - For follow-up purposes - Detailed actions to address the recommendations in the report

Recommendation 1

The Executive Director, ITO, should update program documents to reflect how the compliance risk factor is currently being assessed in the application phase. Further, ITO should use an overall risk indicator that reflects the assessment of all individual risk factors to determine the baseline frequency of monitoring in the work phase.

Planned Action on the Recommendation

The Executive Director will update program documents, where applicable, to re-name and describe the compliance risk factor to appropriately reflect how it is currently being assessed in the application phase.

Further, the Executive Director will:

  1. Rely on the overall risk profile to determine the baseline frequency of monitoring in the work phase; and
  2. Review the current weighting of individual risk factors used to determine the overall risk profile in the application phase to ensure it is appropriate to inform the baseline frequency of monitoring.

Context: Though program officials assess a number of risk factors in the application phase as well as develop an overall risk profile, only past experience is currently utilized as a baseline to inform the initial level of monitoring required during the work phase.

Responsible Official (position)

Director, Investment Management, and Director, Portfolio Management

Target completion date

June 30, 2017

Recommendation 2

The Executive Director, ITO, should review recipient progress reporting requirements related to claim submissions to ensure that they reflect the value of funding and risk profile of recipients.

Planned Action on the Recommendation

The Executive Director will review the current recipient progress reporting required as part of claim submissions and align the extent of reporting requirements to the value of funding and recipient risk profile. In doing so, the Executive Director will minimize the associated progress reporting required as part of claim submissions. This change in approach will be documented within program procedures.

Context: According to SADI terms and conditions, "each claim must be accompanied by a brief report of the work completed and details of all costs being claimed". Beyond the above constraints, flexibility exists with progress reporting and this can be more explicitly linked to the value of funding and recipient risk profile.

Responsible Official (position)

Director, Portfolio Management and Manager, Claims Verification

Target completion date

June 30, 2017

Recommendation 3

The Executive Director, ITO, should adjust claims verification procedures to allow for formal consideration of recipient risk, which would help determine the appropriate level of verification required for each recipient.

Planned Action on the Recommendation

The Executive Director will adjust the Claims Verification procedures to allow for formal consideration of risk by:

  1. Defining recipient risk levels to be used in claims verification;
  2. Determining the extent of claims verification procedures required for each cost category commensurate to the defined risk levels; and
  3. Outlining a management oversight role.
Responsible Official (position)

Director, Portfolio Management and Manager, Claims Verification

Target completion date

June 30, 2017