Evaluation of the Sustainable Development Technology Fund

From: Innovation, Science and Economic Development Canada

March 2018

Presented to the Performance Measurement and Evaluation Committee on March 29, 2018

Approved by the Deputy Minister on April 16, 2018

Table of contents

Evaluation of the Sustainable Development Technology Fund[PDF - 590 KB]

• Executive summary
• 1.0 Introduction
  • 1.1 Report Overview
  • 1.2 SDTC
  • 1.3 SDTC Funding
  • 1.4 SD Tech Fund
  • 1.5 Governance and Administration
  • 1.6 Stakeholders
  • 1.7 Logic Model
• 2.0 Methodology
  • 2.1 Evaluation Approach
  • 2.2 Objective And Scope
  • 2.3 Evaluation Issues and Questions
  • 2.4 Data collection methods
  • 2.5 Limitations
• 3.0 Findings
  • 3.1 Relevance
  • 3.2 Performance
  • 3.3 Efficiency and Economy
• 4.0 Conclusions and recommendations
  • 4.1 Relevance
  • 4.2 Performance
  • 4.3 Efficiency and Economy
  • 4.4 Recommendations
• Appendix A: Technology readiness levels scale
• Management Response and Action Plan
• Summary of the Evaluation of the Sustainable Development Technology Fund

List of acronyms

AEB

Audit and Evaluation Branch

CA

Contribution Agreement

CTCGB

Clean Technology and Clean Growth Branch

BDC

Business Development Bank of Canada

EDC

Export Development Canada

GAC

Global Affairs Canada

GHG

Greenhouse Gas

ISED

Innovation, Science and Economic Development Canada

IT

information technology

JAM

Joint Account Management

MOU

Memorandum of Understanding

Mt CO2e

Megatonnes of Carbon Dioxide Equivalent

NRCan

Natural Resources Canada

OAG

Office of the Auditor General of Canada

PPCQ

Post-Project Completion Questionnaire

R&D

Research and Development

SCC

Standards Council of Canada

SDTC

Sustainable Development Technology Canada

S&E

Screening and Evaluation

SME

Small- or Medium-Sized Enterprise

SOI

Statement of Interest

SIPS

Strategy and Innovation Policy Sector

TRL

Technology Readiness Level

List of tables

• Table 1: SD Tech Fund Summary of Project Status Since Inception in 2001 (up to March 31, 2017)
• Table 2: commercialization Status of SD Tech Fund Projects

List of figures

• Figure 1: SD Tech Fund Logic Model
• Figure 2: Sector Characteristics of Completed SD Tech Fund Projects that Demonstrated (N=164)
• Figure 3: Commercial Status of all SD Tech Fund Projects by Sector (N=128)
• Figure 4: Key Issues that Prevented Completed Projects from Reaching Commercial Market (N=30)
• Figure 5: Key Issues that Prevented Terminated Projects from Reaching Commercial Market (N=48)
• Figure 6: Types OF Follow-ON Financing Received by SD Tech Fund Recipients Since 2001 ($ millions)

Executive summary

Program overview

SDTC is an arm's length, not-for-profit organization created in 2001 by the Government of Canada through the Canada Foundation for Sustainable Development Technology Act.  Its goal is to support projects that develop and demonstrate new technologies that address issues related to climate change, air quality, clean water and clean soil.

SDTC's SD Tech Fund provides non-repayable financial contributions and operational support for the development and demonstration of Canadian cleantech projects to facilitate the movement towards scale-up for commercialization.

The target population for the SD Tech Fund is Canadian companies:

• Developing a new and novel technology with significant and quantifiable environmental benefits that will provide a significant retained Canadian benefit following execution of the project;
• With a defined project and looking to demonstrate a pre-commercial technology;
• With a strong end user value proposition and able to validate the market and commercialization potential of the technology; and
• Who form a consortium that includes at least one other partner (partners who are either end users or able to validate the need and market for the technology are recommended).

According to SD Tech Fund administrative data, $893 million has been invested to support 341 cleantech projects since 2001.

Innovation, Science and Economic Development (ISED) is responsible for managing transfer payments and overseeing the administration of the SD Tech Fund Funding Agreement with SDTC to help ensure that government policies and regulations are observed.

Evaluation purpose and methodology

The objective of this evaluation was to address the core issues of relevance and performance in accordance with the Policy on Results and address certain issues identified by ISED's Clean Technology and Clean Growth Branch (CTCGB) and SDTC.  Specifically, the evaluation focuses on program relevance, the Fund's progress toward its intermediate and long-term outcomes, and efficiency and economy.

The scope of the evaluation included 218 projects categorized as completed or terminated between April 1, 2005 and March 31, 2017.  The evaluation employed seven data collection methods, including a literature review; document review; financial, administrative and performance data review; statistical analysis of administrative data; interviews; case studies; and a survey of SD Tech Fund recipients.

Findings

Relevance

Multiple lines of evidence show that there is a continued need for the SD Tech Fund to support cleantech producers to further develop and demonstrate their technologies, as it directly addresses the ongoing pre-commercial funding gap.  SDTC's due diligence process is valued by investors, who appreciate SDTC's process for evaluating technical and project management competence.

Performance

Evidence shows that recipient firms are forming collaborative partnerships across the innovation chain, due in large part to the consortium model of the SD Tech Fund.  In addition, SDTC is actively building and maintaining partnerships across the innovation chain including provincial and federal organizations to facilitate collaboration and sharing of information to support cleantech producers.

Approximately three-quarters of all projects that received SD Tech Fund support have made progress towards demonstration or have successfully demonstrated.  The key factors related to successful demonstration include the availability of financing and the formation of partnerships.  Numerous barriers toward demonstration exist, significantly impacting the progress of about one-third of projects.

Although the mandate of the SD Tech Fund ends at demonstration, at the time of the evaluation, about one-third of SD Tech Fund recipients subsequently reached the market (i.e. commercialize), with larger projects and those receiving other government support having a higher probability of success.  The ability to raise follow-on capital following successful demonstration remains the main barrier to commercialization.  The development of relationships with other partners may help to identify opportunities to enhance commercialization prospects for SD Tech Fund projects.

The SD Tech Fund plays an important role in the government's mission to achieve its GHG emission goals and enable Canadian firms to compete globally in the cleantech space.  Available data and primary research for this evaluation indicates that the SD Tech Fund has helped create environmental and economic benefits for the Canadian economy.

Efficiency and Economy

SDTC has a governance structure in place which supports the effective delivery of the SD Tech Fund.  It has recently implemented several improvements to address findings from recent audits and reviews, although further work is still required to fully implement the recommendations.

Evidence clearly shows that SDTC is efficiently and effectively delivering the SD Tech Fund, although some concerns exist over the timeliness of the application and contracting process.

Currently, there are a range of government support programs for cleantech projects across the innovation chain.  To access support from these programs, cleantech producers may be required to provide similar information that can take time and resources.  In addition, there are a range of organizations collecting post-demonstration information from cleantech producers.

Recommendations

1. CTCGB should encourage SDTC to continue to work with other partners to develop relationships to enhance the identification of opportunities for cleantech producers to progress towards commercialization, including connecting firms with international opportunities/supply chains and sources of follow-on funding.
2. CTCGB should encourage SDTC to work with the Clean Growth Hub to address some of the limitations of the administrative data, including developing standards to enable consistency of the data and collecting information related to key performance indicators, as well as environmental and economic benefits.
3. CTCGB should encourage SDTC to fully implement the recommendations stemming from a recent review of program governance.
4. CTCGB should work with SDTC to identify opportunities to streamline the application and contracting process to improve contracting timelines.
5. CTCGB should encourage SDTC to investigate other information-sharing and streamlined reporting options, including collaboration with other data collection and funding entities, with the aim of minimizing duplication and reducing the administrative burden on companies.

1.0 Introduction

1.1 Report Overview

This report presents the results of an evaluation of the Sustainable Development Technology Fund (SD Tech Fund).  The SD Tech Fund is administered by the Canada Foundation for Sustainable Development Technology, commonly known as Sustainable Development Technology Canada (SDTC).  The purpose of the evaluation is to assess the relevance and performance of the SD Tech Fund.  The report is organized into four sections:

• Section 1 provides the program context, description, delivery and governance, and stakeholders of the SD Tech Fund;
• Section 2 presents the evaluation methodology along with a discussion of evaluation limitations;
• Section 3 presents the findings pertaining to the evaluation issues of relevance and performance; and
• Section 4 summarizes the study's conclusions and provides recommendations.

1.2 SDTC

SDTC was established in 2001 by the Government of Canada, through the Canada Foundation for Sustainable Development Technology Act, to fund the development and demonstration of new technologies that promote sustainable development.^{Footnote 1}  It is an arm's length, not-for-profit (non-share capital) organization that was created to support projects that develop and demonstrate new technologies that address issues related to climate change, air quality, clean water and clean soil.
The objectives of SDTC are to:

• Fund the development and demonstration of new sustainable development technologies related to climate change, clean air, clean water, and clean soil to make progress toward sustainable development;
• Foster and encourage innovative collaboration and partnerships with the private sector, academia and not-for-profit organizations to channel and strengthen the Canadian capacity to develop and demonstrate sustainable development technologies with respect to climate change, clean air, clean water and clean soil; and
• Ensure timely diffusion by funded recipients of new sustainable development technologies in relevant market sectors throughout Canada.^{Footnote 2}

To meet these objectives, SDTC strives to fill the gap between proven research and advancement to a commercial product or service.

1.3 SDTC Funding

Initially, $550 million was provided to SDTC as three separate up-front grants for the SD Tech Fund – $100 million in 2001, $250 million in 2003 and $200 million in 2005.  Subsequent funding for the SD Tech Fund was provided as conditional transfer payments in Budget 2011 ($40 million), Budget 2013 ($325 million) and Budget 2016 ($50 million), bringing the Government of Canada's total investment to $965 million.  Budget 2017 allocated an additional $400 million, specifically for the re-capitalization of the SD Tech Fund, starting in 2017-18.

According to SDTC reporting, since its launch in 2001, SDTC has invested an estimated $987 million to support 347 cleantech projects through the SD Tech Fund (341 projects, $895 million) and Next Generation Biofuels Fund (6 projects, $92 million), and has leveraged more than $2.7 billion from other project partners.^{Footnote 3} In total, these investments have contributed to 226 patents held by SDTC funded cleantech entrepreneurs.

1.4 SD Tech Fund

The SD Tech Fund supports the development and demonstration of technologies to a scale that would allow the technology to move to commercialization, including the development of prototypes, demonstration of technology systems and processes, and the scaling up and piloting of technologies.  Projects with a technology readiness level (TRL)^{Footnote 4} of between 3 and 7 are considered eligible for SD Tech Fund support.  Although there are numerous supports for cleantech producers, the SD Tech Fund is considered to be the major source of government financial support to help bring cleantech projects to the demonstration stage.

The target population for the SD Tech Fund is Canadian companies:

• Developing a new and novel technology with significant and quantifiable environmental benefits that will provide a significant retained Canadian benefit following execution of the project;
• With a defined project and looking to demonstrate a pre-commercial technology;
• With a strong end user value proposition and able to validate the market and commercialization potential of the technology; and
• Who form a consortium that includes at least one other partner (partners who are either end users or able to validate the need and market for the technology are recommended).

To apply for SD Tech Fund funding, a company must first submit a statement of interest (SOI) for assessment by SDTC.  If the SOI is successful, the firm is invited to submit a detailed proposal for evaluation through the screening and evaluation (S&E) process, which includes due diligence, a site visit by SDTC's S&E experts, and a review by external experts.  Following this, the proposal is shared with and reviewed by SDTC and, if ultimately approved by the Board, the applicant is moved through to contracting and enters into a formal agreement with SDTC by signing the Contribution Agreement (CA).

As Table 1 shows, according to the administrative data, the SD Tech Fund alone has invested approximately $893 million to support 341 projects.

Source: SDTC Administrative Data, September 2017

1.5 Governance and Administration

Governance of SDTC

SDTC is governed by a Board of Directors, which reflects the broad interests of the public, private and academic sectors in Canada.  The Board comprises 15 directors, seven of whom are appointed by the Governor in Council, including the Chair.  The remaining eight members are appointed by the SDTC Member Council which is comprised of individuals from the public, private and academic sectors.

There are five Board committees: the Audit Committee, Corporate Governance Committee, Human Resources Committee, Next Generation Biofuels Fund Project Review Committee, and SD Tech Fund Project Review Committee.  Both the Chair and the President/Chief Executive Officer serve on all Board committees as ex-officio, non-voting members.

The members of the Foundation consist of 15 industry leaders, who are appointed by other members in accordance with the Act.  Members provide informed and representative perspectives on the subject of sustainability.

Administration of the SD Tech Fund

SDTC is responsible for the administration of the SD Tech Fund in accordance with the investment guidelines per the Funding Agreement with ISED.  Among other things, SDTC reviews and selects project proposals and makes recommendations to the Board, and enters into contractual agreements with eligible recipients to provide funding for eligible projects.

ISED is responsible for managing transfer payments and overseeing the administration of the SD Tech Fund Funding Agreement with SDTC to help ensure that government policies and regulations are observed.

1.6 Stakeholders

The main groups of external stakeholders/consortium partners (i.e., those who have an interest in the projects supported by the SD Tech Fund) include the following:

• Canadian companies that provide private sector leverage funding for projects;
• Federal government departments/agencies (e.g., Export Development Canada (EDC), Business Development Bank of Canada (BDC), Global Affairs Canada (GAC) and provinces that provide public sector leveraged funding for projects;
• Private-sector investors who provide follow-on financing^{Footnote 5} by way of venture capital, access to public markets, and asset/debt financing, including international investment; and
• Recipient companies for the technologies that achieve commercialization, which explore or adopt the technologies for their own operations.

1.7 Logic model

The logic model (Figure 1) depicts the activities to be pursued, the outputs to be produced and the expected outcomes resulting from the delivery of the SD Tech Fund and how its objectives are to be achieved.

Figure 1: SD Tech Fund Logic Model

2.0 Methodology

This section provides information on the evaluation approach, objective and scope, the specific evaluation issues and questions that were addressed, the data collection methods, and limitations for the evaluation.

2.1 Evaluation Approach

This evaluation was managed and conducted by ISED's Audit and Evaluation Branch (AEB) and adopted a goal-based approach, addressing the program's stated outcomes. 

To maximize the use of evaluation resources, wherever possible, the findings of this evaluation built on findings reported in previous evaluations and reviews.^{Footnote 6}  For this evaluation, data collection efforts focused on addressing the information gap relating to the progress toward the longer-term SD Tech Fund outcomes.

2.2 Objective and Scope

An evaluation of the SD Tech Fund is required in accordance with the Treasury Board Policy on Results and addresses certain issues identified by ISED's Clean Technology and Clean Growth Branch (CTCGB) and SDTC.  Specifically, the evaluation focuses on the progress of the SD Tech Fund toward its intermediate and long-term outcomes (although program relevance and efficiency and economy are also examined).  Given the focus of the evaluation, all 218 projects categorized as completed or terminated by March 31, 2017 were included in the evaluation scope.

2.3 Evaluation Issues and Questions

Based on the program's Performance Measurement Strategy,^{Footnote 7} and subsequent consultations with the program, the evaluation addressed the following questions:

Relevance

1. To what extent does the SD Tech Fund address a demonstrable and unique need?

Performance

2. To what extent are recipients forming collaborative partnerships across the innovation chain?
3. To what extent are SD Tech Fund projects making progress towards demonstration and eventually demonstrating?  What factors influence demonstration?
4. To what extent are SD Tech Fund projects commercializing?  What factors influence commercialization?
5. To what extent are SD Tech Fund projects and/or technologies contributing to the realization of environmental and economic benefits?

Efficiency and Economy

6. Does the governance structure support efficient and effective delivery?
7. How efficient and effective is SDTC in the delivery of the SD Tech Fund? 
8. Are there alternative approaches that would increase the cost-effectiveness of the SD Tech Fund?

2.4 Data collection methods

Multiple lines of evidence were used to address all evaluation questions.  The data collection methods included a literature review; document review; financial, administrative and performance data review; statistical analysis of administrative data; interviews; case studies; and a survey of SD Tech Fund recipients.

AEB conducted most of the lines of evidence, with contractors conducting the case studies and the survey.  The statistical analysis of the SD Tech Fund's administrative data on outcomes was conducted by ISED's Strategy and Innovation Policy Sector (SIPS).

Literature Review

The review of academic literature determined if the SD Tech Fund continues to address a demonstrable and unique need.  Trends and issues in sustainable development technologies in Canada and around the world were also examined.  Further, the literature review looked at how governments in other countries support cleantech in order to gain insight as to whether there are differences or alternative approaches to funding sustainable development technologies.

Document Review

The document review was conducted to gain an understanding of the SD Tech Fund, its alignment with government priorities, its role in Canada's Innovation and Skills Plan, and its achievement of expected outcomes.  Further, it provided insight into the achievement of program objectives and shed light on modifications made to the program since the previous evaluation in 2012. 
The document review included:

• Program foundational documents (e.g., Canada Foundation for Sustainable Development Technology Act, Funding Agreements, Terms and Conditions);
• Program reporting documents (e.g., Corporate Plans, Annual Reports);
• Project files (e.g., CAs, recipient final reports, recipient post-project completion questionnaires, memoranda of understanding);
• Previous internal and external reviews (e.g., audits, evaluations, industry and horizontal reviews); and,
• Government priority-setting documents (e.g., Budgets, Speeches from the Throne, mandate letters).

Financial, Administrative and Performance Data Review

The data review addressed questions on relevance and performance, as well as efficiency and economy.  In particular, it looked at the SD Tech Fund's contributions to the development of sustainable technologies, the number of projects making advancements towards successful demonstration, and how efficiently and economically the SD Tech Fund is delivered. 

The data review also focused on the progress of the SD Tech Fund toward expected outcomes, as well as the SD Tech Fund budgets, expenditures and leveraged funds.  This analysis looked at the efficiency of the internal processes of the program, as well as the burden placed on recipient organizations.

Statistical Analysis of Administrative Data

A quantitative analysis was conducted to identify factors that contribute to the successful completion and demonstration of cleantech innovation projects.  It also identified factors that served as barriers for projects that were not as successful.

Interviews

The objective of the interviews was to gather in-depth information including views, perceptions and factual information that address the evaluation questions.  The interviews were semi-structured and designed to obtain qualitative feedback from a range of respondents.  In total, 22 individual and group interviews were conducted with the following stakeholder groups:

• ISED management and staff (3);
• SDTC Board of Directors (1);
• SDTC Executives (1);
• SDTC Project Managers and staff (2);
• Cleantech experts/private equity (6);
• Other government departments and agencies^{Footnote 8} (5); and
• Other levels of government and regional organizations^{Footnote 9} (4).

Case Studies

As part of the evaluation, nine targeted case studies of SD Tech Fund recipients were conducted to collect in-depth information on SD Tech Fund projects that were completed or terminated early.  The case studies were derived from the 218 SD Tech Fund projects to be broadly representative of the population of projects based on criteria such as region, sector, completion date, size of recipient organization, and size of the SD Tech Fund contribution.  Each case study involved a review of the relevant project files^{Footnote 10} and a telephone interview.

Survey of SD Tech Fund Recipients

A survey of past SD Tech Fund recipients was utilized to collect outcome information from as many firms that received support as possible.  The survey of recipients captured quantitative and qualitative evidence to complement existing data by asking recipients questions pertaining to program design and delivery and the achievement of outcomes.  The survey was conducted by telephone interview with an online option if preferred.  Based on a sample of 127 companies, representatives from 61 cleantech producers participated in the survey.

2.5 Limitations

Gaps in the Administrative Data

There are some gaps in the administrative data which inhibit a complete and accurate measurement of program outcomes in an efficient manner.  For example, the administrative database provided for this evaluation (for projects completed prior to March 31, 2017) did not contain any information on some of the key performance indicators required for the measurement of outcomes indicated in the Performance Measurement Strategy.  Further, the utility of the data was limited due to missing, incomplete, out-of-date, and/or contradictory information.  To overcome the limitations of the administrative data, other lines of evidence were used to help enhance the administrative data information as much as possible.

There are also challenges in calculating commercialization rates and environmental and economic benefits due to a lack of rigorous and reliable data.  One reason is that long-term tracking is a challenge, as it can take five years or more (after demonstration) before any progress on commercialization is made, which is beyond the three-year post-project tracking activities of SDTC.  Separately, the 2017 Office of the Auditor General of Canada (OAG) report noted that its audit of the SD Tech Fund did not find documented evidence of SDTC's Project Review Committee's consideration of environmental benefits when recommending projects to the Board for approval.^{Footnote 11}  The report recommended that SDTC clearly document the assessment of potential GHG emission reductions that it undertakes when approving projects that have reductions as an intended long-term outcome – to which SDTC agreed.

The information shortcomings are known and recognized (Budget 2017 provided $14.5 million to NRCan and ISED to establish a Clean Technology Data Strategy) and work has started on developing more robust approaches and methods to enhance information.  In addition, collaborations are being established to facilitate data sharing across departments.  Further, the Clean Growth Hub will work with SDTC to identify best practices, ensure consistency of data and address barriers to sharing data and information.

Attribution

The ability to attribute long-term outcomes to SD Tech Fund activities or support is a challenge as it is difficult to control for intervening variables.  For example, the likelihood of recipient firms to reach demonstration is impacted not only by financial and non-financial SD Tech Fund support, but also by factors such as a firm's management team and their ability to execute.  Therefore, while the SD Tech Fund plays an important role in the ability of a firm to achieve their outcomes, full attribution cannot be assigned to the SD Tech Fund alone.

Survey Fatigue of SD Tech Fund Recipients

Cleantech producers receive many requests for information, but often have limited resources to respond.  The evaluation found that most provincial development organizations, federal departments and at least two cleantech-focused organizations regularly survey cleantech producers to monitor their progress/results.  Survey fatigue of SD Tech Fund recipients has an impact on one of SDTC's key data collection tools, the annual post-project completion questionnaires (PPCQs) required in each of the first three years post-project funding.  More than one-half of recipient firms did not submit any information via the questionnaires.  A key impact of this trend is that SDTC is not receiving all post-project information from recipient firms and must use a range of other techniques to report on economic and environmental impacts.

It is possible that the burden of having to respond to an evaluation survey may have negatively impacted survey response rates.  Several measures were taken to minimize this impact: (i) diversifying the questions from what was previously asked; (ii) keeping the survey questions to the minimum number required for addressing the evaluation questions; and (iii) inviting as many recipients in the sample as possible to help alleviate any non-response bias.

Access to past research data

SDTC and the SD Tech Fund have been evaluated on three previous occasions and audited twice to fulfill the requirements set out in past funding agreements.  AEB hoped to minimize duplication, build on past findings, and contribute to new knowledge.  However, the extent to which this could be undertaken was limited by the inaccessibility to previous research tools and data.  In order to ensure the rigour of this evaluation, similar questions/measures were used across methodologies to ensure consistency and minimization of any biases.

3.0 Findings

3.1 Relevance

3.1.1 To what extent does the SD Tech Fund address a demonstrable and unique need?

The evaluation examined whether there is an ongoing need for government support for pre-commercial development and demonstration in the cleantech space and whether the SD Tech Fund is meeting that need.

Cleantech is a critical space from an environmental standpoint and contributes to sustainable development, which the Federal Sustainable Development Strategy for Canada, 2016-2019 broadly defines as "development that meets the needs of the present without compromising the ability of future generations to meet their own needs".  In a 2016 report by Environment and Climate Change Canada, 13 goals to support sustainable development were identified, four of which are addressed by the SD Tech Fund (effective action on climate change, clean growth, clean energy, and clean drinking water).^{Footnote 12}

The importance of these four SD Tech Fund sustainable development goals cannot be overstated.  For example, climate change can lead to rising sea levels that threaten coastal communities, more frequent and severe wildfires and pest outbreaks, and extreme weather events such as storms and heat waves.  Further, the impact on Indigenous communities can be particularly harsh, as melting sea ice, snow cover and glaciers can disrupt travel routes and reduce the quality of foods in the North.  As well, in terms of clean growth, the benefits of cleantech include lower GHG emissions and increased resilience to climate change, healthier communities, and more sustainable and competitive economic sectors.^{Footnote 13}

While cleantech is clearly an important space from an environmental standpoint, it is also vital from an economic standpoint – it is a space that is estimated to grow to $3 trillion by 2020, making it the third largest economic sector in the world.^{Footnote 14}   If Canada maintains its share of the global market, the cleantech space is projected to employ over 125,000 people in Canada by 2020.  As well, over 80% of Canada's cleantech small- or medium-sized enterprises (SMEs) are exporters and are nine times more likely to export than similarly-sized SMEs in other sectors.^{Footnote 13}

Findings from the SD Tech Fund recipient survey underscore the importance of government support in meeting these sustainable development challenges.  In the survey, 93% of firms said SD Tech Fund support was extremely or very important for addressing financial challenges associated with the development and demonstration of pre-commercial cleantech projects.  Further, 85% reported that their project would not have gone forward without SD Tech Fund support (the remaining 15% indicated that aspects of their project would have changed in the absence of funding).  The Fund serves to de-risk the investment in cleantech and encourage/facilitate contributions from private investors and other partners.  Additionally, case studies determined that while there are many sources of public sector funding for research, there are few sources to support the development and demonstration of pre-commercial cleantech projects.

Findings from the case studies, survey and interviews noted that the SD Tech Fund helps companies progress from pilot scale to demonstration scale.^{Footnote 15}  Interviewees agreed that the conditions that contribute to the difficulty in progressing from pilot scale to demonstration scale continue to exist and that if the SD Tech Fund no longer existed there would be significant short- and long-term impacts for Canadian cleantech.  The costs for development and preparation for demonstration (TRLs from 3 to 7) are very high in Canada for large-scale cleantech projects.  Funding for large-scale projects is not readily available from private sources (e.g., banks, venture capital, etc.) as it is perceived to be very high risk since the proposed cleantech innovations are untested in real-world/operational settings and would require additional investment/support for scale-up for commercial production.

Aside from providing funding for cleantech projects, another important need addressed by the SD Tech Fund is assessing a project's feasibility and probability of success.  Stakeholder interview findings suggest that receiving Board approval for a project is perceived by private investors and other government partners across the innovation chain to mean that a project is worth pursuing and merits funding – thereby providing companies with the necessary funds to help bring their project to demonstration and, hopefully, subsequent commercialization.  However, it was also noted that SDTC's reputation has the potential to hinder development in the sense that if a company is not successful in attaining SD Tech Fund support then that company may not be able to secure support from other investors.

The existence of a demonstrable and unique need is further evidenced by the number of countries which also help fund their respective cleantech ecosystems.  The literature review showed that Finland and Canada are similar in their national regulatory quality^{Footnote 16} and government effectiveness^{Footnote 17}, both of which contribute to "an institutional framework that attracts business and fosters growth by providing good governance and the correct levels of protection and incentives essential to innovation".^{Footnote 18}  In Germany, much like the SD Tech Fund's consortium model requirement, the cleantech industry is heavily reliant on the extensive use of venture capital and private equity with a smaller proportion of public funding.  Further, Norway is a world leader in cleantech innovation – made possible by the number of incubator and cluster programs that are managed by the private sector, but are funded by the Government of Norway, as well as a rich network of government organizations that are actively engaged in helping start-ups in the clean energy industry, such as Innovation Norway.  Finally, the United States' approach shares many of the same features of the Canadian cleantech innovation model – several internal funding programs and others external to government.  However, in addition to public funding, the United States has a highly active network of cleantech hubs and holds the largest total sum of targeted venture capital in the world.

3.2 Performance

3.2.1 To what extent are recipients forming collaborative partnerships across the innovation chain?

SD Tech Fund recipients are forming collaborative partnerships across the innovation chain due in large part to the consortium model employed by the program.  Several case study participants and stakeholders interviewed for the evaluation emphasized that the SD Tech Fund's requirement for collaborations is perhaps the most important feature of the program's design, which is missing in other government-funded programs that support project demonstration.  In most cases, the recipient firm identified at least two potential collaborators at the time of application to the Fund.  An important aspect noted in the interviews is SDTC's preference that at least one collaborator be a market validation partner (i.e., representative of the target customer).  In one case, a company was able to engage several municipalities to test prototypes of its technology.  This input was invaluable to developing the market-ready product through the provision of actual infrastructure for testing purposes.  The municipalities also provided advice on how to best commercialize the product.

A review of program administrative data on 341 completed, active or terminated projects indicated that each project has an average of 4.13 consortium partners (which includes the project firm as well).  There does not appear to be any difference in the number of consortium partners based on the size of a project – for example, of the 60 projects with funding of $15 million or larger there were an average of 3.90 consortium partners.  By comparison, of the 65 projects with funding of $2 million or smaller there was an average of 3.88 consortium partners.

Administrative data also indicate that for every dollar of SD Tech Fund support the total leveraged amount from all partners and collaborators equalled $2.80 – higher than the target of $2 outlined in the Performance Measurement Strategy.  By year, the amount leveraged varied from $2.07 to $3.42 between 2002 and 2017.^{Footnote 19}

The survey found that 74% of firms agreed or strongly agreed that SD Tech Fund support was important to the formation of partnerships for their projects.  Almost one-half (46%) of survey respondents agreed or strongly agreed that SDTC's engagement activities/support helped to identify potential consortium partners.  The level of involvement of consortium members was reported to be highest during the project (77% of firms indicated that consortium members were often or always involved during the project) and lower pre- and post-SD Tech Fund support (43% and 46%, respectively).

Evidence clearly indicates that SDTC has been developing partnerships across the innovation chain.  For example, SDTC has (or has had) a memorandum of understanding (MOU) with several provincial partners.  The list includes the Province of British Columbia, Nova Scotia Innovacorp (expired agreement), Alberta Innovates, Transition Énergétique Québec, Ontario Centres of Excellence, and Emissions Reduction Alberta.  All MOUs specify the benefits of the agreement between the organizations as:

• An increase in the number of project investment opportunities;
• Opportunities to share due diligence, reduce duplication, process integration and streamlining;
• Opportunities for projects to go further and faster through co-funding; and
• Standardized application and reporting.

In terms of federal and other government departments and agencies, SDTC works collaboratively with EDC, BDC, GAC, NRCan, and the Standards Council of Canada (SCC) – as part of a larger innovation-funding ecosystem (along with private interests) to support demonstration and commercialization.  Together with these partners and other partners, SDTC works to improve access to financing, encourages investment, supports the demonstration of technologies, and builds the capacity necessary for the growth and prosperity of Canadian clean technologies.^{Footnote 20}

Among SDTC partnership activities noted by interviewed stakeholders, the recent engagement in the Joint Account Management (JAM) meetings (launched in summer 2017) was mentioned most frequently.  The goal of the JAM is to facilitate discussion and collaboration among SDTC, EDC, BDC and GAC to share information on high-potential firms in order to understand the firms, the cleantech innovation, their financial needs, and any organizational opportunities or mechanisms that could be considered to support firms.  Recently, the JAM was working on developing a waiver that would allow SDTC to share information with its partners to minimize duplication of efforts in their funding processes for post-demonstration support.

In addition, SDTC is entering into partnerships with regional sector organizations, universities and colleges, as well as with regional economic development organizations.  These were described as partnerships that work to support cleantech across the spectrum – from research, development and demonstration through to commercialization (one of SDTC's long-term outcomes).  Overall, interviewees agreed that SDTC's formation of public and private partnerships is critical to support Canadian cleantech innovation.  Specifically, interviewees indicated that the development of partnerships, including among funding organizations, is key to addressing critical issues of cleantech producers such as application and reporting burden, identifying funding/financing opportunities for eligible firms, and, where possible, developing common data standards, definitions, and information sharing.

3.2.2 To what extent are SD Tech Fund projects making progress towards demonstration and eventually demonstrating?  What factors influence demonstration?

TRLs are a measure used to evaluate the maturity of an evolving innovation and are based on a nine-level scale (see Appendix A).  Levels 1 through 6 represent the early and progressive stages of R&D for innovations.  In contrast, levels 7 through 9 represent the pre-commercialization gap for innovations when projects are working towards demonstrating in an operational environment.  According to SDTC administrative data, almost 74% of the 164 completed SD Tech Fund projects eventually demonstrated (i.e. met or exceeded their TRL goal levels of 7 or higher by the end of the project).  An even greater proportion (88%) of survey respondents reported that their projects had successfully demonstrated.

Figure 2 shows that more than one-half of the firms that demonstrated were in the energy utilization or transportation sectors.

Figure 2: Sector Characteristics of Completed SD Tech Fund Projects that Demonstrated (n=164)

Note: Demonstration is defined as reaching TRL7 or higher by the end of the project.
Source: SDTC administrative data, September 2017

The share of projects successfully demonstrating at TRL7 or higher (74%) is similar to the percentage of projects making progress towards demonstration (75%).  Progression towards demonstration can be measured by either: (1) the number of projects which completed all of the milestones as outlined in their CA; or (2) the share of projects reaching their goal TRL at or near the completion time of their CA.

According to several metrics that SDTC uses to track project progress, overall, projects tend to make good progress in the early stages, but momentum can wane towards project conclusion.  For example, the administrative data shows that between 74% and 100% of SD Tech Fund projects met their scheduled project milestones after years one and two.  However, the share of projects that met their milestones after year three ranged between 56% and 87%.  In addition, more recently developed measures used to track project progress by TRL shows that projects are making considerable progress but are not quite achieving defined targets.  Key program documents^{Footnote 21} have set an overall target level of 75% of projects progressing two TRLs. However, the data shows that only about two-thirds (65.8%) are advancing at least two TRLs by the end of the project.

Several interviewees indicated that one of the main factors contributing to the successful demonstration of Canadian cleantech projects is the availability of financing and/or investment, as demonstrating is an expensive and time-consuming process.  As noted earlier, it is difficult for Canadian cleantech producers to access capital to finance their projects/innovations since cleantech projects are viewed as high risk investments.  SD Tech Fund support for "de-risking" projects to a level that private institutions and venture capitalists perceive as acceptable is seen as a key benefit.

Case study participants added that some of the key factors to successful demonstration included the comprehensive project management process that SDTC uses to ensure that projects are successfully demonstrating, including the detailed proposals outlining milestones and metrics to be achieved, as well as SDTC staff working closely with proponents through the project delivery process.  Survey respondents noted that SD Tech Fund support, collaboration and the formation of partnerships is key to successful demonstration.

Many other factors can affect the progress of cleantech projects towards demonstration.  Stakeholders interviewed for this evaluation noted that projects may be delayed for a range of reasons (e.g., technical advancement is taking longer than anticipated, issues with timeliness among key suppliers, availability of key equipment, management issues, etc.).  Overall, it was estimated that approximately one-third of projects experience issues that significantly impact progress towards demonstration.

3.2.3 To what extent are SD Tech Fund projects commercializing?  What factors influence commercialization?

According to SDTC administrative data, about 34% of the 218 completed or terminated projects had commercialized or were deemed to be 'in the market' as of March 31, 2017 (see Table 2).

Source: SDTC administrative data, September 2017

By sector, firms operating in the energy utilization, forestry or transportation sectors were far more likely to commercialize (see Figure 3).

Figure 3: Commercial Status of all SD Tech Fund Projects by Sector (n=218)

Source: SDTC administrative data, September 2017

In terms of the priority technology area (for some projects, more than one priority is identified), the commercialization rate was by far the highest among the 39 projects associated with enabling technology (51.3%).  For the other five priority technology areas (i.e., hydrogen economy, clean fossil fuel, biomass, biofuel, and renewable electricity) commercialization rates ranged from 15.8% to 27.3%.^{Footnote 22}

Empirical analysis on the progress of projects backed by the SD Tech Fund suggests that larger projects have a higher probability of commercial success.^{Footnote 23}  However, there is no clear statistical relationship between the share of total funding provided by the SD Tech Fund and the probability of commercial success.  Other findings indicate that projects led by firms who have accessed SDTC multiple times have a higher probability of commercial success, although this may have more to do with an enhanced firm-specific capacity, rather than firms experiencing better performance in subsequent projects through learning-by-doing.  As well, the presence of other government funding during a project appears to lead to a greater likelihood of commercial success – not only does other government funding add to the overall investment, it also incents SDTC to increase its own investment.

Interviewees indicated that one factor of success is the development of a customer base.  The ability to export to international customers was seen as being critical – firms must develop a pipeline or range of domestic and international customers for successful commercialization.  Further factors of success which were highlighted included the need for a strong project management team, the presence of consortium partners, and governance for building a talented management team for technical development and planning for demonstration and a path to market.  Interviewees indicated that these are areas that SDTC has provided support to develop.

In terms of barriers, interviewees identified factors that are generally outside the locus of control of SDTC and the SD Tech Fund.  For example, interviewees indicated that the key barrier to commercialization is the investment needed to ramp-up for commercial production, as it is expensive and difficult to secure the financing needed to scale-up for large-scale commercial production.  It was indicated that it is difficult to secure the funding for cleantech projects and projects that have demonstrated only a limited number of times.  Typically, investors would like to have the assurance associated with multiple demonstrations. 

The administrative data confirms these findings in that the majority of firms (particularly those with completed projects) identified financing as the key issue that prevented them from commercializing (see Figures 4 and 5).  Among terminated projects, technical issues and other issues were far more prevalent than among completed projects (54.2% versus 26.7% respectively), although financing remains the key issue.

Figure 4: Key Issues that Prevented Completed Projects from Reaching Commercial Market (n=30)

Source: SDTC administrative data , September 2017

Figure 5: Key Issues that Prevented Terminated Projects from Reaching Commercial Market (n=48)

Source: SDTC administrative data , September 2017

Other barriers noted by interviewees included: identifying customers to take on cleantech solutions beyond immediate consortium partners in international and domestic markets; variations in procurement practices and regulations across Canada that make it difficult for cleantech producers to compete for contracts or gain entry into supply lists; and the need to make investments commercially viable, including producing cleantech at a price point that meets the needs of the end-users in Canadian and international markets.  In this final area, interviewees indicated that the challenge for cleantech producers is that their innovation must be able to replace an existing technology or service that clearly exhibits value to the potential customer through comparable unit pricing, unit/service duration or life, and/or savings in the long-term.

To provide some context to the potential barriers to advancement towards commercialization, the one firm in the case studies whose project had no progress to date indicated that their project did not progress beyond demonstration for reasons other than technology (most notably low oil prices) and that they would likely attempt to bring this technology toward commercialization again in the future.  For this firm, although the technology was perceived as being attractive, the risk of building an expensive commercial-scale facility using commercially unproven technology was seen as being too high.

The administrative data includes information on follow-on funding secured by SD Tech Fund recipients near or at the end of their SD Tech Fund funding and indicates that 40 of the 218 SD Tech Fund projects (or less than 20%) received a total of almost $2.1 billion in follow-on funding, with each project averaging almost $52.4 million in follow-on funding.  The smallest follow-on funding amount was $200,000 while the largest amount was $433.7 million.  The asset types of follow-on financing received are illustrated in Figure 6.^{Footnote 3}

Figure 6: Types of Follow-On Financing Received by SD Tech Fund Recipients Since 2001 ($ millions)

Source: SDTC Annual Report 2016-17 Supplement

However, receipt of follow-on funding is not a guarantee of success.  Although 55% of the 40 projects that received follow-on funding commercialized and 22.5% were in progress to market, an additional 22.5% failed to reach the market.  By comparison, 29.8% of the 178 projects that did not receive follow-on funding made it to market, 31.5% were in progress to market, and 38.7% failed to reach the market.

Additional findings from the survey revealed that 43% of firms agreed or strongly agreed that SDTC's engagement activities/support helped their firm secure follow-on financing from partners after SD Tech Fund support ended.

As noted above, there are many factors that can contribute to or detract from successful entry of cleantech projects and innovations into domestic and international markets.  The two case studies below provide an example of each.

• Nsolv (Calgary, Alberta) developed a process for the extraction of oil from oil sands using a pure condensing solvent instead of the water and energy intensive steam assisted gravity drainage traditional extraction method.  Nsolv initially received SD Tech Fund support in 2005 for their first project and then again was contracted in 2012 for $10 million over five years.  Nsolv's goal was to validate their process in a field pilot known as the BEST (Bitumen Extraction Solvent Technology) Field Pilot Plant.  When operating the BEST pilot, the demonstration site reduced GHG emissions by 75%, increased production rates by as much as three times, and eliminated the need for water in comparison to the standard process. Despite these positive outcomes and all of their milestones being completed successfully, Nsolv has not yet commercialized.  This is because the project was set to demonstrate the technology at a larger scale (via a higher production rate) than its previous demonstration in order to de-risk the technology for potential investors.  However, a key consortium partner pulled out of the project due to the drop in oil prices and Nsolv was not able to secure alternate investors to sponsor a full-scale commercial facility for production.
• SWITCH Materials, Inc. (Burnaby, British Columbia) received $2.4 million in SD Tech Fund support in 2011 to develop and demonstrate smart windows in a commercial application. These windows incorporate a film that automatically darken when exposed to sunlight and lighten in response to a very low electric charge.  The objectives of the project were to develop pilot production and manufacturing of SWITCH film, to achieve a target installation cost of $100/m2, and to demonstrate energy savings and CO2 reductions.  By project completion, SWITCH had developed a prototype and had demonstrated it in an operational environment.  However, the technology was not at a stage where it could be used for architectural windows given the requirement for commercial windows to last 20 years and need for greater cost-efficiency.  SWITCH identified the automotive market as a promising application for the technology and the technology is currently in progress toward market.

Recommendation: CTCGB should encourage SDTC to continue to work with other partners to develop relationships to enhance the identification of opportunities for cleantech producers to progress towards commercialization, including connecting firms with international opportunities/supply chains and sources of follow-on funding.

3.2.4 To what extent are SD Tech Fund projects and/or technologies contributing to the realization of environmental and economic benefits?

SDTC plays an important role in the government's mission to help grow Canadian companies into globally competitive ones.  Within this overall goal, the SD Tech Fund has two objectives: (i) contributing to achieving Canada's environmental objectives, including its GHG emissions reduction goals, and (ii) sustainable economic growth by enabling Canadian entities to compete globally in the cleantech space.^{Footnote 24}

Environmental Benefits

A 2017 OAG audit examined how the SD Tech Fund was contributing to reducing GHG emissions.^{Footnote 25}  The audit found that SDTC had a substantial review and challenge process for assessing proposed environmental benefits in project proposals – using internal experts and industry reviewers to assess intended benefits and emission reduction forecasts.  In addition, the audit found that SDTC had processes in place to estimate GHG emissions reductions as projects progressed.  SDTC had also modified its approach to calculating emission reduction estimates attributable to SD Tech Fund recipients to improve clarity in reports to Parliament.  However, the audit found that SDTC had difficulty in obtaining post-completion information from funding recipients.  Most of the expected post-project reports with environmental information were missing/not submitted.

Interviewees also indicated that environmental benefits are not easily defined or measured (e.g., while a new technology is thought to have environmental benefits, there may not be a measure presently available).  In addition, when estimates are developed, it is difficult to assess the level of benefits (e.g., reductions, etc.) that could be attributed to the investment by the SD Tech Fund versus other public and private investments. However, sometimes benefits are quantifiable, as evidenced by the Agrisoma case study.

Despite these challenges, SDTC has gathered information that SD Tech Fund supported projects have contributed the following environmental benefits since 2001.^{Footnote 26}

• An estimated 10.1 Megatonnes of CO2 equivalents (Mt CO2e) reduction in GHG emissions was attributable to SDTC's projects with technologies in the market in 2016.  This is the equivalent of removing almost 2.1 million cars from Canada's roads.  By sector, the two largest contributors to this reduction were forecasted to be transportation (a 3.7 Mt CO2e reduction) and energy production and exploration (a 3.0 Mt CO2e reduction).
• The annual reduction in GHG emissions is expected to surpass 13.5 Mt CO2e in 2017 and 45.0 Mt CO2e by 2022.  Firms operating in the transportation sector are expected to continue to lead the way (21.6 Mt CO2e reduction in 2022), while companies in the energy utilization sector are projected to reduce CO2e emissions by 14.3 Megatonnes in 2022.

Economic Benefits

There are similar issues with the assessment of economic benefits yielded by SD Tech Fund projects, as it is difficult to obtain post-project information from recipients.  Further, the calculation of benefits and attribution to the SD Tech Fund is difficult.  However, recent Statistics Canada data shows that cleantech and environmental goods and services benefit the Canadian economy in many ways, including through increased gross domestic product (GDP) and employment.

According to SDTC reporting, SD Tech Fund projects have contributed to an estimated 9,437 new jobs (5,302 direct jobs and 4,135 indirect jobs) in the Canadian cleantech space since 2001.^{Footnote 13}  Additional economic benefit estimates indicate that:

• $1.9 billion in estimated annual revenue in 2016 was generated by SDTC-funded companies in the market.  Firms in the transportation sector were forecasted to contribute $928.5 million in revenue and companies in the energy utilization sector were projected to generate $565.2 million in revenue.
• Annual revenues are expected to surpass $2.6 billion in 2017 and $6.4 billion by 2022.  Firms in the transportation sector and energy utilization sector are expected to continue to comprise the majority of revenues in 2022, comprising $5.6 billion of the $6.4 billion respectively.
• The costs avoided due to air quality, clean water and clean soil advancements attributable to SDTC's projects with technologies in the market continue to accelerate – particularly for clean air projects.
  • $136.4 million in estimated annual costs were avoided in 2016 ($113.1 million from the waste management sector alone).
  • Costs avoided are expected to top $139.8 million in 2017 and $285.6 million by 2022, with costs avoided due to clean air improvements forecasted to increase the most (by 565%) compared to 57% for clean water and 29% for clean soil.^{Footnote 3}

Primary research for this evaluation found that SD Tech Fund support for cleantech projects yielded economic benefits.  The majority of survey respondents who reported economic benefits (e.g., increase in the number of employees, sales, etc.) indicated that it was 'likely' or 'definitely' due to SD Tech Fund support.  Additionally, respondents also reported that the SD Tech Fund positively impacted their domestic (80%) and international (73%) sales.

The case studies provided more detailed, qualitative information.  Case studies showed that the SD Tech Fund enabled several Canadian cleantech producers to complete demonstration and to make advancements towards commercialization with significant economic impacts.

• As a result of $3.9 million in funding, a Toronto-based company, Vive Crop Protection Inc. was able to launch their project Vive Formulations of Crop Protection Active Ingredients in order to develop a nanotechnology platform with cost-effective applications in agricultural chemicals and industrial catalysts.  Application of the nanoparticle-based agricultural products occurs at the same time as seed.  As a result, there is no need for the farmer to make a second pass after seeding to spray their crop.  This reduces time and saves on fuel costs for the farmer.  A further economic benefit of the fungicide and pesticide products is that they assist early plant growth resulting in stronger plants with larger stems and increased growth.  This was an unexpected finding of the product testing under the project.  Farmers who use Vive products see a return on their investment of between 200% and 300% for corn, soya beans and potatoes.  As a result, farmers are willing to pay significantly more for the Vive product, as this additional cost will be recouped in greater yields.
• With $2.2 million in SD Tech Fund support, Pure Technologies Ltd. was able to test their PipeDiver prototype in a real-world setting.  This water and wastewater pipeline assessment tool is able to identify distressed pipes while minimizing operational risk.  The economic benefits of PipeDiver are in its ability to minimize infrastructure repair/replacement expenditures by extending the useful life of municipal water infrastructure.  The project was successfully completed in 2012 and is now commercially available with clients all over the world.
• SemiosBio Technologies Inc., headquartered in Vancouver, designed an Automated and Integrated Pest Management (AIPM) System that links sensors, monitors, pheromone release devices and traps for insects.  This system is an alternative to harmful pesticides that farmers use and protects their crop yields and revenues.  In 2012, SDTC provided $4.9 million in funding over five years in order to demonstrate the Semios system.  Semios conducted their demonstration of the AIPM system in three main apple-growing regions in Canada which resulted in economic gains due to a reduction in costs of protecting crops, an increase in the value or volume of the crop, and a reduction in the risk of lost crops.  The technology mainly has benefits for soil and water quality by reducing pesticide use and subsequent contamination.

Recommendation: CTCGB should encourage SDTC to work with the Clean Growth Hub to address some of the limitations of the administrative data, including developing standards to enable consistency of the data and collecting information related to key performance indicators, as well as environmental and economic benefits.

3.3 Efficiency and Economy

3.3.1 Does the governance structure support efficient and effective delivery?

As described in Section 1.4, SDTC is governed by a 15-member Board of Directors with overall responsibility for the stewardship and strategic direction of the SD Tech Fund.  Seven Board Members, including the Chairperson, are appointed by the Governor in Council, while the remaining eight are selected by the SDTC Member Council which is comprised of individuals from the public, private and academic sectors.  Board Members' experience represents a wide range of sectors including cleantech, agriculture, sustainable community development, mining, life sciences, manufacturing, petroleum, finance, venture capital, engineering, environmental issues, entrepreneurship, chemical recycling, commercial wind and natural gas development, and public policy.

The Board and its five committees play a key role in the funding allocation process.  SDTC Board members sit on the Project Review Committee along with private sector investment principals to identify technologies with high potential and oversee funding processes.  However, the Board of Directors is the authority for final approval for SD Tech Fund support.  The Board now schedules at least five meetings per year to approve projects, aligned with the number of Project Review Committee meetings.

Overall, past reviews, audits and evaluations have found that SDTC and the SD Tech Fund have a sound governance structure and supporting processes in place to meet the program's objectives and mandate.  Past audits and evaluations have found that SDTC has strong business processes to screen, fund and support cleantech projects.  Activities and processes are in place to ensure that approved projects are consistent with objectives, transparent approvals and project monitoring.^{Footnote 27}

A 2016 Governance Review assessed the effectiveness of the existing governance structures and processes and the extent to which current governance policies and practices have been implemented as intended.^{Footnote 28}  Overall, the findings of this review were positive.  It found that SDTC has implemented a number of improvements to its governance processes and practices over the past two years, including expectations of the Board, Member Council, and Project Review Committee regarding integrity, values and ethics, as well as the roles and mandates of the governing bodies.

The review also identified some areas for improvement.  Eight recommendations were made in four areas:

• Policy: the expected role and mandate of the Member Council should be clarified and the Conflict of Interest Policy and related guidelines and procedures should be enhanced to improve clarity and consistency.
• Human resources: management should complete the analysis of existing skills and experience for governing bodies to identify gaps or areas in which future recruitment activities should be focused.  As well, annual Board evaluations and skills and competency analysis should be used to inform and action other training and development needs and interests.
• Committees: specific committees should be consolidated given that the roles and mandates of some committees are similar.  Otherwise, a duplication of effort and/or inefficiencies may result.
• Reporting: there is a need to strengthen formal reporting on changes in risks, status of risk mitigation activities for critical risks, and emerging risks, as well as reporting on SDTC's performance against key performance indicators.

Most recently, a 2017 OAG audit found that the SD Tech Fund has a comprehensive process in place for proposal review and approval that uses reviews by internal and external experts, several committees and final approval by the Board.^{Footnote 29}  Further, it found that SDTC has measures in place to mitigate potential conflicts of interest for Board and committee members involved in the assessment and approval of proposals.

The evaluation followed up on progress made to address the recommendations outlined in the 2016 report.  While the evidence provided shows that some steps have been made toward addressing the action items, implementation is ongoing.

Recommendation: CTCGB should encourage SDTC to fully implement the recommendations stemming from a recent review of program governance.

3.3.2 How efficient and effective is SDTC in the delivery of the SD Tech Fund?

In examining the efficiency and effectiveness of the delivery of the SD Tech Fund, four metrics were considered:

• Cost: analysis of program expenditure data and administrative cost data;
• Administration of the fund: views on current program delivery and suggestions for improvement;
• Timeliness: data and opinions on the timeliness of the application process and the receipt of funds; and
• Reporting requirements: thoughts of recipient firms on SDTC's reporting requirements.

Cost

An analysis of allocation and expenditure data provided by SDTC (covering the period from 2012 to 2016) was completed with the goal of analyzing the efficiency of the delivery of the SD Tech Fund.  Program expenditures which consistently exceed the allocation could indicate inefficiency in program delivery.^{Footnote 30}  A 2011 performance audit flagged this as a risk, but evidence indicates this has been mitigated.  With the exception of 2015, program expenditures (i.e. salaries and benefits, non-salary operating expenses, and grants and contributions) were lower in every year than the allocated budget.  Over this five-year period, $355.6 million was allocated for the SD Tech Fund and $327.5 million (or 91.2% of the allocated amount) was spent.

In terms of the administrative costs of delivering the SD Tech Fund, administrative costs (i.e., salaries and non-salary operating expenses) as a percentage of total expenditures (i.e., project disbursements) have been declining since 2015.  In 2011, administrative costs were $11.3 million and project disbursements were $82.3 million, equating to an administrative cost ratio of 13.7%.  This ratio peaked at 21.5% in 2014, but fell significantly to 16.1% in 2015.  For the most recent reporting period in 2016-17, the administrative cost ratio was 14.3%.^{Footnote 31}  For the $965 million in SD Tech Fund support available, SD Tech Fund operating expenses are capped at $167 million – implying an overall administrative cost ratio of 17%.^{Footnote 32}

Administration of the Fund

SDTC has made a number of improvements to the administration of the fund in recent years.  The shift from a periodic call for applications to a continuous intake process in July 2016 was noted by case study participants and interviewees as a notable improvement, as it provides cleantech producers with greater flexibility over when to apply for SD Tech Fund funding.  In addition, SDTC has adopted a more interactive process throughout the application process, including the use of video conference presentations.  SDTC has also increased the frequency of final funding decisions made by the Board of Directors (from two times per year historically to five).

Further, providing funding up-front (the amount of which varies between projects based on the contract and the milestones schedule) was seen as a benefit.  SDTC's prepayment model is designed to allow firms to remain cash flow positive throughout the period that they are receiving SD Tech Fund support.  Survey respondents indicated that some government programs will only reimburse the firm against expenditures.  Having cash upfront is a benefit since cleantech is extremely cash-intensive and is beneficial when compared to the reimbursement model that many government programs employ.

Case study participants unanimously agreed that SDTC staff is helpful, technically competent, professional, and committed to the success of Canada's cleantech producers.  This was echoed in interviews, with special note made of the work of SDTC regional staff.  As well, stakeholders interviewed for the evaluation reported that the SD Tech Fund was being delivered efficiently to meet its goals.  It was identified as the correct model to meet the needs of firms.

Cleantech experts interviewed for the evaluation were asked for suggestions regarding how delivery could be improved.  The following areas were identified:

• Better alignment with provincial and federal organizations.  Some interviewees felt there could be more collaboration with regional and federal partners to maximize benefits for the applicant/recipient firms – leveraging knowledge and expertise already established to enhance understanding of local/regional ecosystems.
• Enhancement of SDTC's presence across the regions to ensure that all regions are receiving support for cleantech producers.  Interviewees noted the potential benefits of:
  • Locating SDTC staff in cleantech intensive areas.  Locating regional directors in key cleantech areas in Toronto, Vancouver, Calgary, Montreal and Halifax would facilitate direct and ongoing access to local cleantech communities.  Where possible, co-locating of SDTC regional directors with regional development organizations would facilitate greater information sharing and alignment of activities.
  • Swift replacement of regional directors in key regions.  Interviewees noted that it has been nearly two years since Ontario has had a regional director and one year for Alberta.  Several interviewees (from Ontario) stated concern about there being no regional director assigned to Ontario, given that Ontario has the largest cleantech presence in Canada.

Timeliness

The timeliness of the application processes was consistently highlighted as an area for improvement by external stakeholders, as it is complex and requires multiple levels of approval.  The application process was noted by 35% of survey respondents to be slow and complicated (up to one year), requiring a lot of paperwork on the part of both applicant firms and SDTC.  Further, survey respondents thought that the application should be tailored to the size of the project.

Cleantech experts suggested that time requirements for the application and due diligence process was slow and misaligned with the pace of the private sector.  Several interviewees noted that the time from application to Board approval and receipt of funding is long.  Specifically, the application process would benefit firms if it was streamlined, simplified and processed faster.

Exploring further the timeliness of the approval of funds, a review of SDTC administrative data of the 218 completed or terminated projects (all of which started between 2002 and 2014 – prior to the change to the continuous intake of applications) found that 78.4% of projects received funding within 24 months of Board approval.^{Footnote 33}  This percentage varied significantly from year-to-year, with the share being 100% in some years (for the 38 projects approved in 2002, 2003 and 2011)^{Footnote 34} and as low as 56.3% for the 16 projects approved in 2004.  The data also indicates that, on average, projects with a higher total cost were more likely to get approved and funded before projects with a lower total cost.

Further, a 2017 report by the Office of the Auditor General of Canada^{Footnote 35} (based on 28 SD Tech Fund projects reviewed) found that the time elapsed from initial project submission to the first payment averaged 22 months.  This was partly attributable to SDTC's extensive due diligence process and to the time required for applicants to secure additional funding and meet other conditions.

It should be noted that there are a range of factors that could impact the timeliness of the application process that are outside of SDTC's control.  For example, many companies supported by SDTC are SMEs with limited staffing resources that may prevent them from responding quickly to requests for supplemental information from SDTC.

Reporting Requirements

Twenty percent of survey respondents suggested that reporting requirements were difficult, perceived as demanding, rigid and burdensome, particularly for smaller firms who may not be able to produce lengthy reports as easily as larger firms.  This was evident in an examination of the performance data, in that from 2010-11 to 2016-17, post-project questionnaire completion, a key reporting tool for economic and environmental impacts, ranged from only 9% to 34%. 

In the case studies, one company found the amount of reporting to be burdensome and suggested that SDTC should streamline its reporting requirements and take advantage of technology (e.g. online reporting).

Recommendation: CTCGB should work with SDTC to identify opportunities to streamline the application and contracting process to improve contracting timelines.

3.3.3 Are there alternative approaches that would increase the cost-effectiveness of the SD Tech Fund?

According to a 2016 report by the Working Group on Clean Technology, Innovation and Jobs, an alternative approach that could increase the cost-effectiveness of the SD Tech Fund (as well as other programs) is to create a "no wrong door approach" or a consolidation of programs to reduce the administrative burden on firms.^{Footnote 36}  Currently, there is difficultly in navigating and accessing the appropriate government support programs for cleantech producers due to the large number of actors, programs and policies.  Federal, provincial and territorial collaboration on research priorities and policies could be achieved through alignment of collective outcomes across technologies and technology platforms.  Program delivery alignment could then help improve accessibility, and help technology developers negotiate public support through different stages of development.

Interviewed stakeholders identified the following approaches to improve the cost-effectiveness of the SD Tech Fund:

• Aligning more closely with American and/or Scandinavian models for examples of better partnerships between public and private sources of capital, as well as coordination between all levels of public support for cleantech;
• Consideration of the problem statement approach in the U.S. (i.e. involving the identification of an issue and calls for solutions).  In this approach, the goal is better integration between cleantech innovation and target market needs;
• Consideration of a more staged approach, whereby the successful completion of a stage would open additional opportunities for financial support; and,
• Consideration of a more government-wide approach in the collection of post-demonstration information from cleantech producers to minimize duplication in data collection efforts, reduce survey burden of cleantech producers, and foster collection of standardized information.

Recommendation: CTCGB should encourage SDTC to investigate other information-sharing and streamlined reporting options, including collaboration with other data collection and funding entities, with the aim of minimizing duplication and reducing the administrative burden on companies.

4.0 Conclusions and Recommendations

4.1 Relevance

• There is a continued need for the SD Tech Fund to support cleantech producers to further develop and demonstrate their technologies, as it directly addresses the ongoing pre-commercial funding gap.  SDTC's due diligence process is valued by investors, who appreciate SDTC's process for evaluating technical and project management competence.

4.2 Performance

• The consortium model of the SD Tech Fund plays an important role in facilitating collaborative partnerships between funding recipients and their suppliers and customers.  In addition, SDTC is actively building and maintaining partnerships across the innovation chain.
• Approximately three-quarters of all projects that received SD Tech Fund support have made progress towards demonstration or have successfully demonstrated.  The key factors related to successful demonstration include the availability of financing and the formation of partnerships.  Numerous barriers toward demonstration exist, significantly impacting the progress of about one-third of projects.
• Although the mandate of the SD Tech Fund ends at demonstration, at the time of the evaluation, about one-third of SD Tech Fund recipients subsequently reached the market (i.e., commercialize), with larger projects and those receiving other government support having a higher probability of success.  The ability to raise follow-on capital following successful demonstration remains the main barrier to commercialization.  The development of relationships with other partners may help to identify opportunities to enhance commercialization prospects for SD Tech Fund projects.
• Despite the documented challenges in estimating environmental and economic benefits, the available data and primary research for this evaluation indicates that the SD Tech Fund has helped create environmental and economic benefits for the Canadian economy.

4.3 Efficiency and Economy

• SDTC has a governance structure in place which supports the effective delivery of the SD Tech Fund.  SDTC has recently implemented several improvements to address findings from recent audits and reviews, although further work is still required to fully implement the recommendations.
• SDTC is efficiently and effectively delivering the SD Tech Fund, although some concerns exist over the timeliness of the application and contracting process.
• Currently, there are a range of government support programs for cleantech projects across the innovation chain.  To access support from these programs, cleantech producers may be required to provide similar information that can take time and resources.  In addition, there are a range of organizations collecting post-demonstration information from cleantech producers.

4.4 Recommendations

• CTCGB should encourage SDTC to continue to work with other partners to develop relationships to enhance the identification of opportunities for cleantech producers to progress towards commercialization, including connecting firms with international opportunities/supply chains and sources of follow-on funding.
• CTCGB should encourage SDTC to work with the Clean Growth Hub to address some of the limitations of the administrative data, including developing standards to enable consistency of the data and collecting information related to key performance indicators, as well as environmental and economic benefits.
• CTCGB should encourage SDTC to fully implement the recommendations stemming from a recent review of program governance.
• CTCGB should work with SDTC to identify opportunities to streamline the application and contracting process to improve contracting timelines.
• CTCGB should encourage SDTC to investigate other information-sharing and streamlined reporting options, including collaboration with other data collection and funding entities, with the aim of minimizing duplication and reducing the administrative burden on companies.

Appendix A: Technology Readiness Levels Scale

Source: Public Works and Government Services Canada description of Technology Readiness Levels.

Management Response and Action Plan

Management Response and Action Plan—Evaluation of the Sustainable Development Technology Fund [PDF - 143 KB]

A - For inclusion in the report

The findings and recommendations of the Evaluation of the Sustainable Development Technology Fund were provided to the Clean Technology and Clean Growth Branch of the Science and Research Sector (SRS) at ISED, as well as to Sustainable Development Technology Canada (SDTC).  Management of these partners have been consulted and agree with the findings included in this report and will take action to address all applicable recommendations by May 2020.

B - For follow-up purposes - Detailed actions to address the recommendations in the report