Evaluation of Industry Canada's Contribution to Genome Canada

Final Report

April 2015

Presented to the Departmental Evaluation Committee on April 2, 2015

Approved by the Deputy Minister on April 14, 2015

Table of Contents

• Executive Summary
• 1.0 Introduction
  • 1.1 Program Overview and Structure
  • 1.2 Program Objectives and Expected Results
  • 1.3 Governance of the Program
  • 1.4 Target Population and Stakeholders
  • 1.5 Program Resources
  • 1.6 Logic Model
• 2.0 Methodology
  • 2.1 Evaluation Objectives and Scope
  • 2.2 Evaluation Questions
  • 2.3 Evaluation Approach
  • 2.4 Data Collection Methods
  • 2.5 Limitations of the Data Collection
• 3.0 Findings
  • 3.1 Relevance
  • 3.2 Performance
• 4.0 Conclusions and Recommendations
  • 4.1 Relevance
  • 4.2 Performance
  • 4.3 Recommendations
• Appendix A – Science‑Metrix evaluation recommendations, key supporting evidence and corresponding AEB analysis/recommendations
• Appendix B – Genome Canada Program Logic Model
• Appendix C – Overview of Science‑Metrix Evaluation Data Collection Methodology
• Appendix D – Data Collection Limitations
• Management Response and Action Plan

Executive Summary

Program Overview

Genome Canada is a not‑for‑profit organization, created and incorporated in 2000 under the Canada Corporations Act with a mandate to develop and implement a national strategy in genomics^{Footnote 1} research for the benefit of all Canadians. Genome Canada seeks to deliver on this mandate by investing in large‑scale genomics research initiatives in sectors of strategic and economic importance to Canada (i.e., health, agriculture, environment, forestry, fisheries, energy and mining), aiming to strengthen genomics research and technical capacity in Canada, and foster multi‑sectorial partnerships nationally and globally. Genome Canada also works to ensure that genomics research efforts consider underlying ethical, environmental, economic, legal or social aspects (GE³LS). As of May 31, 2014, Genome Canada has received almost $1 billion in payments from Industry Canada^{Footnote 2} and has raised over $1.3 billion from co‑funding partners (e.g., provincial governments and agencies, international non‑governmental organizations and research institutes).^{Footnote 3}

Evaluation Purpose and Methodology

In accordance with the Treasury Board Policy on Evaluation and the Directive on the Evaluation Function, the purpose of this evaluation was to assess the core issues of relevance and performance of Industry Canada's contribution to Genome Canada. The evaluation covered the period from March 2008 to January 2014, with more current information added to the extent possible.

This evaluation builds on an evaluation completed by Science‑Metrix in March 2014. The Science‑Metrix evaluation reviewed Genome Canada's activities from 2009 to 2014 and included a limited assessment of relevance (specifically continued need), as well as an assessment of performance. The current evaluation study was conducted by Industry Canada's Audit and Evaluation Branch (AEB) and expands the scope to cover all three core issues of relevance and supplements the performance information reported by Science‑Metrix, with a focus on assessing Genome Canada's performance based on the expected results outlined in Industry Canada's funding agreement with Genome Canada.

The Science‑Metrix evaluation employed five data collection methods: document, file and literature reviews; interviews with key stakeholders; a survey of the Canadian genomics research community, highly qualified personnel (HQP) and other stakeholders; a bibliometric review, an international comparative review and case studies of eight projects. The current study included a further document review, literature review, and interviews with Industry Canada and Genome Canada representatives.

Findings

Relevance

There is a continued need for increased S&T research and innovation capacity as a means to provide social and economic benefits to Canadians. Genome Canada responds to these needs by investing in large‑scale genomics initiatives in sectors of strategic and economic importance to Canada. Large scale research is appropriate in light of the complex, multidisciplinary nature of genomics research and stands to offer significant economic benefits for Canadians.

Support for Genome Canada is consistent with federal government priorities as set out in the 2007 S&T Strategy and subsequent Federal Budgets. Genome Canada's objectives are also in line with Industry Canada's strategic outcome related to advancing S&T, knowledge and innovation in order to strengthen the Canadian economy.

Support for Genome Canada is consistent with federal roles and responsibilities to encourage the development of S&T and aligns with Industry Canada's mandate to foster and promote S&T in Canada.

Performance

Overall, evaluation evidence suggests that Genome Canada is achieving its expected results. The organization has been successful in increasing the breadth and depth of knowledge in genomics, as well as knowledge specifically related to the ethical, environmental, economic, legal and social issues of genomics (GE³LS).

Projects supported by Genome Canada have also enhanced Canada's international profile and visibility in genomics research and contributed to developing a solid base of researchers trained to undertake future genomics research. Genome Canada has also facilitated genomics research by providing leading‑edge technologies through funding the Science and Technology Innovation Centres (STICs).

Genome Canada is contributing to translating genomics research discoveries into applications and there is evidence of socio‑economic benefits associated with Genome Canada‑supported projects. Nonetheless, better defined concepts and measures of translational and socio‑economic benefits could help tell the full story of Genome Canada's impact. Moving forward, it will be important to ensure that these are adequately defined, particularly for commercialization‑focused initiatives such as the recently launched Genomic Applications Partnership Program (GAPP).

Genome Canada has made efforts to improve the efficiency of its own operations during the evaluation period and exceeded Industry Canada's target 1:1 co‑funding leveraging ratio for the evaluation period. Specific areas for improvement include the need for more coordinated communication efforts across the Genome Centres and between Genome Canada and Centres, as well as ongoing improvements in performance measurement. Evaluation evidence suggests that the third‑party delivery model being employed by Industry Canada is an efficient means to support large‑scale genomics research in Canada.

Recommendations

The Science‑Metrix evaluation developed five recommendations for Genome Canada (see Appendix A). Genome Canada is in agreement with the recommendations made in the report and has developed a management response for these recommendations. As an observer on Genome Canada's Board of Directors, Industry Canada will be in a position to follow the implementation of these recommendations.

In response to the findings of this evaluation, and consistent with the finding of the Science Metrix evaluation, it is recommended that:

1. Industry Canada's Science and Innovation Sector (SIS) work with Genome Canada as required on defining and measuring concepts related to translational and socio‑economic impacts based on the expected results of the funding agreement.

1.0 Introduction

This report presents the results of an evaluation of Industry Canada's contribution to Genome Canada. The evaluation assessed the relevance and performance of the contribution to Genome Canada in accordance with the requirements of the Treasury Board Policy on Evaluation. The report is organized into four sections:

• Section 1 presents the program profile;
• Section 2 presents the evaluation methodology;
• Section 3 provides the key findings related to the evaluation issues of relevance and performance; and
• Section 4 summarizes the study's conclusions and provides recommendations.

Program Profile

1.1 Program Overview and Structure

Genome Canada is a not‑for‑profit organization, created and incorporated in 2000 under the Canada Corporations Act with a mandate to develop and implement a national strategy in genomics^{Footnote 4} research for the benefit of all Canadians. Genome Canada seeks to deliver on this mandate by investing in large‑scale genomics research initiatives in sectors of strategic and economic importance to Canada (i.e., health, agriculture, environment, forestry, fisheries, energy and mining), aiming to strengthen genomics research and technical capacity in Canada, and foster multi‑sectorial partnerships nationally and globally. Genome Canada also works to ensure that genomics research efforts consider underlying ethical, environmental, economic, legal or social aspects (GE³LS).

As of May 31, 2014, Genome Canada has received almost $1 billion in payments from Industry Canada^{Footnote 5} and has raised over $1.3 billion through co‑funding commitments^{Footnote 6}, for a total investment of over $2.3 billion^{Footnote 7}. Co‑funding partners include provincial governments and agencies, international non‑governmental organizations and research institutes, industry, universities, and research hospitals.

Genome Canada delivers its mandate through six Genome Centres, one in each of the following regions: British Columbia, Alberta, Prairies, Ontario, Quebec and Atlantic. Aside from administering the funds to research projects, Genome Centres are also responsible for identifying regional strengths and opportunities, monitoring compliance and performance, and helping secure co‑funding from partners.

Genome Canada also provides Canadian scientists with advanced technologies and expertise for funded projects by supporting the operations of five Science and Technology Innovation Centres (STICs).

In terms of structure, funding is contributed by Industry Canada directly to Genome Canada. Genome Canada is responsible for the development of strategies and partnerships at both national and international levels, and launches national competitions and a merit review process for the selection of research projects. Once the selection is completed, Genome Canada in turn funds the Genome Centres and STICs, who then transfer the appropriate funding to selected research projects.

1.2 Program Objectives and Expected Results

The expected results of Industry Canada's funding to Genome Canada as per the 2014 Contribution Agreement are as follows:

• Increased breadth and depth of knowledge in genomics, including knowledge related to the ethical, environmental, economic, legal and social issues of genomics (GE³LS);
• Canada has a solid base of researchers that are trained to undertake future genomics research;
• The translation of research discoveries into application in the public and private sectors leads to socio‑economic benefits either directly or through development toward, for example, commercialization;
• Canadian genomics research is enabled through the provision of leading‑edge genomics, proteomics, metabolomics and bioinformatics technologies; and
• Canada's international profile and visibility in genomics research is enhanced.

1.3 Governance of the Program

Industry Canada's Science and Innovation Sector (SIS) is responsible for the ongoing management of the contributions to Genome Canada.

Genome Canada and Genome Centres are responsible for the management and administration of the program. Industry Canada attends Board of Director meetings of both Genome Canada and Genome Centres to monitor the implementation of the program.

Genome Canada is governed by a Board of Directors with representation from academia, industry, and regions across Canada. Genome Canada's President and Chief Executive Officer (CEO) is responsible for the development and implementation of decisions made by the Board and the overall management of Genome Canada, including performance monitoring and reporting.

The funding agreements between Industry Canada and Genome Canada set out the obligations of both parties vis‑à‑vis the contributions, including Genome Canada's obligations in transferring funds to the Genome Centres.

1.4 Target Population and Stakeholders

The target population for the Genome Canada program is the genomics research community located in Canadian universities, research hospitals, and not‑for‑profit research institutions. A number of additional stakeholder organizations participate in the delivery of the program. The following list provides an overview of the major stakeholder groups:

Federal Funders: The federal granting councils and the Canada Foundation for Innovation (CFI) sometimes partner with Genome Canada in the launch of new competitions. For example, Genome Canada and the Canadian Institutes of Health Research (CIHR) partnered in 2012 to deliver the large‑scale applied research competition in personalized medicine. This enabled both funders to pool their resources and invest in an area of strategic importance for both organizations.

Provincial Governments: Provincial governments support a portion of the operating costs of regional Genome Centres. Some of them (notably British Columbia, Ontario and Quebec) make substantive investments in provincial genomics programs that are also delivered through the Genome Centres, and which enhance and complement national support through Genome Canada.

Co‑funding Partners: Provincial governments, companies, industry associations, not‑for‑profit organizations, other federal government departments and agencies, and foreign research institutions are among the co‑funders contributing to research projects supported by Genome Canada.

Genome Centres: The six regional Genome Centres are also considered stakeholders as they receive funding from other sources (primarily provincial governments) and manage their own regional genomics research programs in addition to managing projects funded by Genome Canada. Efforts are taken to ensure that these projects do not duplicate genomics research projects supported by Genome Canada, but rather complement them or examine different areas based on regional needs and priorities.

STICs: The five regional Science and Technology Innovation Centres (STICs) provide Canadian scientists with advanced technologies and expertise and are supported by Genome Canada, as well as other funders including the federal granting councils, the CFI, provincial governments and research institutions.

Researchers and their Host Institution: Researchers are the applicants to the program and the ultimate recipients of the funds. They undertake the research in genomics in a Canadian research institution, such as a university, a research hospital, or a not‑for‑profit research organization. These researchers are categorized as Principal Investigators (PIs), i.e., lead researchers on projects, or as co‑Principal Investigators (co‑PIs) who lead projects in partnership with other project leads.

Users: It is expected that the research undertaken will benefit users in the public and private sector, specifically in the areas of health, agriculture, forestry, fisheries, aquaculture, energy, mining, and the environment.

1.5 Program Resources

Since 2008, Industry Canada has signed five transfer payment agreements with Genome Canada, totalling $505 million. The most recent agreement runs until March 31, 2020.

Industry Canada disbursements to Genome Canada for research projects vary every year, based on the projects funded and their cash flow requirements. Of the funding provided to Genome Canada, approximately $6.6 million is allotted annually to support its salary and O&M costs, and $4.8 million to support a portion of the salary and O&M costs of the six Genome Centres.

In terms of full‑time equivalents (FTEs) dedicated to the program, Industry Canada has 0.5, Genome Canada 22, and Genome Centres 55.

1.6 Logic Model

A logic model is a visual representation that links a program's activities, outputs and outcomes; provides a systematic and visual method of illustrating the program theory; and shows the logic of how a program, policy or initiative is expected to achieve its objectives. It also provides the basis for developing evaluation strategies, including the evaluation matrix.

The logic model for the program (see Appendix B) was developed in 2013 as part of Performance Measurement Strategy for the Genome Canada Program. Note that this evaluation assesses Genome Canada's performance based on the objectives/expected results outlined in Industry Canada's funding agreements with Genome Canada as opposed to reporting explicitly on each specific outcome in the logic model. This is in line with the approach taken by Science‑Metrix for its evaluation report, which as the next section explains, was a key source of information for this evaluation.

2.0 Methodology

This section provides information on the evaluation objectives and scope, the specific evaluation questions that were addressed, the evaluation approach, data collection methods, and limitations on the data collection.

2.1 Evaluation Objectives and Scope

An evaluation of Industry Canada's contribution to Genome Canada is required under section 42.1 of the Financial Administration Act. In accordance with the Treasury Board Policy on Evaluation and Directive on the Evaluation Function, the purpose of this evaluation was to assess the core evaluation issues of relevance and performance. The evaluation study covered the period from March 2008 to January 2014, with more current information added to the extent possible.

2.2 Evaluation Questions

The evaluation addressed the following questions on relevance and performance:

Relevance

1. Is there a continued need for large‑scale genomics research as supported by Genome Canada?
2. To what extent is the support for Genome Canada aligned with the priorities of the federal government and the strategic outcomes of Industry Canada?
3. Does support for Genome Canada align with federal roles and responsibilities?

Performance

1. To what extent has Genome Canada contributed to increasing the breadth and depth of knowledge in genomics, including knowledge related to ethical, environmental, economic, legal and social issues (GE³LS)?
2. How effective was Genome Canada in developing a solid base of researchers that are trained to undertake future genomics research?
3. To what extent has the provision of enabling technologies contributed to enhancing the quality of Canadian genomics research?
4. To what extent has Genome Canada contributed to enhancing Canada's international profile and visibility in genomics research?
5. To what extent has Genome Canada contributed to translating genomics research discoveries into applications leading to socio‑economic benefits for Canadians?
6. How efficiently and economically is the program being delivered?

2.3 Evaluation Approach

This evaluation builds on an evaluation completed by Science‑Metrix in March 2014. The Science‑Metrix evaluation reviewed Genome Canada's activities from 2009 to 2014 and included a limited assessment of relevance (continued need), as well as an assessment of performance. An Industry Canada Evaluation Directorate representative sat on the Genome Canada Evaluation Steering Committee in order to provide input into the methodology and to provide comments on the report. This report led to a number of recommendations specific to Genome Canada, which can be found in Appendix A along with a summary of the key evidence used to support these recommendations. Corresponding AEB analysis regarding the applicability to SIS (and recommendations for SIS if appropriate) is also included.

The current evaluation study was conducted by Industry Canada's Audit and Evaluation Branch (AEB) and covers all three core issues of relevance and supplements the performance information reported by Science‑Metrix, with a focus on Genome Canada's performance based on the expected results outlined in Industry Canada's most recent (2014) funding agreement with Genome Canada in order to provide the most current information possible.

2.4 Data Collection Methods

Data collection and analysis was primarily conducted by Science‑Metrix for core issues of performance and by AEB for core issues of relevance. The Science‑Metrix evaluation employed five data collection methods: document, file and literature reviews; interviews with key stakeholders; a survey of the principal investigators (PIs), co‑PIs, GE³LS PIs and leaders, highly qualified personnel (HQP) and other stakeholders; a bibliometric review, an international comparative review and case studies of eight projects. Additional information regarding the Science‑Metrix evaluation methodology can be in found in Appendix C.

The current study included a further document review, literature review, and interviews, the details of which are explained below.

2.4.1 Document Review

The review included the Science‑Metrix evaluation report, federal budgets and Speeches from the Throne, Treasury Board Submissions and other relevant policy documents, departmental Reports on Plans and Priorities, Departmental Performance Reports, as well as Genome Canada Strategic Plans and Annual Reports.

2.4.2 Literature Review

This review was conducted primarily to address the core evaluation issues of continued need and federal roles and responsibilities. Specifically, the literature review examined the continued need to increase research and innovation capacity and the role of federal funding in supporting R&D in Canada and other jurisdictions.

2.4.3 Interviews

The objective of the interviews was to collect information, including views, explanations and factual information from Industry Canada and Genome Canada management that address the evaluation questions, as well as to obtain additional information/clarification in relation to the Science‑Metrix evaluation report where required. A total of six interviews were conducted, four with Genome Canada representatives, and two with Industry Canada representatives.

2.5 Limitations of the Data Collection

This evaluation relied primarily on the Science‑Metrix evaluation for findings related to performance. As such, this evaluation faces the same data limitations as the Science‑Metrix evaluation. The key limitations and mitigation strategies are outlined in Appendix D.

3.0 Findings

3.1 Relevance

3.1.1 Is there a continued need for large‑scale genomics research as supported by Genome Canada? Does Genome Canada respond to this need?

General S&T research needs in Canada

Science and technology, specifically the higher education sector, is widely recognized as important to achieving the Government of Canada's innovation and S&T strategies, as well as providing broader social, economic and cultural benefits.^{Footnote 8},^{Footnote 9},^{Footnote 10} Canada's performance in business innovation is relatively poor^{Footnote 11}, pointing to a need to increase and diversify the forms of support for academic‑industry collaboration, which should be provided in addition to support for business and higher education sector R&D. Given its inherent link to the innovation system, S&T research provides a means by which to address Canada's widening labour productivity gap relative to the United States.^{Footnote 12}

The need for large‑scale genomics research as supported by Genome Canada

The economic benefits genomics has to offer are significant and are reflected extensively in the literature. For example, a 2009 Organization for Economic Co‑operation and Development (OECD) report recognized genomics as one of the most important platform technologies that will fuel the development of a global bio‑economy, placing genomics at the heart of the world's economic development for years to come. The OECD projected that "…biotechnology could contribute to 2.7% (or about US$1.1 trillion) of the Gross Domestic Product (GDP) of OECD countries in 2030."^{Footnote 13} Reinforcing the OECD projection, in 2011, the Centre for the Study of Living Standards (CSLS) estimated that biotechnology could represent up to 3.99% (or about $144 billion) of Canadian GDP in 2030, driven by factors such as increased demand for food, energy, and healthcare.^{Footnote 14}

Documentary and interview evidence indicates that genomics research projects are often complex, multi‑disciplinary, and involve the engagement of a broad range of biological sciences, advanced computational power and advanced engineering that result in large scale, costly projects. Furthermore, existing organizations, including Government departments, universities and research institutions, the private sector and the provinces lack the capacity to support the development of the technical platforms and large‑scale research projects required to advance genomics optimally. The Government fills a gap by funding Genome Canada as a private, not‑for‑profit corporation, with a mandate to develop and implement a national strategy in genomics research for the benefit of all Canadians.

Justification for large‑scale investments is also provided in the Science‑Metrix evaluation, wherein interviewees report that Genome Canada's support is critical for large‑scale strategic genomics projects, and more than three quarters of survey respondents agreed. These projects often require access to complex infrastructure, involve a large number of collaborators and personnel, and generate large datasets.

Evaluation evidence also indicates that no other organization operating across Canada provides a similar level or type of funding for such genomics projects. Federal granting councils, that is, the Canadian Institutes of Health Research (CIHR), the Natural Sciences and Engineering Research Council (NSERC) and the Social Sciences and Humanities Research Council (SSHRC) invest in genomics research to some extent but tend to focus on more specific areas and smaller scale projects.

Genome Canada's response to these needs

Genome Canada responds to the needs outlined above through investing in large‑scale genomics initiatives in sectors of strategic and economic importance to Canada. Target sectors include health, agriculture, environment, forestry, fisheries, energy and mining. Genome Canada's 2012–2017 Strategic Plan articulates the importance of each sector to Canada's economic prosperity, its challenges, and the potential contribution of genomics to ensuring future growth and international competitiveness. Results from Science‑Metrix interviews and surveys suggest that Genome Canada is focusing on the right target sectors.

Genome Canada takes a multidisciplinary approach to fostering innovation as a means to generate socio‑economic benefits for Canadians.^{Footnote 15} According to the Jenkins Report (2011), an ecosystem such as this is essential for effective collaboration between business and higher education, and eventual commercialization of research.

Since 2008, emphasis is increasingly being placed on the translation aspects of research with a view to accelerating discoveries into applications. Genome Canada explicitly acknowledges this in its 2012–2017 Strategic Plan, a key objective of which is to "respond to societal needs by generating discoveries and accelerating their translation into applications". Key activities in this regard include: involving downstream experts and end‑users in priority setting and program design; and funding projects with a high potential for translation into practical applications while continuing to fund research and support cutting‑edge technology in order to prime the "pipeline" of innovation. Science‑Metrix notes that several international organizations in countries such as the US, Norway and China are also following a translational trend.

3.1.2 To what extent is the support for Genome Canada aligned with the priorities of the federal government and the strategic outcomes of Industry Canada?

The original federal S&T Strategy, Mobilizing Science and Technology to Canada's Advantage released in 2007 outlines the Government's intention to foster three distinct Canadian S&T advantages: a Knowledge Advantage, a People Advantage, and an Entrepreneurial Advantage. Genome Canada's activities are aligned with all three as follows:

• Knowledge Advantage: is based on the premise that Canadians must be positioned at the leading edge of the important developments that generate health, environmental, societal, and economic benefits. Genome Canada is directly aligned with this priority by supporting large‑scale genomics research initiatives in sectors of strategic and economic importance to Canada.
• People Advantage: is based on the premise that Canada must be a magnet for the highly skilled people needed to thrive in the modern global economy with the best‑educated, most‑skilled, and most flexible workforce in the world. Genome Canada is aligned with this priority by funding research that attracts and retains leading genomics researchers to Canada and enhancing opportunities for post‑secondary students.
• Entrepreneurial Advantage: is based on the premise that Canada must do more to translate knowledge into commercial applications. Genome Canada supports this translation by actively facilitating collaborations between the academic and private sectors.

The Government's updated Science, Technology and Innovation Strategy, Seizing Canada's Moment: Moving Forward in Science, Technology and Innovation was released in December 2014 and reinforces these directions with a focus on three pillars: Knowledge, People and Innovation.

Recent federal budgets have reiterated the Government's commitment to S&T priorities and to Genome Canada's activities specifically. Some examples of these references are as follows:

• Budget 2011: "Maintaining Canada's leadership in genomics research" is explicitly highlighted as a means to strengthen Canada's research advantage. Genome Canada is provided an additional $65 million to "continue its ground‑breaking work."
• Budget 2012: Reiterates the Government's priority to support jobs and growth by providing additional funding to help Canada reinforce its leadership in fundamental research, including $60 million for Genome Canada.
• Budget 2013: To build on Genome Canada's achievements to date, Economic Action Plan 2013 provides $165 million in 2014–15 to support Genome Canada's multi‑year strategic plan.

Support for Genome Canada also aligns with Industry Canada's priorities under the Science, Technology and Innovation Capacity Program Activity of Industry Canada's Program Alignment Architecture (PAA). This Program Activity contributes to the Strategic Outcome: Advancements in Science and Technology, Knowledge, and Innovation Strengthen the Canadian Economy. This focus is consistent with Genome Canada's mission to lead the Canadian Genomics Enterprise by "connecting ideas and people across public sectors to find new uses and application for genomics; investing in large scale science and technology to fuel innovation; and translating discoveries into applications to maximize impact across all sectors", along with their vision to "harness the transformative power of genomics to deliver benefits to Canadians", both of which focus on delivering benefits derived from genomics research to Canadians.^{Footnote 16}

3.1.3 Does support for Genome Canada align with federal roles and responsibilities?

Industry Canada's Mandate

Industry Canada's mandate for supporting Canadian S&T activities and policy goals stems from the Department of Industry Act, 1995. Subsection 4(1) defines the powers, duties and function of the Minister, which include matters related to industry, technology, and science in Canada. The Minister is responsible for initiating, recommending, coordinating, directing, promoting and implementing national policies, programs, projects and practices with respect to the objectives set out in section 5 of the Act, notably section 5(d) "[to] encourage the fullest and most efficient and effective development and use of science and technology" and section 5(e) "[to] foster and promote science and technology in Canada."

Federal Government Role

The legitimacy of the federal government's role in supporting S&T is supported by a review of policy documents and peer‑reviewed literature. S&T policy‑related documents outlining the rationale for such support were observed in multiple jurisdictions, including the US,^{Footnote 17} Australia,^{Footnote 18} Japan,^{Footnote 19} the OECD,^{Footnote 20} and Europe^{Footnote 21}. These documents all identify a key role for direct and indirect public sector support for research and innovation in order to ensure the country/region achieves a strong and competitive economy. This support is needed for basic and applied research conducted in the higher education sector, including activities that contribute to downstream business innovation.

In the Canadian context, the importance of government support for research in the higher education sector was clearly highlighted in the Jenkins Report (2011), "The federal and provincial governments play an important role in fostering an economic climate that encourages business innovation–for example, by supporting basic and applied research and related training of highly qualified, skilled people […] the higher education and government sectors are key players in Canada's innovation system and complement the role of business."^{Footnote 22} Documentary evidence from the Science‑Metrix evaluation also highlights the importance of national support for genomics in that it helps build national capacity, establish/maintain competitiveness in strategic areas, and position a country internationally.

The provincial governments are key co‑funders of Genome Canada's initiatives and account for the largest percentage (21%) of leveraged co‑funding as of May 31, 2014^{Footnote 23}. Industry Canada staff report that, in addition to supporting Genome Canada‑funded projects in their respective province, some provincial governments (e.g., British Columbia, Ontario, Quebec) invest in large‑scale regional genomics research programs led and managed by their regional Genome Centre, which complement those delivered by Genome Canada and typically focus on regional or provincial priorities.

3.2 Performance

3.2.1 To what extent has Genome Canada contributed to increasing the breadth and depth of knowledge in genomics, including knowledge related to ethical, environmental, economic, legal and social issues (GE³LS)?

The Science‑Metrix evaluation assessed Genome Canada's performance in this area by conducting bibliometric analysis assessing the organization's contribution to increasing genomics research output (i.e., number of peer‑reviewed papers produced), scientific impact (i.e., citations in papers, proportion of papers in the top 10% most cited by researchers) and research focusing on targeted sectors, the results of which indicate that Genome Canada is having an impact in these areas.

Science‑Metrix also examined Genome Canada's contribution to generating actual and potential discoveries among the wider genomics scientific community^{Footnote 24} from supported projects. All lines of evidence in the Science‑Metrix evaluation indicate that Genome Canada has had a positive impact in this regard. Notable examples include work on gene discovery (e.g., rare diseases, cancer, autism), genome sequencing (e.g., salmon, conifer) and identification of important plant and animal traits (e.g., disease resistance, environmental adaptation).

Genome Canada's contribution to increasing knowledge related to broader ethical, economic, environmental, legal and social implications associated with genomics research and its potential applications was also examined. Genome Canada accomplishes this by requiring that these aspects are taken into consideration in all of its research projects and by supporting large‑scale projects focusing exclusively on GE³LS research.^{Footnote 25} The Science‑Metrix evaluation concluded that "Canada has become a world leader in the field of GE³LS research".^{Footnote 26} Bibliometric data analysis showed that Canada ranks third in the production of GE³LS peer‑reviewed papers worldwide and is one of few countries that specialize in this area. Associated survey, case study and interview evidence indicates that Genome Canada has contributed to reinforcing Canada's position as a leader in the field. For example, nearly 80% of GE³LS researchers reported that Genome Canada has effectively increased the quantity of GE³LS research in Canada, while 60% agreed it has increased its quality.

Despite these findings, the Science‑Metrix evaluation revealed differences of opinion among stakeholders on whether GE³LS should be integrated into all large‑scale projects. Genome Canada has already recognized that the integrating model may not be appropriate in all cases, for example targeted, smaller‑scale projects or those projects further down the innovation pipeline, and thus chose not to require that projects funded under the Genomic Applications Partnership Program (GAPP) include a GE³LS component.

Science‑Metrix reports that this approach is generally consistent with what is done in other countries. For example, the National Human Genome Research Institute (NHGRI) and the Wellcome Trust (a global charitable foundation based in the UK, dedicated to achieving extraordinary improvements in human and animal health) ^{Footnote 27} do not require that all funded projects incorporate GE³LS considerations. Some international interviewees also cautioned that requiring GE³LS integration in all funded projects may lead to an inappropriate use of resources, as not all projects carry GE³LS issues.

3.2.2 How effective was Genome Canada in developing a solid base of researchers that are trained to undertake future genomics research?

The Science‑Metrix evaluation provides a number of indications that Genome Canada has contributed to developing a solid base of researchers, also known as highly qualified personnel (HQP) ^{Footnote 28}, trained to undertake future genomics research. The majority (80%) of Science‑Metrix survey respondents indicated that Genome Canada's funding contributed to attracting and retaining HQP in genomics. The case studies conducted as part of the Science‑Metrix evaluation also demonstrate that Genome Canada's research projects are attracting students; more than 260 students participated in the eight projects examined, ranging from 15 to 60 students per project^{Footnote 29}.

Researchers and HQP surveyed as part of the Science‑Metrix evaluation reported that HQP received high quality training in research and gained valuable skills and experience while involved in large‑scale genomics projects. HQP frequently reported having benefited from increased networking opportunities, notably through participation in genomics conferences, colloquia and seminars, as well as collaborations with national and international colleagues. Subsequent to their work on Genome Canada‑funded projects, the majority of HQP were offered academic opportunities or research positions.^{Footnote 30} Evidence from Science‑Metrix case studies also shows that Genome Canada research participants are working in positions in the private sector (such as pharmaceutical companies) and in the public sector (such as conducting research for organizations like Natural Resources Canada's Canadian Forest Service).

3.2.3 To what extent has the provision of enabling technologies contributed to enhance the quality of Canadian genomics research?

Genome Canada supports the provision of enabling technologies primarily by providing funding to five STICs across Canada:

• McGill University and Genome Quebec Innovation Centre (MUGQIC), Montreal
• The Centre for Applied Genomics (TCAG), Toronto
• The Metabolomics Innovation Centre (TMIC), Edmonton & Victoria
• Genomics Innovation Centre at the BC Cancer Agency Genome Sciences Centre, Vancouver
• University of Victoria – Genome BC Proteomics Centre, Victoria

The Science‑Metrix Evaluation concluded that Genome Canada provided adequate and sufficient access to enabling technologies, as well as analytical expertise, mainly through the STICs. These STICs provide researchers with access to high throughput genomics and proteomics technologies, such as DNA sequencing, RNA expression, protein identification and quantitation, as well as new methods and protocols development, data analysis, and bioinformatics. The STICs also assist researchers in the development of research proposals by providing advice on appropriate technologies and study design that improve the quality of the research.^{Footnote 31}

3.2.4 To what extent has Genome Canada contributed to enhance Canada's international profile and visibility in genomics research?

The Science‑Metrix evaluation concluded that "Genome Canada has had a positive influence on Canada's international profile and visibility in genomics research as a result of the high‑impact collaborative work conducted in the context of large‑scale projects and international consortia".

In terms of international visibility, bibliometric analyses conducted by Science‑Metrix indicated that Canada's scientific impact increased over the last 16 years and ranks within the top five countries based on output.

With respect to international profile, while Science‑Metrix noted that Canada's leadership in genomics remains modest compared with larger countries such as the US and UK, it reported that "International interviewees and documents confirm that Canada performs quality research in health, conducting internationally recognized work on autism, cancer stem cells and rare diseases." Positive results were also found in Science‑Metris survey results, wherein over 80% of PIs and 65% of co‑PIs reported that Genome Canada helped the country become a world leader in genomics research.

Finally, it found that "Canada is also internationally recognized as pivotal in coordinating disparate groups working in similar areas of genomics. Specifically, Genome Canada is reputed to be a neutral broker, bringing together different people and ensuring that similar global projects are coordinated appropriately (e.g., International Bovine Sequencing Project, International Cancer Genome Consortium and the International Barcode of Life)".

3.2.5 To what extent has Genome Canada contributed to translating genomics research discoveries into applications leading to socio‑economic benefits?

This section examines Genome Canada's performance based on the following expected result: "The translation of research discoveries into application in the public and private sectors leads to socio‑economic benefits either directly or through development toward, for example, commercialization." Note that the focus is on Genome Canada's contribution to developing applications with the longer‑term objective being that these lead to socio‑economic benefits for Canadians.

Science‑Metrix survey results point to a gradual increase in the development of applications that have, or have the potential to, lead to socio‑economic benefits. The surveys pointed to several different types of practical applications that had already been applied as a result of Genome Canada‑supported research (see Table 1). This was in addition to practical applications that are currently being explored or actively developed. More than two‑thirds of survey respondents said that Genome Canada contributed at least partially to the exploration, development or applied direct or indirect technology and knowledge transfer for new and improved commercial products, processes or services; new or improved health care; and new or improved public policies or programs.

With respect to socio‑economic benefits stemming from these applications, there are well‑known difficulties associated with quantifying such impacts. Such benefits are indirect and widespread, and even when research reaches the translation phase, the time to achieve commercial and broader societal outcomes may take some time and there are many potential intervening variables which create challenges in measurement and attribution.

For these reasons, documentary data tends to focus on projected versus actual economic impacts. For example, one Canadian study estimates that by 2030, biotech development and use (including genomics research) will generate about $144 billion in economic value, equivalent to 3.99% of the GDP in Canada^{Footnote 32}. Genome Canada commissioned a study in 2009 to investigate applying a partial benefit‑cost methodology^{Footnote 33} to assess the impacts of Genome Canada‑supported projects, the results of which suggest that successful Genome Canada projects may produce large economic benefits that exceed the costs of the program. Examples include $230 million in increased forestry revenue over 35 years resulting from a forestry sector genomics research project and $12.2–$14.6 billion in terms of the value of reduced mortality of Canadian infants from a health sector project. While these studies suggest that Genome Canada has contributed to socio‑economic benefits, it is important to remember that such studies are estimates and projections only, and given the challenges associated with assessing such benefits (outlined earlier in this section of the report) cannot be attributed solely to Genome Canada.

Other indications of socio‑economic impact include the number of spin‑off companies created and patent applications. In terms of spin‑off companies, Genome Canada records, as of September 2014, show that Genome Canada‑supported projects were linked to the creation or advancement of a total of 58 companies^{Footnote 34}. Genome Canada also reported that as of June 2014, funded projects had led to more than 350 patent applications and patent awards and 24 licence agreements. In the future, Genome Canada intends to consistently collect data and report socio‑economic related indicators. ^{Footnote 35}

Another indicator of the potential for Genome Canada to lead to socio‑economic benefits is the extent to which the organization has been successful in leveraging external funding.^{Footnote 36} Genome Canada has performed well in this regard as is further discussed in more detail in section 3.2.6 of this report.

Real socio‑economic impacts are also being seen in the sectors being targeted by Genome Canada. For example, in the agriculture sector, Genome Canada has supported research related to the development of genomic selection techniques to boost genetic improvement in cattle. This bovine genome has provided the cattle and dairy industry with the tools to undertake genome selection for desirable traits (e.g., meat quality and milk production), which has yielded annual benefits of over $180 million to Canada and enhanced food safety. In the fisheries sector, an international research project on Atlantic halibut and Senegal sole to find genes related to traits that are attractive to aquaculture producers (such as rapid growth rates and disease resistance) led to a 20% reduction in grow out time to market for halibut.

While there is evidence of positive socio‑economic benefits associated with Genome Canada projects, the Science‑Metrix evaluation concluded that "there is room for improvement regarding the translation of research into practical applications". It suggested that the major barriers to translation include external/contextual factors in the innovation system, such as complex regulatory processes, a funding gap (e.g., in public funding and venture capital) at the product validation stage, limited engagement of the private sector, as well as limited public and end‑user understanding of the benefits of genomics. The external barrier related to a funding gap was also identified as a major barrier by Genome Canada and Industry Canada staff. Specifically, they suggest that Canada does not have large entrepreneurial firms that can consider and further develop potential genomic applications, and that potential opportunities are not being exploited because of this gap.

Stakeholders interviewed as part of the Science‑Metrix evaluation also expressed a need for increased funding support and time to carry out research translation activities (e.g., trials, knowledge translation, partnerships with industry). The report stated that "Genome Canada recognizes that it also needs to conduct more upfront and in‑depth engagement and communication with potential and actual partners".

In this regard, Genome Canada introduced the Genomic Applications Partnership Program (GAPP) in 2013. The objectives of GAPP are to:

• Promote the application of genomics‑derived solutions to address key sector challenges or opportunities facing users ‑ user "pull";
• Promote commercialization of genomics technologies by enabling the transfer of genomics‑derived solutions from academia to users as well as reducing associated risk and incentivizing follow‑on investment from public and private partners;
•  Increase the socio‑economic impact of genomics research by accelerating its translation to application or market; and
• Create and foster a more productive interface between academia and users.

Further, one of the project review criteria for GAPP is that the project provides significant socio‑economic benefit to Canada in the near term (i.e., three to five years).

Genome Canada staff suggest that GAPP is addressing some of the existing barriers to the commercialization of research applications. Also, many Science‑Metrix evaluation interviewees see GAPP as a potential facilitator to translation as it is designed to help reduce the gap between academia and end users/industry, which is seen as one of the main barriers to translation.

The Science‑Metrix evaluation also identified that "there is a need to better define concepts and measures of transformative, translational, and socio‑economic impacts" in order to better tell the full story of Genome Canada's impact. Moving forward, it will be important to ensure that concepts and performance measures related to translation and socio‑economic impact are adequately defined based on the expected results outlined in Industry Canada's funding agreement with Genome Canada, particularly for commercialization focused initiatives such as GAPP.

3.2.6 How efficiently and economically is the program being delivered?

The extent to which Genome Canada is being delivered efficiently and economically was reviewed by assessing the following:

• evidence on the efficiency and economy of Genome Canada; and
• evidence on the efficiency and economy of Industry Canada's management of the Genome Canada program.

Efficiency and economy of Genome Canada

Science‑Metrix reports that, over the period 2008–2009 to 2012–2013, Genome Canada was provided a total of $45 million by Industry Canada for operating expenses but was able to conduct its operations for only $33.3 million.^{Footnote 37} Genome Canada also reduced the annual base funding provided to the Genome Centres for their operations from $5.5 million (2009–10 to 2011–2012) to $4.8 million (2012–2013). Through these savings, Genome Canada was able to make more funds available for projects.

It is common to assess the efficiency of third‑party delivery organizations by looking at the ratio of operating to total costs. Genome Canada's operating percentage from 2008–09 to 2013–14 ranged from 13.2% to 19.7%^{Footnote 38}, which is higher than the 3% to 5% typically observed with the granting councils.^{Footnote 39} However, interview and documentary evidence indicate that Genome Canada's operations are fundamentally different from granting council operations. Genome Canada focuses on funding large‑scale research projects that are very goal‑oriented, with specific milestones and deliverables. As a result there is a considerable amount of management effort involved in monitoring these projects. Genome Canada also invests considerably more time leveraging external funding compared to the granting councils.

Additional key findings based on the Science‑Metrix Evaluation Report in this area are as follows:

• Regional Model: Science‑Metrix evaluation interview and survey evidence identified duplication in communication efforts among the Genome Centres and between Centres and Genome Canada "that will require particular attention in the coming years", resulting from a lack of coordinated efforts (e.g., messaging, sharing of tools and expertise).^{Footnote 40}
• Performance Measurement: The Science‑Metrix evaluation found that while Genome Canada has made efforts to improve its performance measurement practice, steps remain to be taken toward the implementation of a fully effective performance measurement system and a number of weaknesses need to be addressed, specifically in relation to pulling information from existing databases and adequately defining concepts and measures of translation and socio‑economic impacts.

Finally, Science Metrix found no evidence that Genome Canada should fundamentally revisit its current design and delivery model. Generally, interviewees and survey respondents were supportive of Genome Canada's mandate to dedicate funding specifically to genomics research in the form of large‑scale projects. These features (i.e., dedicated funding and focus on large‑scale projects) were in fact considered by surveyed researchers as the organization's best features, contributing to international competitiveness and visibility.

Efficiency and Economy of Industry Canada Management

In funding Genome Canada, Industry Canada is employing a third party delivery model. One of the principal benefits of employing an independent organization is the ability to leverage funding from other sources^{Footnote 41}. Genome Canada has performed well in this regard. The program secured a total of $431 million from external sources from 2009–2013, which exceeded Industry Canada's co‑funding leverage ratio target of 1:1 ratio. Beginning in 2012, it has committed to a 2:1 ratio and, as per Genome Canada's recently published 2013–2014 Annual Report, has exceeded this ratio (2.1:1) based on all new programs launched since then.^{Footnote 42}

In addition to the ability to leverage, Genome Canada and Industry Canada representatives noted a number of advantages associated with a third‑party delivery model that would not be possible were Genome Canada administered directly by the federal government. Examples include an increased ability to take risks and partner with industry that leads to greater external funding targets and achievements; and greater flexibility than through a direct government program to develop and adapt programming to best meet the needs of targeted R&D in a timely manner.

The literature indicates that a drawback of a third party delivery model is the additional transaction costs incurred by both parties. In the case of Genome Canada, there do not appear to be significant transaction costs resulting from the delivery structure as Industry Canada currently allocates roughly 0.5 FTE to manage the delivery of the contributions. While third‑party delivery has been seen increasingly in other international jurisdictions, not all countries are using this model to support large‑scale genomics research. International representatives interviewed as part of the Science‑Metrix evaluation suggested that the third party delivery model is appropriate in the Canadian context due to the modest size and broad regional distribution of the research community.

4.0 Conclusions and Recommendations

The following are the conclusions to each of the nine evaluation questions.

4.1 Relevance

Regarding the relevance of the program, the evaluation determined that:

• There is a continued need for increased S&T research and innovation capacity as a means to provide social and economic benefits to Canadians. Genome Canada responds to these needs by investing in large‑scale genomics initiatives in sectors of strategic and economic importance to Canada. Large scale research is appropriate in light of the complexity, multi‑disciplinary nature, required engagement of a broad range of biological sciences, advanced computational power and advanced engineering associated with genomics research.
• Support for Genome Canada is consistent with federal government priorities as set out in the 2007 S&T Strategy and subsequent Federal Budgets. Genome Canada's objectives are also in line with Industry Canada's strategic outcome related to advancing S&T, knowledge and innovation in order to strengthen the Canadian economy.
• Support for Genome Canada is consistent with federal roles and responsibilities to encourage the development of S&T and aligns with Industry Canada's mandate to foster and promote S&T in Canada.

4.2 Performance

Regarding the effectiveness of the program, the evaluation determined that:

• Overall, evidence suggests that Genome Canada is achieving its expected results. Genome Canada has increased the breadth and depth of knowledge in genomics as demonstrated by its contributions to increased genomics research output, scientific impact and research focusing on sectors of strategic importance to Canada, as well as by generating discoveries within the wider genomics scientific community.
• Genome Canada has increased knowledge related to the ethical, environmental, economic, legal and social issues of genomics (GE³LS) by requiring that these aspects be considered in all of its research projects; however results indicate that GE³LS may not be appropriate for all genomics research projects.
• Projects supported by Genome Canada have enhanced Canada's international profile and visibility in genomics research and have contributed to developing a solid base of researchers trained in to undertake future genomics research. Genome Canada has also facilitated genomics research by providing leading‑edge technologies through funding the STICs.
• Genome Canada is contributing to translating genomics research discoveries into applications and there is evidence of socio‑economic benefits associated with Genome Canada‑supported projects. Nonetheless, better defined concepts and measures of translational and socio‑economic benefits could help tell the full story of Genome Canada's impact. Moving forward, it will be important to ensure that these are adequately defined, particularly for commercialization‑focused initiatives such as the recently launched Genomic Applications Partnership Program (GAPP).
• Genome Canada made efforts to improve its efficiency during the evaluation period and exceeded Industry Canada's target 1:1 co‑funding leveraging ratio for the evaluation period. Specific areas for improvement included the need for more coordinated communication efforts across the Genome Centres and between Genome Canada and Centres, as well as ongoing improvements in performance measurement. Evaluation evidence suggests that the third‑party delivery model being employed by Industry Canada is an efficient means to support large‑scale genomics research in Canada.

4.3 Recommendations

The Science‑Metrix evaluation developed five recommendations for Genome Canada (see Appendix A). Genome Canada has developed a management response for these recommendations. As an observer on Genome Canada's Board of Directors, Industry Canada will be in a position to follow the implementation of these recommendations.

In response to the findings of this evaluation, and consistent with the finding of the Science Metrix evaluation, it is recommended that:

• Industry Canada's Science and Innovation Sector (SIS) work with Genome Canada as required on defining and measuring concepts related to translational and socio‑economic impacts based on the expected results of the funding agreement.

• Appendix A
• Appendix B
• Appendix C
• Appendix D

Appendix A: Science‑Metrix Evaluation recommendations, key supporting evidence and corresponding AEB analysis/recommendations

Below are the recommendations from the Science‑Metrix evaluation of Genome Canada that were put forth relating to Industry Canada's 2014 contribution agreement with Genome Canada. Genome Canada developed an action plan that was approved by its Board of Directors on June 14, 2014. Of note, Science‑Metrix prefaced its recommendations by stating that they should be considered as a means to complement and/or help focus on these continuous improvements already being made by Genome Canada in some of the related areas.

Appendix B: Genome Canada Program Logic Model

Genome Canada Program Logic Model

View larger image

Appendix C. Overview of Science‑Metrix Evaluation Data Collection Methodology

The Science‑Metrix evaluation employed five data collection methods:

• Management and delivery review: As part of the review, 24 interviews were conducted, and a document and file review of over 200 documents related to Genome Canada's activities was performed. The purpose of the management and delivery review was to examine the history, processes and performance of Genome Canada across the evaluation period (2009–2014). Interviews were used to inform issues of continued need, efficiency and economy, whereas documents were one of the main sources of evidence – including quantitative evidence (i.e., financial and output data) – on the achievement of outcomes, as well as efficiency and economy issues.
• Survey: Web surveys were conducted with five stakeholder groups, namely the Principal Investigators (PIs; n=53), co‑applicants and other investigators (co‑PIs; n=153), GE³LS PIs and leaders (n=52), highly qualified personnel (HQP; n=153) and other stakeholders (e.g., partners, collaborators, current and potential end‑users; n=137) involved in Genome Canada‑supported projects. These surveys mainly sought to collect information on performance issues to address data gaps relating to outcomes. Views regarding cost‑effectiveness and delivery were also collected.
• Bibliometrics: Using various indicators (e.g., scientific output; specialization, citation impact), Science‑Metrix assessed the scientific performance of Genome‑Canada's funded researchers in genomics and in each of its strategic sectors, namely agriculture, health, environment, fisheries/aquaculture, forestry and energy/mining. This performance was also examined within the broader Canadian context (e.g., comparison to non‑funded researchers, benchmarking with other countries), in an attempt to determine Genome Canada's contribution to the national standing in genomics over the years. The level of national and international scientific collaboration of Genome Canada and of individual researchers was also measured, as an indicator of the organization's effectiveness in coordinating genomics research efforts.
• Case studies: Eight projects funded by Genome Canada were examined in depth as part of the case study method. The selection of case studies focused on emerging and natural resource sectors, namely agriculture, environment/energy/mining, fisheries, and forestry in order to address particular needs relating to the implementation of the 2012–2017 Strategic Plan. Each case involved conducting two or three interviews with relevant stakeholders (e.g., project contributors, partners and/or end users). A project‑level document review was also conducted, examining applications, quarterly, interim and annual reports as well as information such as collaboration agreements, scientific publications and any other related outputs as provided by Genome Canada and interviewees. The purpose of the case studies was to provide insight on key themes that span across projects and that relate to the success and impact of large‑scale genomics projects.
• International comparative review: The data collection and analysis for the international comparative review were based on two methods: a literature review of five organizations comparable with Genome Canada and targeted interviews with representatives from three of these organizations. The literature review component focused on identifying and extracting relevant information and data from the organizations' websites and other relevant sources such as grey literature, funding announcements, etc. A total of six interviews were also conducted with individuals knowledgeable about their organization and/or the country's overall support mechanisms for genomics research. This line of evidence was used to assess the continued need for national support of genomics research and contribution of Genome Canada to Canada's global leadership in this field, as well as to position Genome Canada in the global context according to elements such as operating environment, strategy development, design and delivery of programs or projects, and best practices.

Appendix D. Data Collection Limitations

This evaluation relied primarily on the Science‑Metrix evaluation for findings related to performance. As such, this evaluation faces the same data limitations as the Science‑Metrix evaluation. The key limitations and mitigation strategies are outlined in Appendix D. discussed below:

• Interviews: Relatively few individuals within each external stakeholder sub‑group were interviewed; responses are thus not representative of the whole population. The use of other lines of evidence partially compensate for the limited number of unfunded researchers interviewed.
• Document and file review: Data from the document/literature reviews did not always provide evidence to show the relationship between activities and outcomes (i.e., attribution), particularly when there may be other factors/players contributing to the outcomes. Information from the document and file review were considered in combination with primary data from the interviews consulted, in order to gain a better understanding of the contribution of Genome Canada to outcomes.
• Survey: Small sample size combined with low response rates limited possible analyses for some surveys and/or questions. Pre‑notice emails were sent by Genome Canada to potential survey respondents. Reminders were also sent at regular intervals by Science‑Metrix. Bounced back emails were verified manually in order to find other contact information. Nonetheless, the margins of error remain slightly higher than 5% (between 6.1% and 10.5%). The survey results should therefore be interpreted with caution, in particular for distributions that are close to 50–60%, and especially PI and GE³LS survey results.
• Case studies: Relatively few individuals for each case could be interviewed in the interest of time and level of effort required. To the extent possible, interview findings were verified against all available documentary material for each case.
• International comparisons: The Science‑Metrix evaluation makes a number of observations based on the views of international interviewees. It should be noted that Science‑Metrix interviewed only six international representatives of four countries, so these views may not be representative of the international community at large.