Evaluation of the Perimeter Institute for Theoretical Physics

PDF version

273 KB, 28 pages

March 2017

Presented to the Performance Measurement and Evaluation Committee on March 30, 2017

Approved by the Deputy Minister on May 19, 2017

Table of Contents

• Executive Summary
• 1.0 Introduction
  • 1.1 General description of Perimeter
  • 1.2 Program objectives
  • 1.3 Program mandate and activities
  • 1.4 Program governance
  • 1.5 Logic model
• 2.0 Methodology
  • 2.1 Evaluation scope and objectives
  • 2.2 Evaluation approach
  • 2.3 Evaluation questions
  • 2.4 Data collection methods
  • 2.5 Data limitations
• 3.0 Findings
  • 3.1 Relevance
  • 3.2 Performance
• 4.0 Conclusions and recommendations
  • 4.1 Conclusions
  • 4.2 Recommendations
• Summary of the evaluation of the Perimeter Institute for Theoretical Physics

List of acronyms

AEB

Audit and Evaluation Branch

CAUBO

Canadian Association of University Research Administrators

CERN

The European Council for Nuclear Research

CHIME

Canadian Hydrogen Intensity Mapping Experiment

CTP

Centre for Theoretical Physics (MIT)

DGP

Discovery Grants Program (NSERC)

EHT

Event Horizon Telescope

FAA

Financial Administration Act

FTE

Full-time Equivalent

IQC

Institute for Quantum Computing

ISED

Innovation, Science and Economic Development Canada

ISSYP

International Summer School for Young Physicists

KITP

Kavli Institute for Theoretical Physics (University of California)

LHC

Large Hadron Collider

LIGO

The Laser Interferometric Gravitational-Wave Observatory

MIT

Massachusetts Institute of Technology

NSERC

Natural Sciences and Engineering Research Council

OECD

Organization for Economic Cooperation and Development

PDF

Post-Doctoral Fellow

Perimeter

Perimeter Institute for Theoretical Physics

Princeton IAS

Princeton University, Institute for Advanced Study

SAC

Scientific Advisory Committee

SKA

Square Kilometre Array

SNOLAB

Sudbury Neutrino Observatory Laboratory

STEM

Science, Technology, Engineering and Math

UK

United Kingdom

US

United States

Table

Table 1: Operational benchmarking percentage of administration expenses

Figure

Figure 1: Perimeter logic model

Appendice

Appendix A: Third-party evaluation recommendations

Executive Summary

Program overview

Perimeter Institute for Theoretical Physics (Perimeter) is an independent, resident-based research institute that supports scientific research, training and educational outreach in foundational theoretical physics. Founded in 1999, Perimeter's mission is to advance the understanding of the universe at the most fundamental level, stimulating breakthroughs that could transform our future.

Evaluation purpose and methodology

The purpose of this evaluation was to assess the relevance and performance of Perimeter. The evaluation covered the period from April 2012 to December 2016 and expanded on the June 2016 third-party evaluation Perimeter undertook in accordance with its funding agreement. The evaluation employed four primary data collection methods including a document review, literature review, interviews and operational benchmarking.

Findings

Relevance

There is a continued need for the foundational research conducted by Perimeter and its excellence objectives as it contributes to Canada's overall strength and competitiveness in scientific knowledge generation internationally and supports broader societal objectives to develop science literacy and skills to advance innovation in Canada.

The federal government has a role in the funding of foundational or basic research as a general contribution to societal need. Without such public sector intervention, the private market would likely not adequately support certain types of research. Further, Perimeter's objectives are aligned with federal government priorities to invest in world-class institutions and research, as well as highly-skilled people, to drive innovation.

Performance

Perimeter has significantly added to Canadian capability in foundational theoretical physics through its many scientific achievements, attraction of a very high caliber of researchers and deep linkages to experimental and observational research centres. This is successfully positioning Canada as a world leader, and Perimeter's influence on Canada's reputation is higher than just five years ago.

The institute is having a positive effect on recruitment, within and to Canada, of top quality research trainees through unique programming and student exposure to top researchers across many fields. While overall trainee recruitment is highly successful, the attraction of top undergraduate students may not be on par with other categories of researchers.

Perimeter's education and outreach program is highly regarded and demonstrates considerable effectiveness in promoting science and physics through the noted high use of its training materials. High school teachers, students and public audiences continue to increase and positive influences from Perimeter's efforts have been identified.

Perimeter has been able to implement a model supporting the efficient delivery of its programming through high ratios of private/public funding, maintenance of uniform administrative costs and the implementation of initiatives targeted to improve operating efficiency. Its ratio of indirect costs to total costs of all program activities is in line with the cost ratio ranges of other research institutes.

ISED is meeting its planned commitments in relation to expenditures of FTEs in the management of Perimeter's funding agreement. It has also met set service standards in relation to the execution of the agreement.

Recommendations

Overall, the evaluation results demonstrated Perimeter's effectiveness in achieving performance results and its ongoing relevance. No recommendations resulted from this evaluation.

The recommendations highlighted in the third-party evaluation are broader than the expected results stipulated in ISED's funding agreement and focus on continued improvement. These can be found in Appendix A.

1.0 Introduction

This report presents the results of an evaluation of ISED's contribution to the Perimeter Institute for Theoretical Physics.

The purpose of the evaluation was to assess the relevance and performance of Perimeter. The report is organized in four sections.

• Section 1.0 provides the program context and profile of Perimeter;
• Section 2.0 presents the evaluation methodology along with a discussion of the limitations to the evaluation;
• Section 3.0 presents the findings pertaining to the evaluation issues of relevance and performance; and
• Section 4.0 summarizes the evaluation conclusions and outlines the recommendations.

1.1 General description of Perimeter

Located in Waterloo, Ontario, the Perimeter Institute for Theoretical Physics was founded in 1999 through significant private philanthropic investments and subsequent public-private partnerships with the federal and Ontario governments.

Perimeter is an independent, resident-based research institute that supports scientific research, training and educational outreach in foundational theoretical physics. Its mission is to advance the understanding of the universe at the most fundamental level, stimulating breakthroughs that could transform our future.^{Footnote 1}

The institute's resident-based research operations began in 2001 and, tied to growth, moved into a custom-built 55,000 square foot facility in 2004. In the fall of 2011, Perimeter completed a significant expansion, the Stephen Hawking Centre, which increased the size of the facility to 120,000 square feet. Perimeter can now accommodate over 250 scientists and research trainees.

1.2 Program objectives

Perimeter aims to vault Canada to a leading position at the frontier of modern science and brand the nation as a world leader in basic research in physics. As stated in Perimeter's most current Corporate Five-Year Plan (2012-13) its vision is:

To create the world's foremost centre for foundational theoretical physics, uniting public and private partners, and the world's best scientific minds, in a shared enterprise to achieve breakthroughs that will transform our future.

The current 2012-2017 ISED multi-year funding investment of $50 million over five years to Perimeter outlines the following objectives:

1. Strengthen the capacity of the Institute to undertake world-class research by increasing the number of researchers and increasing engagement with experimental and observational research centres;
2. Expand the capacity of the Institute to attract and train the next generation of researchers in theoretical physics, including both masters and PhD programs; and,
3. Enhance and expand the Institute's public education and outreach activities to effectively promote science and physics and demonstrate how the research from theoretical physics can be applied.

1.3 Program mandate and activities

Research is the first pillar of Perimeter's mandate, for which it provides a dynamic atmosphere of scientific interaction to enable resident and visiting researchers to focus on profound issues in several overlapping sub-disciplines of basic research. Perimeter focuses on nine research fields including Condensed Matter, Cosmology, Mathematical Physics, Particle Physics, Quantum Fields and Strings, Quantum Foundations, Quantum Gravity, Quantum Information, and Strong Gravity. Perimeter strongly encourages interactions amongst researchers with different scientific orientations and specializations, developing a culture where both orthodox and more speculative approaches are pursued simultaneously in a highly cooperative manner.

Perimeter also works to collaborate constructively with the surrounding academic community, in particular by co-recruiting, fostering joint-hires, and creating educational research opportunities for graduate students. The training of young scientists is the second pillar of its mandate.

The third pillar of Perimeter's mandate is the provision of an educational outreach program that raises scientific literacy and conveys the importance of theoretical physics. Efforts focus on high school students, teachers and the public across Canada, along with select international engagements and resource sharing.

1.4 Program governance

Perimeter is an independent federally incorporated registered charity governed by a volunteer Board of Directors drawn from the private sector and academic community. The Board is the final authority on all matters related to the general structure and development of the Institute. The federal government does not perform direct oversight or management functions. Rather, it supports Perimeter Institute as a centre of research excellence through the contribution of funding, and attracting support from other levels of government as well as the private sector.

Perimeter is led by an Institute Director who reports directly to the volunteer Board of Directors. The Institute Director is responsible for developing and implementing the overall strategic direction of the organization. A Scientific Advisory Committee (SAC), an oversight body composed of international scientists that help ensure objectivity and a high standard of scientific excellence, supports the Board of Directors and Institute Director. The SAC provides third party, independent reviews and reports to the Board on Perimeter's scientific progress.

1.5 Logic model

The logic model in Figure 1 depicts Perimeter's program theory. It illustrates how Perimeter's activities are expected to lead to certain outputs and various levels of outcomes and ultimately to one of ISED's strategic outcomes.

The logic model covers all of Perimeter's activity areas including research, training and outreach. It was developed in 2012 as part of Perimeter's Performance Measurement Strategy.

Figure 1: Perimeter Logic Model

2.0 Methodology

2.1 Evaluation scope and objectives

Perimeter undertook an independent third-party evaluation that was completed in June 2016^{Footnote 2} (hereafter referred to as the third-party evaluation) in accordance with its federal funding agreement with ISED, which assessed performance against its logic model.^{Footnote 3} As part of ISED's commitments under the Financial Administration Act (FAA), it was required to conduct its own evaluation of Perimeter in 2016-17. The scope of this evaluation built upon the results of Perimeter's third-party evaluation, covering all core evaluation issues of relevance and performance from the TB Policy on Evaluation.^{Footnote 4}

This evaluation focuses on evaluating Perimeter's performance against the objectives outlined in the 2012-17 ISED multi-year funding agreement (to which the Perimeter logic model elements align).

The evaluation covers the period from April 2012 to December 2016 (similar to the third-party independent evaluation) in alignment with Perimeter's current funding agreement with ISED.

2.2 Evaluation Approach

The evaluation was managed by the Audit and Evaluation Branch, and was conducted by KPMG.

The evaluation approach relied on evidence derived from various sources and methods. An analysis was undertaken to identify gaps in the information and results obtained from the June 2016 third-party evaluation of Perimeter in comparison to the current ISED evaluation requirement. The results focused this evaluation to target primary data collection efforts towards the issues of relevance and efficiency and economy. Performance questions were addressed by leveraging the results of the third-party independent evaluation.

2.3 Evaluation questions

Based on Perimeter's Performance Measurement Strategy and consultations with ISED's Audit and Evaluation Branch (AEB) and Program Coordination Branch (within the Science and Innovation Sector), the evaluation addressed the following questions.

Relevance

1. To what extent does Perimeter continue to address a demonstrable need and be responsive to the needs of Canadians?
2. Are Perimeter's objectives and intended outcomes consistent with the federal government's policy goals and priorities and the strategic objectives of Innovation, Science and Economic Development Canada?
3. Is the funding of Perimeter Institute consistent with federal roles and responsibilities?

Performance

4. To what extent has Perimeter strengthened its capacity to undertake world-class research, including increasing the number of researchers and increasing engagement with experimental and observational research centres?
5. To what extent has Perimeter expanded its capacity to attract and train the next generation of researchers in theoretical physics, including both masters and PhD programs?
6. To what extent has Perimeter enhanced and expanded its public education and outreach activities to effectively promote science and physics and demonstrate how the research from theoretical physics can be applied?
7. Have Perimeter's structures and processes facilitated efficient delivery of its programming?
8. To what extent is Innovation, Science and Economic Development Canada efficiently managing its contribution to Perimeter Institute?

2.4 Data collection methods

Described below are the lines of evidence used to address the evaluation questions.

Document Review

The document review supported the understanding of broad federal government priorities and its role in developing and supporting basic research as part of the innovation continuum, as well as ISED-specific objectives in this area. Examples of key documents reviewed include public announcements on the Innovation Agenda, mandate letters for the Ministers of Science and Innovation, Science and Economic Development, federal Budget documents, and ISED's Reports on Plans and Priorities. Additional documentation was collected and reviewed to support the assessment of economy and efficiency including a review of Perimeter's most recent initiatives for cost-efficiency.

The document review was also used to summarize the performance results from Perimeter's 2016 third-party evaluation. The third-party evaluation employed a methodology consisting of interviews with Perimeter's Scientific Advisory Committee and Advancement Partners^{Footnote 5}; five web surveys^{Footnote 6}, and a document and performance data review.

Literature Review

A review of published and grey literature informed the external representation of the significance of foundational research and scientific discovery to the innovation continuum as well as the importance of federal support in this area.

Interviews

Qualitative information related to relevance and efficiency and economy were gathered through interviews. Participants included the following groups of stakeholders:

• Perimeter Senior Management (3)
• ISED Program Coordination Branch (2)
• Perimeter Board Members (2)

Operational Benchmarking

An operational benchmarking exercise was undertaken to attempt to learn about the ranges of administrative cost ratios of institutions similar to Perimeter to assess cost-efficiency. Initial research on 14 other research and/or research-oriented organizations was completed by ISED's Program Coordination Branch and was leveraged to determine the feasibility of conducting the review as well as identifying possible institutes/organizations of interest. Primary desk research was conducted on many organizations but three scenarios were used to model results from the following institutions: Massachusetts Institute of Technology (MIT) and the Centre for Theoretical Physics (CTP), Princeton University Institute for Advanced Study (IAS), and the Canadian Association of University Research Administrators (CAUBO)^{Footnote 7}. The operational benchmarking supported learning with respect to ranges of cost-efficiency results in similar organizations.

Economy and Efficiency Analysis

Perimeter's financial data and policy, procedures and recent infrastructure efficiency data were used to assess its leveraged funding position, trend in operational efficiency and continuous improvement efforts.

2.5 Data limitations

Potential Bias in Opinion Data

In developing the interview sample, the number of stakeholders who had broad enough exposure to Perimeter and its core elements was limited due to the recent completion of the third-party evaluation and the risk of interview fatigue. During the design phase of the evaluation, it was determined that the individuals holding the broadest knowledge of Perimeter and all its core elements included only those individuals involved in Perimeter's governance structures and those involved in managing the ISED funding agreement. The opinions and perspectives of these two groups may be somewhat biased due to their closeness to Perimeter. However, the possibility of this limitation is believed to be minor. The evaluation mitigated this limitation through triangulation of findings from the document and literature review in addition to the interview and survey results (which covered a broad range of stakeholders) from the third-party evaluation.

Strength of Operational Benchmarking

Completing a fair determination of a cost-efficiency figure for Perimeter based on what appears to be a simplistic comparison of administrative expenses versus other program costs was limited by the following factors:

1. Comparability of models: There is no other institute fully comparable to Perimeter in terms of mission or model.
2. Cost comparisons: The lack of comparability means that the “cost” side of the cost-efficiency equation is virtually impossible to define. There are simply too many differences among models, including differences in:
   • Type of activities funded: The nature of activities funded varies widely between institutions. For example, some institutes may support research only, and the research supported may be in the sense of providing only research grants, or providing all the resources required (e.g., the base research funding, but also salaries, buildings, lab and teaching space within those buildings, utilities, roads and parking, etc.). Each represents demonstrative value-add benefits, but adds to costs in hard-to-determine ways, and the indirect costs for these components (as opposed to for the institute overall) is often never known.
   • Categorization of costs: The problem above is exacerbated where there are inconsistencies in the categorization and/or allocation of indirect costs between organizations. For example, in some cases, indirect costs may be represented by specific line items in the organization's overall budget but some may be allocated within research program budgets. Others may be included within another institution's budget, such as a large host university.
3. Lack of data: The detailed data necessary to calculate cost-efficiency are often missing or insufficiently detailed to permit comparability or consistency in analysis.

These factors limited the extent to which evaluators could reliably benchmark Perimeter's operations and thus draw conclusions regarding Perimeter's cost-efficiency.

3.0 Findings

3.1 Relevance

3.1.1 To what extent does Perimeter continue to address a demonstrable need and be responsive to the needs of Canadians?

Most scientific research is a mix of new knowledge generation and subsequent exploitation. Major innovations are rarely possible without prior generation of new knowledge founded on basic research.^{Footnote 8 Footnote 9}

The importance of and contributions from basic research are cited in literature going back decades. Notable contributions of investment in basic research include increasing useful knowledge, training new graduates, creating new firms, and increasing a country's competitiveness. Several pieces of literature have measured the long-term economic impacts stemming from basic research.

There are four key reasons cited by the International Council for Science regarding the importance of investing in foundational research:

1. Basic and applied science are a continuum. They are inter-dependent. The integration of basic and applied research is crucial to problem solving, innovation and product development.
2. Knowledge is more than the information and data that might be provided via the internet; it is fundamentally a matter of cognitive capability, skills, training and learning. The exploitation and application of scientific information requires skilled scientists with a good understanding of the basic theories and practice of science. The successful transfer of scientific knowledge requires well-trained scientists at both ends of the exchange.
3. Excessive dependency on scientific progress in other countries is rarely likely to lead to the resolution of local problems. Countries need to be able to generate their own scientific knowledge and adapt this to their own local context and needs.
4. The practice of science is increasingly international and the research agenda is set by those who participate. A country with no basic scientific research capacity effectively excludes itself from having any real influence on the future directions of science.^{Footnote 10}

Salter and Martin also find that advanced industrial countries need their own well-developed basic research capabilities in order to appropriate the knowledge generated by others and to sustain technological development.^{Footnote 11}

Perimeter does this strategically, focusing on high-impact fields to bring Canada to the forefront of research and, in part, to support international collaborations. For example, Perimeter has held a number of “first workshops in the world” following up on discoveries made by major science initiatives such as the discovery of the Higgs boson at the Large Hadron Collider (LHC) and planning for future exploitation of the experiments at the European Council for Nuclear Research (CERN).

The 2013 Horizontal Review report by Science Metrix identified that there is ample evidence that basic research (and academic research more generally) is necessary to sustain the innovation system and contribute to Canada's prosperity. Several studies have measured positive long-term economic impacts of academic research, such as through spin-off companies in Canada – particularly in those stemming from basic research in physics (which is of particular interest in the context of the Perimeter Institute).^{Footnote 12} For example, Perimeter and ISED interviewees noted that Perimeter has been the catalyst for the Quantum Valley in Waterloo, which aligns the continuum of theoretical, experimental, and application (venture capital) organizations in quantum information science.

The report also notes the socio-economic benefits generated by basic research are not limited to potential “useful knowledge” that is produced but rather are increasingly seen to include benefits that are indirect, subtle, heterogeneous, and difficult to track or measure, such as increasing a society's learning capabilities or competence building. Salter and Martin's review of studies on the benefits of publically-funded basic research identified the following types of contributions of such research to economic growth:

• increasing the stock of useful knowledge;
• training skilled graduates (i.e., supply of skilled graduates and researchers);
• creating new scientific instrumentation and methodologies;
• forming networks and stimulating social interaction;
• enhancing/increasing the capacity for scientific and technological problem-solving;
• creating new firms; and
• providing social knowledge.^{Footnote 13}

The 2015 report of the Expert Panel on Science, Technology, Engineering and Math (STEM) Skills for the Future suggests that maintaining and developing the skills and abilities of Canadians in a highly trained workforce is central to Canada's future prosperity.^{Footnote 14} The Panel found that “high-quality investments in STEM skills — in both early education and in more advanced training — are critical to Canada's prosperity. Beyond preparing students and the labour force for a range of future possibilities, these investments appear to be one of several components required to improve Canada's poor innovation and productivity record.”^{Footnote 15} The Panel noted several ways STEM benefits society, from breakthrough drugs to safe structures, more efficient and sustainable forms of transportation, convenient apps, and innovative forms of workplace organization as examples.

Recent Organization for Economic Cooperation and Development (OECD) reporting also identifies science literacy as increasingly linked to economic growth and necessary for finding solutions to complex social and environmental problems, where all citizens, not just future scientists and engineers, need to be willing and able to confront science-related dilemmas.^{Footnote 16} The report further found that providing students with more opportunities to learn science will help them to learn to “think like a scientist”, a skill that has become all but essential in the 21^st century (even if students choose not to work in a science-related career later on). Perimeter's education and outreach is targeted to provide opportunities for audiences to develop science literacy skills of critical thinking and problem solving.

Interviews with Perimeter Board members and ISED representatives identified Perimeter's contribution to scientific study in foundational areas of importance to be at a level far above any individual University physics department in Canada. Perimeter is situated outside of (but in close connection with) the Canadian university system. Its “Centre of Excellence” model provides it with increased flexibility in its administration and its role according to Perimeter and ISED interviewees. Interviewees noted that Perimeter is innovative, and operates within a culture of agility and entrepreneurial spirit in strategizing its research efforts unlike a university, where mandates and programming are more rigid, and tightly defined.

Perimeter and ISED interviewees did not see the counterfactual scenario of having the activities/programming of Perimeter carried out by other parties (e.g., the absence of Perimeter) having the same impact. It was noted that quality programs in academic research would be reasonable but fragmented and the concentration of worldwide expertise that Perimeter supports would not be possible.

3.1.2 Are Perimeter's objectives and intended outcomes consistent with the federal government's policy goals and priorities and the strategic objectives of Innovation, Science and Economic Development Canada?

There is strong evidence that Perimeter's objectives have kept pace with government priorities to invest in research and develop and attract highly-skilled people to drive innovation.

Ministerial Mandate letters identify overarching goals and priorities of the government to invest in research. Both the Minister of ISED and the Minister of Science are tasked with supporting scientific research in the government's investments and policy choices. The Minister of ISED is to improve the quality and impact of programs that support innovation, scientific research and entrepreneurship. The Minister of Science is to support science to create sustainable economic growth and support and grow the middle class.

Budget 2016 acknowledged the “creation of knowledge and development of highly-qualified people are vital for Canada's prosperity in the global economy.”^{Footnote 17} The support for investment in discovery research, the recognition of the fundamental role of investigator-led discovery research in an innovative society and the importance of making investments to attract and retain world-class researchers is also laid out.^{Footnote 18} In fact, further federal investment in Perimeter is specifically identified in Budget 2016 so that Perimeter can continue to strengthen its position as a world-leading research centre.

Perimeter's objectives are well-aligned with the Government's Inclusive Innovation Agenda, which underscores the role of world-class researchers in positioning Canada as a global leader in research excellence. The Institute aims to attract and train the next generation of researchers in theoretical physics, which expands Canada's talent pool. It also engages in outreach to promote science and physics as well as raise scientific literacy among high school students, teachers and the public across Canada, which contributes to fostering interest in careers in science, technology, engineering and mathematics (STEM), another component of the Agenda.

Further, Perimeter supports ISED's overall commitment to “support world-class excellence in fundamental research to facilitate new discoveries that lead to business growth, build human capital, and support environmental and social goals for the well-being of Canadians.”^{Footnote 19} Within ISED's Program Alignment Architecture, Perimeter is a sub-component of the Science, Technology and Innovation Capacity Program and Science and Technology Partnerships sub-program, aligning to the strategic outcome “Advancements in science and technology, knowledge, and innovation strengthen the Canadian economy.”

3.1.3 Is the funding of Perimeter Institute consistent with federal roles and responsibilities?

Science in Canada is part of the defined powers, duties and functions of the Minister of ISED under the Department of Industry Act 1995. The Department of Industry Act mandates that science and technology be fostered and promoted in Canada and that the Department encourage the fullest and most efficient and effective development and use of science and technology.^{Footnote 20}

The 2013 Horizontal Review report by Science Metrix found that increasingly, academic research [as conducted by Perimeter] is also being called upon to “contribute to the solution of societal problems and to support innovations and economic growth.” The legitimacy of research institutions is largely based on the perceived benefits they provide, i.e., their ability to contribute to economic competitiveness, cultural enrichment and social progress, which ultimately justify the substantial investments made to them, including via public funds.

Other studies also identify the importance of the role of the federal government in supporting basic research. The independent panel report, Innovation Canada: A Call to Action, identifies both the federal and provincial governments as playing an important role in fostering an economic climate that encourages business innovation by supporting basic and applied research and related training of highly qualified and skilled people.^{Footnote 21} It is also noted that “there are conditions under which markets do not allocate resources efficiently, and governments intervene to try to correct or at least diminish “market failures” — for example, to provide public goods such as basic research that generate benefits for society at large or to overcome problems of inadequate information.

The primary economic rationale for a government's funding role in basic research is that, without such intervention, the private market would not adequately supply certain types of research. Reasons for this include: length of time to generate a result (that leads to profits), concern with private benefit relative to social optimum, and no or limited incentive to invest in national innovation systems.^{Footnote 22Footnote 23Footnote 24}

Perimeter and ISED interviewees also held this opinion. Both Perimeter and ISED interviewees believe the role of government in funding research (and Perimeter) elevates the perceived security of funding for researchers and other contributors, creating a more stable investment for attracting interest and commitments to basic research. It was noted that the government is the needed balance to provide support to both fundamental science and applied efforts.

3.2 Performance

3.2.1 To what extent has Perimeter strengthened its capacity to undertake world-class research, including increasing the number of researchers and increasing engagement with experimental and observational research centres?

Multiple lines of evidence from the third-party Perimeter evaluation indicate that Perimeter has significantly added to Canadian capability and capacity in foundational theoretical physics. The results of the researcher survey indicates that Perimeter has made Canadian research capacity, on average, better to much better in all fields that Perimeter addresses (83%-100% of researchers depending on the field) and a researcher's own research program better due to their affiliation with Perimeter (55%-71% of researchers depending on the area). The results of SAC interviews in the third-party Perimeter evaluation were also very positive about Perimeter's contribution to Canadian capability, in large part due to Perimeter's creation of strong groups of investigators. Perimeter is noted to have attracted a high caliber of researchers and Post-Doctoral Fellows (PDFs) at the top of their fields and against competition from top institutions worldwide. All private donor interviewees also commented positively on Perimeter's ability to attract top faculty members and PDFs, noting that this also helps local and Canadian universities recruit talent.

Multiple lines of evidence from the third-party evaluation also support the accomplishment of many scientific achievements of high interest and importance worldwide, including Perimeter researchers named to the “World's Most Influential Scientific Minds” list by Thomson Reuters^{Footnote 25} and many researchers being awarded prestigious scientific prizes. Perimeter produces many research papers of very high quality published in top ranking scientific journals. SAC interview respondents rated the Perimeter fields in which they had the highest expertise as having, on average, research work with a scientific importance at or near “world-leading.”

The third-party evaluation concluded that Perimeter has had a significant impact on Canada's reputation in foundational theoretical physics. It is recognized as a true world-leading institution by stakeholders, and has been ranked second in theoretical physics worldwide in the Max Planck Society's study “Mapping Research Excellence”. These findings are consistent with the 2015 independent SAC report that noted “Perimeter Institute has established itself beyond doubt as a top international player in theoretical physics and contributes significantly to Canada's visibility in fundamental and potentially transformative research.” Perimeter is now considered to be “the default” (according to the SAC) when international researchers think about Canadian foundational theoretical physics. The researcher survey respondents agreed, stating that Perimeter has made Canada's reputation in Perimeter's specific fields better to much better than it was previously (83% - 100% of researchers depending on the field), and provided higher ratings of Canada's reputation than in the 2011 evaluation.

The third-party Perimeter evaluation also identified Perimeter researchers as engaging in a number of experimental and observational connections contributing to the exchange of research knowledge and results. These efforts include connections to the Large Hadron Collider (LHC) at the European Council for Nuclear Research (CERN), the Event Horizon Telescope (EHT), the Laser Interferometric Gravitational-Wave Observatory (LIGO),^{Footnote 26} the Planck satellite, and the Square Kilometre Array (SKA). At the same time, Perimeter's scientists work with leading Canadian experiments – such as the Sudbury Neutrino Observatory Laboratory (SNOLAB) and the Canadian Hydrogen Intensity Mapping Experiment (CHIME). SAC interviewees from the third-party evaluation who commented on this stated that Perimeter views such interactions somewhat more favourably^{Footnote 27} at an institute/corporate level than is usual; i.e., more than what is routine for a given field, with Perimeter theorists helping to understand mathematically what is experimentally observed. The SAC respondents commented that these connections may position Perimeter to be slightly more ready to capitalize on new data and discoveries as a result, while cautioning that this should not in any way be seen to suggest that Perimeter should become directly involved with short-term, strictly practical applications.

As noted by Perimeter interviewees, these connections are important and Perimeter researchers are able to create the strong deep links to the experimental facilities, as their time is not fractured by other university needs.

3.2.2 To what extent has Perimeter expanded its capacity to attract and train the next generation of researchers in theoretical physics, including both masters and PhD programs?

Overall, the third-party Perimeter evaluation found Perimeter to be successful in its recruitment of research trainees. The researcher and partner survey results identify Perimeter as attracting PhD and Masters students of the highest international caliber to a great extent (80% - 85% of Partners depending on the trainee level and 71% - 81% of Researchers depending on the trainee level).^{Footnote 28} The partner survey results also noted that access to higher quality students is a direct benefit resulting from their partnership with Perimeter (five out of eight partners).

The perceived extent to which Perimeter is attracting top caliber undergraduate students for its summer program is slightly lower than the other categories of trainees according to the results of the researcher and partner survey (53% of Researchers rated from a great to very great extent and 75% of Partners did not know). While it is acknowledged that this is a very small program (6-10 students per year completing a two to four month summer project), and is not a major focus of Perimeter, the results in this area stand out somewhat against the results for PhD and Masters students.

The document review identified that undergraduate recruitment for Perimeter has doubled for Canadian applicants since program inauguration in 2008 and the most recent application intake is five times greater (includes international students). Graduate students applying to the Perimeter Scholars International program have more than doubled since inauguration where the latest intake saw 570 applicants for 30 available positons.

Through the trainee survey conducted as part of the third-party Perimeter evaluation, results indicated that Perimeter is considered by trainees (75% to 86% of trainees depending on the area) to provide a very high level of research training to students in areas such as exposure to theories, providing supervision by top researchers, and providing opportunities to interact with top researchers in the student's field as well as across fields. These Perimeter trainees identified that their intent to pursue an academic research career is more likely – and more likely to be successful – because of exposure to Perimeter.

Interestingly, the results of private donor interviews in the third-party Perimeter evaluation found several of the donors associated with science or technology organizations noting that Perimeter also has a significant “people impact” in two areas not directly associated with its core mandate, but that are potentially of considerable societal significance. The first is that Perimeter is proactive in attracting and supporting women in the STEM disciplines, an area respondents noted as being traditionally less well served for women in Canada. The second is that Perimeter is seen to have – or potentially have – important impacts in attracting young people in general into the STEM fields.

3.2.3 To what extent has Perimeter enhanced and expanded its public education and outreach activities to effectively promote science and physics and demonstrate how the research from theoretical physics can be applied?

Evidence from the third-party Perimeter evaluation indicated that high school teachers make high use of Perimeter teaching materials, consistent with Perimeter's 2011 evaluation results. The Perimeter materials were viewed by teachers as being reliable, trustworthy, well planned, and engaging, while addressing topics that are more cutting-edge than can easily be developed independently. The Perimeter materials have an exceptionally broad reach, with potentially 12,500 teachers and 750,000 students using these materials each year based on an extrapolation of teacher survey results from the third-party evaluation.

Analysis of administrative data held by Perimeter show an increase in demand for its educational programming, with the interest from applicants exceeding the planned capacity for a number of programs. As an example, Perimeter's International Summer School for Young Physicists (ISSYP) applications have doubled to 445 since 2013 with only 40 seats available each year. Teacher attendance at Perimeter sponsored workshops has also increased by about 60% over the evaluation timeframe.

With respect to its more general outreach activities and materials, Perimeter is also well regarded. The results of the general public survey conducted as part of the third-party Perimeter evaluation indicated positive impacts of Perimeter's outreach initiatives on public interest in theoretical physics and science in general (82% indicating from a moderate to very great extent). Perimeter outreach and communications were consistently found (88% - 96% of the general public depending on the criteria) to be credible, of high quality, addressing leading-edge topics, inspirational, and relevant to daily and community life. In addition, the teacher's survey results suggested that Perimeter has a positive effect on student decisions to engage in a STEM career for at least some students (56% of teachers reporting Perimeter's influence from a moderate to very great extent).

Several respondents from the donor interviews and outreach partner interviews conducted as part of the third-party Perimeter evaluation commented these outreach effects were broader than perhaps originally intended. Perimeter is seen to be helping support the public education system generally (e.g., by helping provide a central source of top quality physics teaching material in the context of more constrained education budgets), and commented that if possible they should be extended and expanded. These effects were noted not just in Canada, but also in the United Kingdom (UK) and United States (US).

3.2.4 Have Perimeter's structures and processes facilitated efficient delivery of its programming?

Analysis of administrative data from the third-party Perimeter evaluation found that Perimeter employs a public-private model that is roughly equally funded by government (53.5%) and private (46.5%) sources. For every dollar invested by the federal government, Perimeter has been able to leverage $2.56 of other funding, exceeding the 1:1 matched funding ratio required by the ISED multi-year agreement. Perimeter has maintained administrative costs steady at approximately 20% of total operating costs from 2013 – 2015.

Through the document review, it was found that Perimeter addresses opportunities for continuous improvement by reviewing policies and administrative practices regularly. This practice has resulted in a number of process efficiency improvements such as the implementation of upgraded human resource and financial systems that reduce manual and paper processes and increase the timeliness of operations. It also uses modern collaboration tools and technologies to serve all programming from researcher collaboration to outreach material distribution and access (e.g., leading to reduced travel costs and material production/distribution costs).

The operational benchmarking identified three cost ratio results most applicable for consideration with Perimeter as identified in the following table:

The two institutes used as examples in the table above are very different from Perimeter with one being approximately 120 times the size of Perimeter with it undertaking commercialization activities and the other being somewhat smaller but broader in its fields of study. Neither of these institutes undertake education and outreach activities, as does Perimeter. Thus, this limits the extent to which evaluators can draw further conclusions regarding Perimeter's cost-efficiency (also discussed in section 2.5). Nonetheless, the comparison table shows Perimeter is in line with the Institutes used as examples, such that percentages are not so high to cause concern there are significant inefficiencies and percentages are not so low to cause concern that costs are hidden or that not enough support is provided to researchers.

There were no opportunities for improvement identified by either Perimeter or ISED interviewees.

3.2.5 To what extent is Innovation, Science and Economic Development Canada efficiently managing its contribution to Perimeter Institute?

ISED's management of the federal contribution to Perimeter is part of a portfolio of funding agreements. The planned allocation of Full-time Equivalents (FTE) is documented to fluctuate between 0.5 of an FTE over the course of the funding agreement to manage monitoring activities while increasing to 0.75 of an FTE when policy/renewal consideration activities take place. This effort peaks at the end of the five-year agreement. ISED program interviewees identified the Perimeter agreement to be a straightforward agreement and Perimeter has always been in compliance, so administrative efforts are in line with plans.

ISED service standards in relation to the management of the Perimeter funding agreement were in place and measured for the 2015-16 fiscal year. Applicable service standards include acknowledging receipt of submitted reports (within two days), providing feedback on reporting (within two weeks) and releasing/advancing payments (within 45 days) – all within specified response times. According to the 2016-17 Management Accountability Framework (MAF) assessment, all service standards applicable to Perimeter were met in 2015-16.

4.0 Conclusions and recommendations

4.1 Conclusions

The evaluation reached the following conclusions about relevance and performance.

Relevance

• There is a continued need for the foundational research conducted by Perimeter and its excellence objectives as it contributes to Canada's overall strength and competitiveness in scientific knowledge generation internationally and supports broader societal objectives to develop science literacy and skills to advance innovation in Canada.
• Perimeter's objectives are aligned with federal government priorities to invest in world- class institutions and research, as well as highly-skilled people, to drive innovation.
• The federal government has a role in the funding of foundational or basic research as a general contribution to societal need and without such public sector intervention, the private market would likely not adequately support certain types of research.

Performance

• Perimeter has significantly added to Canadian capability in foundational theoretical physics through its many scientific achievements, attraction of a very high caliber of researchers and deep linkages to experimental and observational research centres. This is successfully positioning Canada as a world leader, and Perimeter's influence on Canada's reputation is higher than just five years ago.
• Perimeter is having a positive effect on recruitment, within and to Canada, of top quality research trainees through unique programming and student exposure to top researchers across many fields. While overall trainee recruitment is highly successful, the attraction of top undergraduate students may not be on par with other categories of researchers.
• Perimeter's education and outreach program is highly regarded and demonstrates considerable effectiveness in promoting science and physics through the noted high use of its training materials. High school teachers, student and public audiences continue to increase and positive influences from Perimeter's efforts were identified.
• Perimeter has been able to implement a model supporting the efficient delivery of its programming through high ratios of private/public funding, maintenance of uniform administrative costs and the implementation of initiatives targeted to improve operating efficiency. Its ratio of indirect costs to total costs of all program activities is in line with the cost ratio ranges of other research institutes.
• ISED is meeting its planned commitments in relation to expenditures of FTEs in the management of Perimeter's funding agreement. It has also met set service standards in relation to the execution of the agreement.

4.2 Recommendations

Overall, the evaluation results demonstrated Perimeter's effectiveness in achieving performance results and its ongoing relevance. No recommendations resulted from this evaluation.

The recommendations highlighted in the third-party evaluation are broader than the expected results stipulated in ISED's funding agreement and focus on continued improvement. These can be found in Appendix A.

Appendix A: Third-party evaluation recommendations

Below are the recommendations made in the third-party evaluation report. These recommendations are broader than the expected results stipulated in ISED's funding agreement and were identified in the interest of continued improvement:

1. Maintain and possibly increase partnerships for outreach and education. The results of the third-party evaluation suggest that Perimeter is having considerable success in increasing the general public interest for physics and science and its education and outreach material are perceived to be highly compelling. There appears to be opportunity for Perimeter to continue to extend these activities – likely through external partnerships, to prevent dilution of its central mission – in order to create even broader societal benefits. In particular, benefits may be realized for youth, women, and the disadvantaged, and in certain regions, where career opportunities are often slim in the Science, Technology, Engineering and Math (STEM) fields.
2. Investigate the undergraduate programs. In the third-party evaluation, Perimeter's ability to attract top undergraduate students was the lowest of the ratings regarding attraction of various categories of researchers and research trainees, and the average rating was somewhat lower than the Perimeter evaluation conducted in 2011. It is worth considering whether there is any underlying issue or opportunity in this area, and, if so, whether it can be addressed through additional management action. While it is acknowledged that this is a very small program (6-10 students per year completing a two to four month summer project), and is not a major focus of Perimeter, the results in this area stand out somewhat amongst the other very positive ones.